research-article

Can Computers Outperform Humans in Detecting User Zone-Outs? Implications for Intelligent Interfaces

Authors:

Sidney K. D'MelloAuthors Info & Claims

ACM Transactions on Computer-Human Interaction, Volume 29, Issue 2

Article No.: 10, Pages 1 - 33

https://doi.org/10.1145/3481889

Published: 16 January 2022 Publication History

Abstract

The ability to identify whether a user is “zoning out” (mind wandering) from video has many HCI (e.g., distance learning, high-stakes vigilance tasks). However, it remains unknown how well humans can perform this task, how they compare to automatic computerized approaches, and how a fusion of the two might improve accuracy. We analyzed videos of users’ faces and upper bodies recorded 10s prior to self-reported mind wandering (i.e., ground truth) while they engaged in a computerized reading task. We found that a state-of-the-art machine learning model had comparable accuracy to aggregated judgments of nine untrained human observers (area under receiver operating characteristic curve [AUC] = .598 versus .589). A fusion of the two (AUC = .644) outperformed each, presumably because each focused on complementary cues. Furthermore, adding more humans beyond 3–4 observers yielded diminishing returns. We discuss implications of human–computer fusion as a means to improve accuracy in complex tasks.

References

[1]

Benjamin Baird, Jonathan Smallwood, Michael D. Mrazek, Julia W. Y. Kam, Michael S. Franklin, and Jonathan W. Schooler. 2012. Inspired by distraction: Mind wandering facilitates creative incubation. Psychological Science 23, 10 (October 2012), 1117–1122. DOI:

[2]

Ryan S. Baker, Sidney K. D'Mello, Ma. Mercedes T. Rodrigo, and Art Graesser. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68, 4 (April 2010), 223–241. DOI:https://doi.org/10.1016/j.ijhcs.2009.12.003

Digital Library

[3]

Tadas Baltrušaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. OpenFace 2.0: Facial behavior analysis toolkit. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face Gesture Recognition, IEEE, Piscataway, NJ, 59–66. DOI:

[4]

Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. 2019. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest 20, 1 (July 2019), 1–68. DOI:

[5]

Marian Stewart Bartlett, Gwen C. Littlewort, Mark G. Frank, and Kang Lee. 2014. Automatic decoding of facial movements reveals deceptive pain expressions. Current Biology 24, 7 (March 2014), 738–743.

[6]

Bogdan Batrinca and Philip C. Treleaven. 2015. Social media analytics: A survey of techniques, tools and platforms. AI & Society 30, 1 (February 2015), 89–116. DOI:https://doi.org/10.1007/s00146-014-0549-4

Digital Library

[7]

Mathias Benedek, David Daxberger, Sonja Annerer-Walcher, and Jonathan Smallwood. 2018. Are you with me? Probing the human capacity to recognize external/internal attention in others’ faces. Visual Cognition 26, 7 (August 2018), 511–517. DOI:

[8]

Robert Bixler, Nathaniel Blanchard, Luke Garrison, and Sidney K. D'Mello. 2015. Automatic detection of mind wandering during reading using gaze and physiology. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ACM, New York, NY, 299–306. DOI:https://doi.org/10.1145/2818346.2820742

[9]

Nathaniel Blanchard, Robert Bixler, Tera Joyce, and Sidney K. D'Mello. 2014. Automated physiological-based detection of mind wandering during learning. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems Lecture Notes in Computer Science, Springer, Cham, CH, 55–60. DOI:https://doi.org/10.1007/978-3-319-07221-0_7

[10]

John Bohannon. 2011. Social science for pennies. Science 334, 6054 (October 2011), 307–307. DOI:

[11]

Nigel Bosch and Sidney K. D'Mello. in press. Automatic detection of mind wandering from video in the lab and in the classroom. IEEE Transactions on Affective Computing (in press). DOI:

[12]

Charles Vernon Boys. 1890. Soap-bubbles, and the forces which mould them. Society for Promoting Christian Knowledge.

[13]

Taylor A. Burke, Brooke A. Ammerman, and Ross Jacobucci. 2019. The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: A systematic review. Journal of Affective Disorders 245, (February 2019), 869–884. DOI:

[14]

Rafael A. Calvo and Sidney K. D'Mello. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing 1, 1 (January 2010), 18–37. DOI:https://doi.org/10.1109/T-AFFC.2010.1

Digital Library

[15]

James Carpenter and John Bithell. 2000. Bootstrap confidence intervals: When, which, what? A practical guide for medical statisticians. Statistics in Medicine 19, 9 (2000), 1141–1164. DOI:

[16]

Kalina Christoff, Alan M. Gordon, Jonathan Smallwood, Rachelle Smith, and Jonathan W. Schooler. 2009. Experience sampling during fMRI reveals default network and executive system contributions to mind wandering. In Proceedings of the National Academy of Sciences 106, 21 (May 2009), 8719–8724. DOI:

[17]

Kalina Christoff, Zachary C. Irving, Kieran C. R. Fox, R. Nathan Spreng, and Jessica R. Andrews-Hanna. 2016. Mind-wandering as spontaneous thought: A dynamic framework. Nature Reviews Neuroscience 17, 11 (November 2016), 718–731. DOI:

[18]

Scott Clifford, Ryan M. Jewell, and Philip D. Waggoner. 2015. Are samples drawn from Mechanical Turk valid for research on political ideology? Research & Politics 2, 4 (October 2015), 1–9. DOI:

[19]

Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum, Hillsdale, NJ.

[20]

Carlos Crivelli and Alan J. Fridlund. 2018. Facial displays are tools for social influence. Trends in Cognitive Sciences 22, 5 (May 2018), 388–399. DOI:

[21]

Carlos Crivelli and Alan J. Fridlund. 2019. Inside-out: From basic emotions theory to the behavioral ecology view. Journal of Nonverbal Behavior 43, 2 (June 2019), 161–194. DOI:

[22]

M. Csikszentmihalyi. 1990. Flow: The psychology of optimal experience. Harper and Row.

[23]

Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, 1–12. DOI:https://doi.org/10.1145/3313831.3376638

Digital Library

[24]

Sidney K. D'Mello. 2019. What do we think about when we learn? In Proceedings of the Deep Comprehension: Multi-Disciplinary Approaches to Understanding, Enhancing, and Measuring Comprehension, Keith Millis, J. Magliano, D. Long and K. Wiemer (Eds.). Routledge, New York, NY, 52–67.

[25]

Sidney K. D'Mello, Ed Dieterle, and Angela L. Duckworth. 2017. Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist 52, 2 (April 2017), 104–123.

[26]

Sidney K. D'Mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys 47, 3 (February 2015), 43:1–43:36.

Digital Library

[27]

Sidney K. D'Mello, Blair Lehman, Reinhard Pekrun, and Art Graesser. 2014. Confusion can be beneficial for learning. Learning and Instruction 29, 1 (2014), 153–170.

[28]

Kunio Doi. 2007. Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Computerized Medical Imaging and Graphics 31, 4 (June 2007), 198–211. DOI:

[29]

Damien Dupré, Eva G. Krumhuber, Dennis Küster, and Gary J. McKeown. 2020. A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PLOS ONE 15, 4 (April 2020), 0231968:1–17. DOI:

[30]

Dominic B. Dwyer, Peter Falkai, and Nikolaos Koutsouleris. 2018. Machine learning approaches for clinical psychology and psychiatry. Annual Review of Clinical Psychology 14, 1 (2018), 91–118. DOI:

[31]

Paul Ekman, Richard J. Davidson, and Wallace V. Friesen. 1990. The Duchenne smile: Emotional expression and brain physiology: II. Journal of Personality and Social Psychology 58, 2 (1990), 342–353.

[32]

Paul Ekman and Wallace V. Friesen. 1978. Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press.

[33]

El-Sayed A. El-Dahshan, Heba M. Mohsen, Kenneth Revett, and Abdel-Badeeh M. Salem. 2014. Computer-aided diagnosis of human brain tumor through MRI: A survey and a new algorithm. Expert Systems with Applications 41, 11 (September 2014), 5526–5545. DOI:

[34]

Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik. 2016. First I “like” it, then I hide it: Folk theories of social feeds. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, 2371–2382. DOI:https://doi.org/10.1145/2858036.2858494

[35]

Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. “I always assumed that I wasn't really that close to [her]”: Reasoning about invisible algorithms in news feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM, New York, NY, 153–162. DOI:https://doi.org/10.1145/2702123.2702556

[36]

Myrthe Faber, Robert Bixler, and Sidney K. D'Mello. 2018. An automated behavioral measure of mind wandering during computerized reading. Behavior Research Methods 50, 1 (2018), 134–150. DOI:

[37]

Myrthe Faber and Sidney K. D'Mello. 2018. How the stimulus influences mind wandering in semantically rich task contexts. Cognitive Research: Principles and Implications 3, 35 (2018), 1–14. DOI:

[38]

Myrthe Faber, Kristina Krasich, Robert E. Bixler, James R. Brockmole, and Sidney K. D'Mello. 2020. The eye–mind wandering link: Identifying gaze indices of mind wandering across tasks. Journal of Experimental Psychology: Human Perception and Performance 46, 10 (2020), 1201–1221. DOI:

[39]

Myrthe Faber, McKenzie Rees, and Sidney K. D'Mello. 2018. Mind wandering during conversations affects subjective but not objective outcomes. In Proceedings of the CogSci 2018.

[40]

David J. Frank, Brent Nara, Michela Zavagnin, Dayna R. Touron, and Michael J. Kane. 2015. Validating older adults’ reports of less mind-wandering: An examination of eye movements and dispositional influences. Psychology and Aging 30, 2 (2015), 266. DOI:

[41]

Michael S. Franklin, James M. Broadway, Michael D. Mrazek, Jonathan Smallwood, and Jonathan W. Schooler. 2013. Window to the wandering mind: Pupillometry of spontaneous thought while reading. The Quarterly Journal of Experimental Psychology 66, 12 (December 2013), 2289–2294. DOI:

[42]

Jennifer A. Fredricks, Phyllis C. Blumenfeld, and Alison H. Paris. 2004. School engagement: Potential of the concept, state of the evidence. Review of Educational Research 74, 1 (2004), 59–109. DOI:

[43]

Cédric Galéra, Ludivine Orriols, Katia M'Bailara, Magali Laborey, Benjamin Contrand, Régis Ribéreau-Gayon, Françoise Masson, Sarah Bakiri, Catherine Gabaude, Alexandra Fort, Bertrand Maury, Céline Lemercier, Maurice Cours, Manuel-Pierre Bouvard, and Emmanuel Lagarde. 2012. Mind wandering and driving: Responsibility case-control study. BMJ Clinical Research 345, 7888, (December 2012), e8105. DOI:

[44]

Fiona J. Gilbert, Susan M. Astley, Maureen G. C. Gillan, Olorunsola F. Agbaje, Matthew G. Wallis, Jonathan James, Caroline R. M. Boggis, and Stephen W. Duffy. 2008. Single reading with computer-aided detection for screening mammography. New England Journal of Medicine 359, 16 (October 2008), 1675–1684. DOI:

[45]

Joseph K. Goodman, Cynthia E. Cryder, and Amar Cheema. 2013. Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making 26, 3 (July 2013), 213–224. DOI:

[46]

Romain Grandchamp, Claire Braboszcz, and Arnaud Delorme. 2014. Oculometric variations during mind wandering. Frontiers in Psychology 5, (February 2014), 31. DOI:

[47]

Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: Using mechanical turk to assess visualization design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, NY, 203–212. DOI:https://doi.org/10.1145/1753326.1753357

[48]

Y. H. Holkamp and J. Schavemaker. 2014. A comparison of human and machine learning-based accuracy for valence classification of subjects in video fragments. In Proceedings of Measuring Behavior 2014, Noldus Information Technology, Wageningen, NL, 73–76.

[49]

John J. Horton, David G. Rand, and Richard J. Zeckhauser. 2011. The online laboratory: Conducting experiments in a real labor market. Experimental Economics 14, 3 (September 2011), 399–425. DOI:

[50]

Stephen Hutt, Kristina Krasich, Caitlin Mills, Nigel Bosch, Shelby White, James R. Brockmole, and Sidney K. D'Mello. 2019. Automated gaze-based mind wandering detection during computerized learning in classrooms. User Modeling and User-Adapted Interaction 29, 4 (2019), 821–867. DOI:

Digital Library

[51]

Afsaneh Jalalian, Syamsiah B. T. Mashohor, Hajjah Rozi Mahmud, M. Iqbal B. Saripan, Abdul Rahman B. Ramli, and Babak Karasfi. 2013. Computer-aided detection/diagnosis of breast cancer in mammography and ultrasound: A review. Clinical Imaging 37, 3 (May 2013), 420–426. DOI:

[52]

Joris H. Janssen, Paul Tacken, J. J. G. Gert-Jan de Vries, Egon L. van den Broek, Joyce H. D. M. Westerink, Pim Haselager, and Wijnand A. IJsselsteijn. 2013. Machines outperform laypersons in recognizing emotions elicited by autobiographical recollection. Human-Computer Interaction 28, 6 (November 2013), 479–517. DOI:https://doi.org/10.1080/07370024.2012.755421

[53]

Matthew A. Killingsworth and Daniel T. Gilbert. 2010. A wandering mind Is an unhappy mind. Science 330, 6006 (November 2010), 932–932. DOI:

[54]

Kristopher Kopp, Sidney K. D'Mello, and Caitlin Mills. 2015. Influencing the occurrence of mind wandering while reading. Consciousness and Cognition 34, (July 2015), 52–62.

[55]

Kristina Krasich, Robert McManus, Stephen Hutt, Myrthe Faber, Sidney K. D'Mello, and James R. Brockmole. 2018. Gaze-based signatures of mind wandering during real-world scene processing. Journal of Experimental Psychology: General 147, 8 (2018), 1111–1124. DOI:

[56]

Jukka M. Leppänen and Charles A. Nelson. 2009. Tuning the developing brain to social signals of emotions. Nature Reviews Neuroscience 10, 1 (January 2009), 37–47. DOI:

[57]

J. Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 37, 1 (January 1991), 145–151.

Digital Library

[58]

Gwen Littlewort, J. Whitehill, Tingfan Wu, Ian Fasel, M. Frank, J. Movellan, and M. Bartlett. 2011. The computer expression recognition toolbox. In Proceedings of the 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops. 298–305.

[59]

Bethany T. McDaniel, Sidney K. D'Mello, Brandon G. King, Patrick Chipman, Kristy Tapp, and Art Graesser. 2007. Facial features for affective state detection in learning environments. In Proceedings of the 29th Annual Cognitive Science Society. 467–472.

[60]

Jennifer C. McVay and Michael J. Kane. 2009. Conducting the train of thought: Working memory capacity, goal neglect, and mind wandering in an executive-control task. Journal of Experimental Psychology: Learning, Memory, and Cognition 35, 1 (2009), 196–204. DOI:http://psycnet.apa.org/doi/10.1037/a0014104

[61]

Jennifer C. McVay, Michael J. Kane, and Thomas R. Kwapil. 2009. Tracking the train of thought from the laboratory into everyday life: An experience-sampling study of mind wandering across controlled and ecological contexts. Psychonomic Bulletin & Review 16, 5 (October 2009), 857–863.

[62]

Vikram Mohanty, David Thames, Sneha Mehta, and Kurt Luther. 2019. Photo Sleuth: Combining human expertise and face recognition to identify historical portraits. In Proceedings of the 24th International Conference on Intelligent User Interfaces, ACM, New York, NY, 547–557. DOI:https://doi.org/10.1145/3301275.3302301

[63]

Benjamin W. Mooneyham and Jonathan W. Schooler. 2013. The costs and benefits of mind-wandering: A review. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale 67, 1 (2013), 11–18.

[64]

Jaclyn Ocumpaugh, Ryan S. Baker, and Ma. Mercedes T. Rodrigo. 2015. Baker Rodrigo Ocumpaugh Monitoring Protocol 2.0 Technical and Training Manual. Technical Report. New York, NY: Teachers College, Columbia University, Manila, Philippines: Ateneo Laboratory for the Learning Sciences.

[65]

Gabriele Paolacci, Jesse Chandler, and Panagiotis G. Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision Making 5, 5 (June 2010), 411–419.

[66]

Gregory Park, H. Andrew Schwartz, Johannes C. Eichstaedt, Margaret L. Kern, Michal Kosinski, David J. Stillwell, Lyle H. Ungar, and Martin E. P. Seligman. 2015. Automatic personality assessment through social media language. Journal of Personality and Social Psychology 108, 6 (2015), 934–952. DOI:

[67]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (November 2011), 2825–2830.

Digital Library

[68]

Phuong Pham and Jingtao Wang. 2015. AttentiveLearner: Improving mobile MOOC learning via implicit heart rate tracking. In Proceedings of the Artificial Intelligence in Education. Lecture Notes in Computer Science, Springer, Cham, CH, 367–376. DOI:

[69]

Martin F. Porter. 2001. Snowball: A language for stemming algorithms. Retrieved on 15 October, 2018 from http://snowball.tartarus.org/texts/introduction.html.

[70]

R. Core Team. 2013. R: A language and environment for statistical computing. Retrieved on 20 May, 2018 from https://www.R-project.org.

[71]

Jason G. Randall, Frederick L. Oswald, and Margaret E. Beier. 2014. Mind-wandering, cognition, and performance: A theory-driven meta-analysis of attention regulation. Psychological bulletin 140, 6 (2014), 1411–1431.

[72]

Erik D. Reichle, Andrew E. Reineberg, and Jonathan W. Schooler. 2010. Eye movements during mindless reading. Psychological Science 21, 9 (September 2010), 1300–1310.

[73]

Evan F. Risko, N. Anderson, A. Sarwal, M. Engelhardt, and Alan Kingstone. 2012. Everyday attention: Variation in mind wandering and memory in a lecture. Applied Cognitive Psychology 26, 2 (2012), 234–242.

[74]

Evan F. Risko, Dawn Buchanan, Srdan Medimorec, and Alan Kingstone. 2013. Everyday attention: Mind wandering and computer use during lectures. Computers & Education 68, (October 2013), 275–283. DOI:

[75]

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez, and Markus Müller. 2011. pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12, 1 (March 2011), 77. DOI:

[76]

Robert Rosenthal. 2005. Conducting judgment studies: Some methodological issues. In Proceedings of the New Handbook of Methods in Nonverbal Behavior Research. J. A. Harrigan, R. Rosenthal and K. R. Scherer (Eds.). Oxford University Press, New York, NY, 199–234.

[77]

Konrad Schindler, Luc Van Gool, and Beatrice de Gelder. 2008. Recognizing emotions expressed by body pose: A biologically inspired neural model. Neural Networks 21, 9 (November 2008), 1238–1246. DOI:https://doi.org/10.1016/j.neunet.2008.05.003

[78]

Jonathan W. Schooler, Erik D. Reichle, and David V. Halpern. 2005. Zoning out while reading: Evidence for dissociations between experience and metaconsciousness. In Proceedings of the Thinking and Seeing: Visual Metacognition in Adults and Children. Daniel T. Levin (Ed.). Cambridge, MA, MIT Press, 204–226.

[79]

H. Andrew Schwartz, Johannes C. Eichstaedt, Margaret L. Kern, Lukasz Dziurzynski, Stephanie M. Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin E. P. Seligman, and Lyle H. Ungar. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PLOS ONE 8, 9 (September 2013), e73791. DOI:

[80]

Paul Seli, Jonathan S. A. Carriere, D. R. Thomson, J. Allan Cheyne, K. A. E. Martens, and Daniel Smilek. 2014. Restless mind, restless body. Journal of Experimental Psychology: Learning, Memory, and Cognition 40, 3 (2014), 660–668. DOI:

[81]

Paul Seli, Michael J. Kane, Jonathan Smallwood, Daniel L. Schacter, David Maillet, Jonathan W. Schooler, and Daniel Smilek. 2018. Mind-wandering as a natural kind: A family-resemblances view. Trends in Cognitive Sciences 22, 6 (June 2018), 479–490. DOI:

[82]

Daniel J. Simons and Christopher F. Chabris. 2012. Common (mis)beliefs about memory: A replication and comparison of telephone and mechanical turk survey methods. PLoS ONE 7, 12 (December 2012), e51876. DOI:

[83]

Jonathan Smallwood, Emily Beach, Jonathan W. Schooler, and Todd C. Handy. 2008. Going AWOL in the brain: Mind wandering reduces cortical analysis of external events. Journal of Cognitive Neuroscience 20, 3 (2008), 458–469.

Digital Library

[84]

Jonathan Smallwood and Jonathan W. Schooler. 2006. The restless mind. Psychological Bulletin 132, 6 (2006), 946–958.

[85]

Jonathan Smallwood and Jonathan W. Schooler. 2015. The science of mind wandering: Empirically navigating the stream of consciousness. Annual Review of Psychology 66, 1 (2015), 487–518. DOI:

[86]

Daniel Smilek, Jonathan S. A. Carriere, and J. Allan Cheyne. 2010. Out of mind, out of sight: Eye blinking as indicator and embodiment of mind wandering. Psychological Science 21, 6 (2010), 786–789.

[87]

Angela Stewart, Nigel Bosch, Huili Chen, Patrick J. Donnelly, and Sidney K. D'Mello. 2016. Where's your mind at? Video-based mind wandering detection during film viewing. In Proceedings of the 2016 Conference on User Modeling, Adaptation, and Personalization, ACM, New York, NY, 295–296. DOI:https://doi.org/10.1145/2930238.2930266

[88]

Angela Stewart, Nigel Bosch, and Sidney K. D'Mello. 2017. Generalizability of face-based mind wandering detection across task contexts. In Proceedings of the 10th International Conference on Educational Data Mining, International Educational Data Mining Society, 88–95.

[89]

Neil Stewart, Christoph Ungemach, Adam J. L. Harris, Daniel M. Bartels, Ben R. Newell, Gabriele Paolacci, and Jesse Chandler. 2015. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making 10, 5 (2015), 479–491.

[90]

Siddharth Suri and Duncan J. Watts. 2011. Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE 6, 3 (March 2011), e16836. DOI:https://doi.org/10.1371/journal.pone.0016836

[91]

Karl K. Szpunar, Novall Y. Khan, and Daniel L. Schacter. 2013. Interpolated memory tests reduce mind wandering and improve learning of online lectures. In Proceedings of the National Academy of Sciences 110, 16 (April 2013), 6313–6317. DOI:

[92]

Sarah Uzzaman and Steve Joordens. 2011. The eyes know what you are thinking: Eye movements as an objective measure of mind wandering. Consciousness and Cognition 20, 4 (December 2011), 1882–1886. DOI:

[93]

B. Van Ginneken, B. M. Ter Haar Romeny, and M. A. Viergever. 2001. Computer-aided diagnosis in chest radiography: A survey. IEEE Transactions on Medical Imaging 20, 12 (December 2001), 1228–1241. DOI:

[94]

E. S. Venkatraman and Colin B. Begg. 1996. A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83, 4 (December 1996), 835–848. DOI:

[95]

Jacqueline Kory Westlund, Sidney K. D'Mello, and Andrew M. Olney. 2015. Motion Tracker: Camera-based monitoring of bodily movements using motion silhouettes. PLoS ONE 10, 6 (June 2015), e0130293.

[96]

J. Whitehill, Z. Serpell, Yi-Ching Lin, A. Foster, and J. R. Movellan. 2014. The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing 5, 1 (January 2014), 86–98.

[97]

David P. Williams, Michel Couillard, and Samantha Dugelay. 2014. On human perception and automatic target recognition: Strategies for human-computer cooperation. In Proceedings of the 2014 22nd International Conference on Pattern Recognition, IEEE, Piscataway, NJ, 4690–4695. DOI:https://doi.org/10.1109/ICPR.2014.802

[98]

Matthew R. Yanko and Thomas M. Spalek. 2014. Driving with the wandering mind: The effect that mind-wandering has on driving performance. Human Factors 56, 2 (March 2014), 260–269. DOI:

[99]

Neta Yitzhak, Nir Hiladi, Tanya Gurevich, Daniel S. Messinger, Emily B. Prince, Katherine Martin, and Hillel Aviezer. 2017. Gently does it: Humans outperform a software classifier in recognizing subtle, nonstereotypical facial expressions. Emotion 17, 8 (2017), 1187–1198. DOI:

[100]

X. Yu, S. Zhang, Z. Yan, F. Yang, J. Huang, N. E. Dunbar, M. L. Jensen, J. K. Burgoon, and D. N. Metaxas. 2015. Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues. IEEE Transactions on Cybernetics 45, 3 (March 2015), 492–506. DOI:https://doi.org/10.1109/TCYB.2014.2329673

Cited By

Salimzadeh SHe GGadiraju U(2024)Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision MakingProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641905(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641905
Bühler BBozkir EGoldberg PSümer ÖD’Mello SGerjets PTrautwein UKasneci E(2024)From the Lab to the Wild: Examining Generalizability of Video-based Mind Wandering DetectionInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00412-2Online publication date: 17-Jun-2024
https://doi.org/10.1007/s40593-024-00412-2
Salimzadeh SHe GGadiraju U(2023)A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision MakingProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3592959(215-227)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3565472.3592959
Show More Cited By

Index Terms

Can Computers Outperform Humans in Detecting User Zone-Outs? Implications for Intelligent Interfaces

Recommendations

Compensating for the loss of human distinctiveness: The use of social creativity under Human–Machine comparisons
Abstract
Human-machine intellectual comparisons increasingly threaten the distinctiveness of humans. Drawing on social identity theory, we assume that people will compensate for the loss of human distinctiveness in a given area by valuing “...
Highlights
- Human-machine comparison threatens the distinctiveness of human identity.
- The ...
Interactive humanoids and androids as ideal interfaces for humans
IUI '06: Proceedings of the 11th international conference on Intelligent user interfaces

We, humans, anthropomorphize targets of communication. In this sense, humanoids or androids can have ideal interface for humans. This paper focuses on two new fundamental issues in the human interface studies. There are two relationships between robots ...
Watching humans and detecting their abnormalities

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 29, Issue 2

April 2022

347 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/3505202

Editor:
Kristina Höök
KTH Royal Institute of Technology, Sweden

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 January 2022

Accepted: 01 August 2021

Revised: 01 August 2020

Received: 01 October 2019

Published in TOCHI Volume 29, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Science Foundation (NSF)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
836
Total Downloads

Downloads (Last 12 months)144
Downloads (Last 6 weeks)6

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Salimzadeh SHe GGadiraju U(2024)Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision MakingProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641905(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641905
Bühler BBozkir EGoldberg PSümer ÖD’Mello SGerjets PTrautwein UKasneci E(2024)From the Lab to the Wild: Examining Generalizability of Video-based Mind Wandering DetectionInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00412-2Online publication date: 17-Jun-2024
https://doi.org/10.1007/s40593-024-00412-2
Salimzadeh SHe GGadiraju U(2023)A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision MakingProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3592959(215-227)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3565472.3592959
Southwell RPeacock CD’Mello S(2023)Getting the Wiggles Out: Movement Between Tasks Predicts Future Mind Wandering During Learning ActivitiesArtificial Intelligence in Education10.1007/978-3-031-36272-9_40(489-501)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-36272-9_40

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents