research-article

Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management

Authors:

Brian P. Bailey,

Shamsi T. IqbalAuthors Info & Claims

ACM Transactions on Computer-Human Interaction (TOCHI), Volume 14, Issue 4

Article No.: 21, Pages 1 - 28

https://doi.org/10.1145/1314683.1314689

Published: 19 January 2008 Publication History

Abstract

Notifications can have reduced interruption cost if delivered at moments of lower mental workload during task execution. Cognitive theorists have speculated that these moments occur at subtask boundaries. In this article, we empirically test this speculation by examining how workload changes during execution of goal-directed tasks, focusing on regions between adjacent chunks within the tasks, that is, the subtask boundaries. In a controlled experiment, users performed several interactive tasks while their pupil dilation, a reliable measure of workload, was continuously measured using an eye tracking system. The workload data was extracted from the pupil data, precisely aligned to the corresponding task models, and analyzed. Our principal findings include (i) workload changes throughout the execution of goal-directed tasks; (ii) workload exhibits transient decreases at subtask boundaries relative to the preceding subtasks; (iii) the amount of decrease tends to be greater at boundaries corresponding to the completion of larger chunks of the task; and (iv) different types of subtasks induce different amounts of workload. We situate these findings within resource theories of attention and discuss important implications for interruption management systems.

References

[1]

Adamczyk, P. D. and Bailey, B. P. 2004. If not now when&quest; The effects of interruptions at different moments within task execution. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 271--278.

Digital Library

[2]

Anttonen, J. and Surakka, V. 2005. Emotions and heart rate while sitting on a chair. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 491--499.

Digital Library

[3]

Bailey, B. P., Adamczyk, P. D., Chang, T. Y., and Chilson, N. A. 2006. A framework for specifying and monitoring user tasks. J. Comput. Human Behav. 22, 4, 685--708.

[4]

Bailey, B. P. and Konstan, J. A. 2006. On the need for attention aware systems: Measuring effects of interruption on task performance, error rate, and affective state. J. Comput. Human Behav. 22, 4, 709--732.

[5]

Beatty, J. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psych. Bull. 91, 2, 276--292.

[6]

Bradshaw, J. L. 1967. Pupil size as a measure of arousal during information processing. Nature 216, 515--516.

[7]

Card, S., Moran, T., and Newell, A. 1983. The psychology of human-computer interaction. Lawrence Erlbaum Associates, Hillsdale.

Digital Library

[8]

Chen, D. and Vertegaal, R. 2004. Using mental load for managing interruptions in a physiologically attentive user interface. In Proceedings of ACM Conference on Human Factors in Computing Systems. ACM, New York, 1513--1516.

Digital Library

[9]

Cutrell, E., Czerwinski, M., and Horvitz, E. 2001. Notification, disruption and memory: Effects of messaging interruptions on memory and performance. In Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (Tokyo, Japan). 263--269.

[10]

Czerwinski, M., Cutrell, E., and Horvitz, E. 2000a. Instant messaging and interruption: Influence of task type on performance. In Proceedings of the Annual Conference of the Human Factors and Ergonomics Society of Australia (OZCHI) (Sydney, Australia). C. Paris, N. Ozkan, S. Howard and S. Lu, Eds, 356--361.

[11]

Czerwinski, M., Cutrell, E., and Horvitz, E. 2000b. Instant messaging: Effects of relevance and timing. In People and Computers XIV: Proceedings of HCI, S. Turner and P. Turner, Eds, 71--76.

[12]

Dabbish, L. and Kraut, R. E. 2004. Controlling interruptions: Awareness displays and social motivation for coordination. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM, New York, 182--191.

Digital Library

[13]

Degani, A. and Wiener, E. 1993. Cockpit checklists: Concepts, design, and use. Human Factors 35, 2, 345--359.

[14]

Dey, A. K. and Abowd, G. D. 2000. CybreMinder: A context-aware system for supporting reminders. In Proceedings of 2nd International Symposium on Handheld and Ubiquitous Computing, 172--186.

Digital Library

[15]

Dismukes, K., Young, G., and Sumwalt, R. 1998. Cockpit interruptions and distractions. ASRS Direct. 10.

[16]

Donchin, E., Kramer, A. F., and Wickens, C. D. 1986. Applications of brain event related potentials to problems in engineering psychology. Guildford, New York.

[17]

Fogarty, J., Ko, A. J., Aung, H. H., Golden, E., Tang, K. P., and Hudson, S. E. 2005. Examining task engagement in sensor-based statistical models of human interruptibility. In Proceeding of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 331--340.

Digital Library

[18]

Gale, A. and Edwards, J. 1983. The EEG and human behavior. Academic Press, New York.

[19]

Gevins, A. and Schaffer, R. 1980. A critical review of electroencephalographic (EEG) correlates of higher cortical functions. CRT Crit. Rev. Bioeng. 4, 113--164.

[20]

Guerlain, S. and Willis, R. 2001. The tactical tomahawk weapons control system: Operator interface design project. In Human Systems Integration Symposium.

[21]

Hart, S. G. and Staveland, L. E. 1988. Development of a multi-dimensional workload rating scale: Results of empirical and theoretical research. In Human Mental Workload P. A. Hancock and N. Meshkati (Eds.), Elsevier, Amsterdam, The Netherlands, pp. 138--183.

[22]

Hess, E. H. 1972. Pupillometrics: A method of studying mental, emotional and sensory processes. In Handbook of Psychophysiology, N. S. Greenfield and R. A. Sternbach, (Eds.) Holt, Rinehart & Winston, New York, pp. 491--531.

[23]

Hess, E. H. and Polt, J. M. 1964. Pupil size in relation to mental activity during simple problem solving. Science 132, 1190--1192.

[24]

Hoecks, B. and Levelt, W. 1993. Pupillary dilation as a measure of attention: A quantitative system analysis. Behav. Res. Meth. Instrum. Comput. 25, 16--26.

[25]

Horvitz, E. and Apacible, J. 2003. Learning and reasoning about interruption. In Proceedings of the 5th ACM International Conference on Multimodal Interfaces. ACM, New York, 20--27.

Digital Library

[26]

Horvitz, E., Jacobs, A., and Hovel, D. 1999. Attention-sensitive alerting. In Conference Proceedings on Uncertainty in Artificial Intelligence, 305-313.

[27]

Horvitz, E., Koch, P., and Apacible, J. 2004. BusyBody: Creating and fielding personalized models of the cost of interruption. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (Chicago, IL). ACM, New York, 507--510.

Digital Library

[28]

Hudson, S. E., Fogarty, J., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J., and Yang, J. 2003. Predicting human interruptibility with sensors: A wizard of Oz feasibility study. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 257--264.

Digital Library

[29]

Hytintk, J., Tommola, J., and Alaja, A. 1995. Pupil dilation as a measure of processing load in simultaneous interpretation and other language tasks. Quart. J. Exper. Psych. 48A, 3, 598--612.

[30]

Iqbal, S. T., Adamczyk, P. D., Zheng, S., and Bailey, B. P. 2005. Towards an index of opportunity: Understanding changes in mental workload during task execution. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 311--320.

Digital Library

[31]

Iqbal, S. T. and Bailey, B. P. 2005. Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 1489--1492.

Digital Library

[32]

Iqbal, S. T. and Bailey, B. P. 2006. Leveraging characteristics of task structure to predict costs of interruption. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 741-750.

Digital Library

[33]

Iqbal, S. T. and Bailey, B. P. 2007. Understanding and developing models for detecting and differentiating breakpoints during interactive tasks. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 697--706.

Digital Library

[34]

Iqbal, S. T., Zheng, X. S., and Bailey, B. P. 2004. Task evoked pupillary response to mental workload in human-computer interaction. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 1477--1480.

Digital Library

[35]

Jackson, T. W., Dawson, R. J., and Wilson, D. 2001. The cost of email interruption. J. Syst. Inf. Tech. 5, 1, 81--92.

[36]

Juris, M. and Velden, M. 1977. The pupillary response to mental overload. Phys. Psych. 5, 4, 421--424.

[37]

Just, M. A., Carpenter, P. A., and Miyake, A. 2003. Neuroindices of cognitive workload: Neuroimaging, pupillometric, and event-related potential studies of brain work. Theoret. Issues Ergonom. 4, 56--88.

[38]

Kahneman, D. 1967. Pupillary responses in a pitch-discrimination task. Percept. Psych. 2, 101--105.

[39]

Kahneman, D. 1973. Attention and Effort. Prentice-Hall, Englewood Cliffs, N.J.

[40]

Kok, A. 1997. Event-related-potential (ERP) reflections of mental resources: A review and synthesis. Biological Psychology 45, 19--56.

[41]

Kramer, A. F. 1991. Physiological metrics of mental workload: A review of recent progress. In Multiple-Task Performance, D. L. Damos, Ed. Taylor and Francis, London, England, pp. 279--328.

[42]

Kramer, A. F., Schneider, W., Fisk, A. D., and Donchin, E. 1986. The effects of practice and task structure on components of event related brain potential. Psychophysiology 23, 33--47.

[43]

Kreifeldt, J. G. and McCarthy, M. E. 1981. Interruption as a test of the user-computer interface. In Proceedings of the 17th Annual Conference on Manual Control, Jet Propulsion Laboratory, California Institute of Technology, JPL Publication 81--95, 655--667.

[44]

Latorella, K. A. 1996. Investigating interruptions: An example from the flight deck. In Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting. 249--253.

[45]

Latorella, K. A. 1998. Effects of modality on interrupted flight deck performance: Implications for data link. In 42nd Annual Meeting of the Human Factors and Ergonomics Society. 87--91.

[46]

Lee, J. D., Hoffman, J. D., and Hayes, E. 2004. Collision warning design to mitigate driver distraction. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 65--72.

Digital Library

[47]

Lin, Y., Zhang, W. J., and Watson, L. G. 2003. Using eye movement parameters for evaluating human--machine interface frameworks under normal control operation and fault detection situations. Int. J. Human-Comput. Stud. 59, 6, 837--873.

Digital Library

[48]

Maglio, P. and Campbell, C. S. 2003. Attentive agents. Commun. ACM 46, 3, 47--51.

Digital Library

[49]

Marshall, S. P. 2002. The index ofs cognitive activity: Measuring cognitive workload. In Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants. IEEE Computer Society Press, Los Alamitos, CA, 7.5--7.9.

[50]

McCrickard, S., Chewar, C. M., Somervell, J. P., and Ndiwalana, A. 2003. A model for notification systems evaluation--assessing user goals for multitasking activity. ACM Trans. Computer-Hum. Interact. 10, 4, 312--338.

Digital Library

[51]

McFarlane, D. C. 1999. Coordinating the interruption of people in human-computer interaction. In Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction. 295--303.

[52]

McFarlane, D. C. and Latorella, K. A. 2002. The scope and importance of human interruption in HCI design. Human-Computer Interaction 17, 1, 1--61.

Digital Library

[53]

Miyata, Y. and Norman, D. A. 1986. Psychological issues in support of multiple activities. In User Centered System Design: New Perspectives on Human-Computer Interaction. D. A. Norman and S. W. Draper, Eds. Lawrence Erlbaum Associates, Hillsdale, N.J., pp. 265--284.

[54]

Monk, C. A., Boehm-Davis, D. A., and Trafton, J. G. 2002. The attentional costs of interrupting task performance at various stages. In Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting.

[55]

Nair, R., Voida, S., and Mynatt, E. 2005. Frequency-based detection of task switches. In Proceedings of the 19th British HCI Group Annual Conference (Edinburgh, Scotland). 94--99.

[56]

Nakayama, M. and Takahashi, K. 2002. The act of task difficulty and eye-movement frequency for the oculo-motor indices. In Proceedings of Eye Tracking Research and Applications. 37--42.

Digital Library

[57]

Newell, A. and Rosenbloom, P. S. 1981. Mechanisms of skill acquisition and the law of practice. In Cognitive Skills and their Acquisition, J. R. Anderson, Ed. Erlbaum, Hillsdale, NJ, pp. 1--55.

[58]

O'Donnell, R. D. and Eggemeier, F. T. 1986. Workload assessment methodology. In Handbook of Perception and Human Performance. Volume II, Cognitive Processes and Performance, K. R. Boff, L. Kaufman and J. P. Thomas, Eds., Wiley, New York, pp. 42/41--42/49.

[59]

Phelps, M. P. and Mazziotta, J. 1985. Positron emission tomography: Human brain function and biochemistry. Science 228, 799--809.

[60]

Pomplun, M. and Sunkara, S. 2003. Pupil dilation as an indicator of cognitive workload in human-computer interaction. In Proceedings of the 10th International Conference on Human-Computer Interaction, V. D. D. Harris, M. Smith and C. Stephanidis, Eds.

[61]

Rowe, D. W., Sibert, J., and Irwin, D. 1998. Heart rate variability: indicator of user state as an aid to human-computer interaction. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, 480--487.

Digital Library

[62]

Rubinstein, J. S., Meyer, D. E., and Evans, J. E. 2001. Executive control of cognitive processes in task switching. J. Exper. Psych.: Human Percept. Perform. 27, 4, 763--797.

[63]

Shneiderman, B. 1997. Designing the user interface. Pearson Addison Wesley, Reading, MA, Third Edition.

Digital Library

[64]

Speier, C., Valacich, J. S., and Vessey, I. 1999. The influence of task interruption on individual decision making: An information overload perspective. Decis. Sci. 30, 2, 337--360.

[65]

Stanton, N., Ed. 1994. Human Factors in Alarm Design. Taylor and Francis, London, UK.

Digital Library

[66]

Takahashi, K., Nakayama, M., and Shimizu, Y. 2000. The response of eye-movement and pupil size to audio instruction while viewing a moving target. In Proceedings of the ACM Conference on Eye Tracking Research and Applications. ACM, New York, 131-138.

Digital Library

[67]

Tobii-Systems. http://www.tobii.se/.

[68]

Trafton, J. G., Altmann, E. M., Brock, D. P., and Mintz, F. E. 2003. Preparing to resume an interrupted task: effects of prospective goal encoding and retrospective rehearsal. Int. J. Human-Comput. Stud. 58, 583--603.

Digital Library

[69]

Verney, S. P., Granholm, E., and Dionisio, D. P. 2001. Pupillary responses and processing resources on the visual backward masking task. Psychophysiology 38, 1, 76--83.

[70]

Verney, S. P., Granolm, E., and Marshall, S. 2004. Pupillary responses during the visual backward masking task predict cognitive ability. Int. J. Psychophys. 52, 23--36.

[71]

Wickens, C. D. 1980. The structure of attentional resources. In Attention and Performance VIII, R. Nickerson, Ed. Lawrence Erlbaum, Hillsdale, NJ, pp. 239--257.

[72]

Wickens, C. D. 1991. Processing resources and attention. In Multiple-Task Performance, D. L. Damos, Ed. Taylor & Francis, London, UK, pp. 3--34.

[73]

Wickens, C. D. 2002. Multiple resources and performance prediction. Theoret. Iss. Ergon. Sci. 3, 2, 159--177.

[74]

Zacks, J., Tversky, B., and Iyer, G. 2001. Perceiving, remembering, and communicating structure in events. J. Exper. Psych. General 130, 1, 29--58.

[75]

Zijlstra, F. R. H., Roe, R. A., Leonora, A. B., and Krediet, I. 1999. Temporal factors in mental work: Effects of interrupted activities. J. Occupat. Org. Psych. 72, 163--185.

Cited By

Hirsch PMoretti LLeichtmann BKoch INitsch V(2025)Opportune moments for task interruptions: examining the cognitive mechanisms underlying interruption-timing effectsFrontiers in Psychology10.3389/fpsyg.2024.146532315Online publication date: 15-Jan-2025
https://doi.org/10.3389/fpsyg.2024.1465323
Lee JHam Y(2025)Paradox of Time Pressure: Cognitive and Task Performance during Time-Sensitive and Challenging TeleoperationJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-15417151:1Online publication date: Jan-2025
https://doi.org/10.1061/JCEMD4.COENG-15417
McMullan RAryal NLi LWiggins MClive JWestbrook J(2025)Working memory capacity improves checking performance for errors on a simulated rail control taskApplied Ergonomics10.1016/j.apergo.2025.104482125(104482)Online publication date: May-2025
https://doi.org/10.1016/j.apergo.2025.104482
Show More Cited By

Index Terms

Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
  2. Interaction design
    1. Interaction design process and methods
      1. User centered design

Recommendations

Towards an index of opportunity: understanding changes in mental workload during task execution
CHI '05: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

To contribute to systems that reason about human attention, our work empirically demonstrates how a user's mental workload changes during task execution. We conducted a study where users performed interactive, hierarchical tasks while mental workload ...
Multitasking and monotasking: the effects of mental workload on deferred task interruptions
CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Recent research has found that forced interruptions at points of higher mental workload are more disruptive than at points of lower workload. This paper investigates a complementary idea: when users experience deferrable interruptions at points of ...
Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

This work investigates the use of workload-aligned task models for predicting opportune moments for interruption. From models for several tasks, we selected boundaries with the lowest (Best) and highest (Worst) mental workload. We compared effects of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 14, Issue 4

January 2008

204 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/1314683

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 January 2008

Accepted: 01 May 2007

Revised: 01 October 2006

Received: 01 August 2005

Published in TOCHI Volume 14, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Division of Information and Intelligent Systems

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

244
Total Citations
View Citations
3,223
Total Downloads

Downloads (Last 12 months)122
Downloads (Last 6 weeks)15

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hirsch PMoretti LLeichtmann BKoch INitsch V(2025)Opportune moments for task interruptions: examining the cognitive mechanisms underlying interruption-timing effectsFrontiers in Psychology10.3389/fpsyg.2024.146532315Online publication date: 15-Jan-2025
https://doi.org/10.3389/fpsyg.2024.1465323
Lee JHam Y(2025)Paradox of Time Pressure: Cognitive and Task Performance during Time-Sensitive and Challenging TeleoperationJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-15417151:1Online publication date: Jan-2025
https://doi.org/10.1061/JCEMD4.COENG-15417
McMullan RAryal NLi LWiggins MClive JWestbrook J(2025)Working memory capacity improves checking performance for errors on a simulated rail control taskApplied Ergonomics10.1016/j.apergo.2025.104482125(104482)Online publication date: May-2025
https://doi.org/10.1016/j.apergo.2025.104482
Iarlori SPerpetuini DTritto MCardone DTiberio AChinthakindi MFilippini CCavanini LFreddi AFerracuti FMerla AMonteriù A(2024)An Overview of Approaches and Methods for the Cognitive Workload Estimation in Human–Machine Interaction Scenarios through Wearables SensorsBioMedInformatics10.3390/biomedinformatics40200644:2(1155-1173)Online publication date: 7-May-2024
https://doi.org/10.3390/biomedinformatics4020064
Rick VBrandl CMertens ANitsch V(2024)Work interruptions of office workers: The influence of the complexity of primary work tasks on the perception of interruptionsWork10.3233/WOR-22068477:1(185-196)Online publication date: 12-Jan-2024
https://doi.org/10.3233/WOR-220684
Bahnsen KTiemann LPlabst LGrundgeiger T(2024)Augmented Reality Cues Facilitate Task Resumption after Interruptions in Computer-Based and Physical TasksProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642666(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642666
Chen LXiao SChen YSong YWu RSun L(2024)ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642229(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642229
Koundal NAbdalhadi AAl-Quraishi MElamvazuthi IMoosavi MGuillet CMerienne FSaad N(2024)Effect of Interruptions and Cognitive Demand on Mental Workload: A Critical ReviewIEEE Access10.1109/ACCESS.2024.338314212(54405-54425)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3383142
Ouyang AChen SEpps J(2024)Evaluation of Event-Based and Timing Features from Head Movement and Eye Activity for Wearable Task Transition Detection2024 International Conference on Activity and Behavior Computing (ABC)10.1109/ABC61795.2024.10651760(1-10)Online publication date: 29-May-2024
https://doi.org/10.1109/ABC61795.2024.10651760
Hirsch PWong VHensen SKoch I(2024)A task-interruption study on the activation of the primary task during the processing of the secondary taskJournal of Cognitive Psychology10.1080/20445911.2024.2439621(1-20)Online publication date: 12-Dec-2024
https://doi.org/10.1080/20445911.2024.2439621
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents