research-article

Public Access

Monitoring task engagement using facial expressions and body postures

Authors:

Akilesh Rajavenkatanarayanan,

Ashwin Ramesh Babu,

Konstantinos Tsiakas,

Fillia MakedonAuthors Info & Claims

IWISC '18: Proceedings of the 3rd International Workshop on Interactive and Spatial Computing

Pages 103 - 108

https://doi.org/10.1145/3191801.3191816

Published: 12 April 2018 Publication History

Abstract

As more industries adopt the use of robots to increase productivity, there is an increased need for effective human-robot interaction training, especially in the case of heavy and high precision robots. This implies the need for easy assessment methods that ensure accurate and personalized employee training. Most current assessments are done via manual observation and surveys. This paper addresses the need for the design of intelligent systems to assess a user's training needs based on the user's behavior and engagement while performing a vocational task simulation. In this paper, we propose a multi-sensory intelligent system to predict user engagement using facial expression and body posture data while the user performs a task to provide cognitive assessment of the user's capabilities, a critical factor in successful vocational performance using robots.

References

[1]

Maher Abujelala, Cheryl Abellanoza, Aayush Sharma, and Fillia Makedon. 2016. Brain-ee: Brain enjoyment evaluation using commercial eeg headband. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, 33.

Digital Library

[2]

Timo Ahonen, Abdenour Hadid, and Matti Pietikainen. 2006. Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence 28, 12 (2006), 2037--2041.

Digital Library

[3]

Octavio Arriaga, Matias Valdenegro-Toro, and Paul Plöger. 2017. Real-time Convolutional Neural Networks for Emotion and Gender Classification. arXiv preprint arXiv.1710.07557 (2017).

[4]

Ashwin Ramesh Babu, Akilesh Rajavenkatanarayanan, Maher Abujelala, and Fillia Makedon. 2017. Votre: A vocational training and evaluation system to compare training approaches for the workplace. In International Conference on Virtual, Augmented and Mixed Reality. Springer, 203--214.

[5]

Maria Bannert. 2002. Managing cognitive load - recent trends in cognitive load theory. Learning and instruction 12, 1 (2002), 139--146.

[6]

Chris Berka, Daniel J Levendowski, Michelle N Lumicao, Alan Yau, Gene Davis, Vladimir T Zivkovic, Richard E Olmstead, Patrice D Tremoulet, and Patrick L Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine 78, 5 (2007), B231--B244.

[7]

Nadia Bianchi-Berthouze. 2013. Understanding the role of body movement in player engagement. Human-Computer Interaction 28, 1 (2013), 40--75.

[8]

Nadia Bianchi-Berthouze, Paul Cairns, Anna Cox, Charlene Jennett, and Whan Woong Kim. 2006. On posture as a modality for expressing and recognizing emotions. In Emotion and HCI workshop at BCS HCI London.

[9]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2016. Realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1611.08050 (2016).

[10]

Raymundo Cassani, Hubert Banville, and Tiago H Falk. 2015. MuLES: An open source EEG acquisition and streaming server for quick and simple prototyping and recording. In Proceedings of the 20th International Conference on Intelligent User Interfaces Companion. ACM, 9--12.

Digital Library

[11]

Mark Coulson. 2004. Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence. Journal of Nonverbal Behavior 28, 2 (01 Jun 2004), 117--139.

[12]

Beatrice De Gelder. 2006. Towards the neurobiology of emotional body language. Nature Reviews Neuroscience 7, 3 (2006), 242--249.

[13]

Kristin Fraser, Irene Ma, Elise Teteris, Heather Baxter, Bruce Wright, and Kevin McLaughlin. 2012. Emotion, cognitive load and learning outcomes during simulation training. Medical education 46, 11 (2012), 1055--1062.

[14]

Leo Galway, Paul McCullagh, Gaye Lightbody, Chris Brennan, and David Trainor. 2015. The potential of the brain-computer interface for learning: a technology review. In Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), 2015 IEEE International Conference on. IEEE, 1554--1559.

[15]

Susan E Gathercole and Alan D Baddeley. 2014. Working memory and language. Psychology Press.

[16]

Stephen Grossberg and Lance R Pearson. 2008. Laminar cortical dynamics of cognitive and motor working memory, sequence learning and performance: toward a unified theory of how the cerebral cortex works. Psychological review 115, 3 (2008), 677.

[17]

Martijn Haak, Steven Bos, Sacha Panic, and LJM Rothkrantz. 2009. Detecting stress using eye blinks and brain activity from EEG signals. Proceeding of the 1st driver car interaction and interface (DCII 2008) (2009), 35--60.

[18]

Larry E Humes and Shari S Floyd. 2005. Measures of working memory, sequence learning, and speech recognition in the elderly. Journal of Speech, Language, and Hearing Research 48, 1 (2005), 224--235.

[19]

Thrasyvoulos Karydis, Filipe Aguiar, Simmie L Foster, and Andreas Mershin. 2015. Performance characterization of self-calibrating protocols for wearable EEG applications. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, 38.

Digital Library

[20]

Azfar Khalid, Pierre Kirisci, Zied Ghrairi, Jürgen Pannek, and Klaus-Dieter Thoben. 2017. Safety Requirements in Collaborative Human-Robot Cyber-Physical System. In Dynamics in Logistics. Springer, 41--51.

[21]

Mu Li and Bao-Liang Lu. 2009. Emotion classification based on gamma-band EEG. In Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE. IEEE, 1223--1226.

[22]

Wenhui Liao, Weihong Zhang, Zhiwei Zhu, and Qiang Ji. 2005. A real-time human stress monitoring system using dynamic Bayesian network. In Computer Vision and Pattern Recognition-Workshops, 2005. CVPR Workshops. IEEE Computer Society Conference on. IEEE, 70--70.

Digital Library

[23]

Timothy McMahan, Ian Parberry, and Thomas D Parsons. 2015. Evaluating Electroencephalography Engagement Indices During Video Game Play. In FDG.

[24]

Hanneke KM Meeren, Corné CRJ van Heijnsbergen, and Beatrice de Gelder. 2005. Rapid perceptual integration of facial expression and emotional body language. Proceedings of the National Academy of Sciences of the United States of America 102, 45 (2005), 16518--16523.

[25]

Ziad S Nasreddine, Natalie A Phillips, Valérie Bédirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L Cummings, and Howard Chertkow. 2005. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society 53, 4 (2005), 695--699.

[26]

Reinhard Pekrun and Lisa Linnenbrink-Garcia. 2012. Academic emotions and student engagement. In Handbook of research on student engagement. Springer, 259--282.

[27]

Russell A Poldrack. 2006. Can cognitive processes be inferred from neuroimaging data? Trends in cognitive sciences 10, 2 (2006), 59--63.

[28]

Ashok Samal and Prasana A Iyengar. 1992. Automatic recognition and analysis of human faces and facial expressions: A survey. Pattern recognition 25, 1 (1992), 65--77.

Digital Library

[29]

Jyotirmay Sanghvi, Ginevra Castellano, Iolanda Leite, André Pereira, Peter W McOwan, and Ana Paiva. 2011. Automatic analysis of affective postures and body motion to detect engagement with a game companion. In Proceedings of the 6th international conference on Human-robot interaction. ACM, 305--312.

Digital Library

[30]

J Stastny, Pavel Sovka, and A Stancak. 2001. EEG signal classification. In Engineering in Medicine and Biology Society, 2001. Proceedings of the 23rd Annual International Conference of the IEEE, Vol. 2. IEEE, 2020--2023.

[31]

Yingli Tian, Takeo Kanade, and Jeffrey F Cohn. 2011. Facial expression recognition. In Handbook of face recognition. Springer, 487--519.

[32]

Konstantinos Tsiakas, Cheryl Abellanoza, Maher Abujelala, Michalis Papakostas, Tasnim Makada, and Fillia Makedon. {n. d.}. Towards Designing a Socially Assistive Robot for Adaptive and Personalized Cognitive Training. ({n. d.}).

[33]

Jeroen JG Van Merrienboer and John Sweller. 2005. Cognitive load theory and complex learning: Recent developments and future directions. Educational psychology review 17, 2 (2005), 147--177.

[34]

Bernard Weiner. 1972. Theories of motivation: From mechanism to cognition. (1972).

Cited By

Tarantino LAttanasio MDi Mascio TDe Gasperis GValenti MMazza M(2023)On the Evaluation of Engagement in Immersive Applications When Users Are on the Autism SpectrumSensors10.3390/s2304219223:4(2192)Online publication date: 15-Feb-2023
https://doi.org/10.3390/s23042192
Nasihati Gilani SPollard KTraum D(2023)Multimodal Prediction of User's Performance in High-Stress Dialogue InteractionsCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3617166(71-75)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3617166
Jin LRen QMitchell VMay A(2023)A video processing and machine learning based method for evaluating safety-critical operator engagement in a motorway control roomErgonomics10.1080/00140139.2023.222378467:3(356-376)Online publication date: 27-Jun-2023
https://doi.org/10.1080/00140139.2023.2223784
Show More Cited By

Recommendations

Multimodal approach for cognitive task performance prediction from body postures, facial expressions and EEG signal
MCPMD '18: Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data

Recent developments in computer vision and the emergence of wearable sensors have opened opportunities for the development of advanced and sophisticated techniques to enable multi-modal user assessment and personalized training which is important in ...
Facial expression recognition with Convolutional Neural Networks

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
Face Recognition Through Different Facial Expressions

Face recognition has become an accessible issue for experts as well as ordinary people as it is a focal non-interfering biometric modality. In this paper, we introduced a new approach to perform face recognition under varying facial expressions. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IWISC '18: Proceedings of the 3rd International Workshop on Interactive and Spatial Computing

April 2018

118 pages

ISBN:9781450354394

DOI:10.1145/3191801

General Chair:
Prabhakaran Balakrishnan
University of Texas at Dallas
,
Program Chair:
Ryan P. McMahan
University of Texas at Dallas

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

IWISC 2018

IWISC 2018: 3rd International Workshop on Interactive and Spatial Computing

April 12 - 13, 2018

Texas, Richardson

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
457
Total Downloads

Downloads (Last 12 months)109
Downloads (Last 6 weeks)23

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tarantino LAttanasio MDi Mascio TDe Gasperis GValenti MMazza M(2023)On the Evaluation of Engagement in Immersive Applications When Users Are on the Autism SpectrumSensors10.3390/s2304219223:4(2192)Online publication date: 15-Feb-2023
https://doi.org/10.3390/s23042192
Nasihati Gilani SPollard KTraum D(2023)Multimodal Prediction of User's Performance in High-Stress Dialogue InteractionsCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3617166(71-75)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3617166
Jin LRen QMitchell VMay A(2023)A video processing and machine learning based method for evaluating safety-critical operator engagement in a motorway control roomErgonomics10.1080/00140139.2023.222378467:3(356-376)Online publication date: 27-Jun-2023
https://doi.org/10.1080/00140139.2023.2223784
Bustos-López MCruz-Ramírez NGuerra-Hernández ASánchez-Morales LCruz-Ramos NAlor-Hernández G(2022)Wearables for Engagement Detection in Learning Environments: A ReviewBiosensors10.3390/bios1207050912:7(509)Online publication date: 11-Jul-2022
https://doi.org/10.3390/bios12070509
Castiblanco Jimenez IGomez Acevedo JMarcolin FVezzetti EMoos S(2022)Towards an integrated framework to measure user engagement with interactive or physical productsInternational Journal on Interactive Design and Manufacturing (IJIDeM)10.1007/s12008-022-01087-617:1(45-67)Online publication date: 9-Nov-2022
https://doi.org/10.1007/s12008-022-01087-6
Koutny RMiesenberger K(2021)An Accessible User Interface Concept for Non-Verbal and Spatial Aspects of Business Meetings for Blind and Visually Impaired PeopleProceedings of Mensch und Computer 202110.1145/3473856.3474020(125-130)Online publication date: 5-Sep-2021
https://dl.acm.org/doi/10.1145/3473856.3474020
Ramesh Babu AZadeh MJaiswal ALueckenhoff AKyrarini MMakedon FTruong KHeylen DCzerwinski MBerthouze NChetouani MNakano M(2020)A Multi-modal System to Assess Cognition in Children from their Physical MovementsProceedings of the 2020 International Conference on Multimodal Interaction10.1145/3382507.3418829(6-14)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3382507.3418829
Nuno-Maganda MTorres-Huitzil CHernandez-Mier YDe La Calleja JMartinez-Gil CZambrano JManriquez A(2020)Smartphone-Based Remote Monitoring Tool for e-LearningIEEE Access10.1109/ACCESS.2020.30053308(121409-121423)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3005330
Babu AZakizadeh MBrady JCalderon DMakedon F(2019)An Intelligent Action Recognition System to assess Cognitive Behavior for Executive Function Disorder2019 IEEE 15th International Conference on Automation Science and Engineering (CASE)10.1109/COASE.2019.8843199(164-169)Online publication date: Aug-2019
https://doi.org/10.1109/COASE.2019.8843199
Bakry AAl-khatib RNegm RSabra EMaher MMohamed ZShawky DBadawi A(2019)Using Eye Movement to Assess Auditory Attention10.1007/978-3-030-14118-9_20(200-208)Online publication date: 17-Mar-2019
https://doi.org/10.1007/978-3-030-14118-9_20
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents