research-article

Open access

Challenges of learning human digital twin: case study of mental wellbeing: Using sensor data and machine learning to create HDT

Authors:

Elena Vildjiounaite,

Johanna Kallio,

Julia Kantorovitch,

Simão Ferreira,

Matilde A. Rodrigues,

Nuno RochaAuthors Info & Claims

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

Pages 574 - 583

https://doi.org/10.1145/3594806.3596538

Published: 10 August 2023 Publication History

All formats PDF

Abstract

Human Digital Twin (HDT) is a powerful tool to create a virtual replica of a human, to be used for example for designing interactions with physical systems, preventing cognitive overload, managing human capital, and maintaining a healthy and motivated workforce. Building human twins is a challenging task due to the need to reliably represent each corresponding human being, and the fact that human beings notably differ from each other. Therefore, relying solely on expert knowledge is insufficient, and human twins must learn the specifics of each individual in order to accurately represent them. This paper focuses on AI methods for modelling the mental wellbeing of knowledge workers because the mounting cognitive demands of both white-collar and blue-collar work lead to employees’ stress, and stress leads to diminished creativity and motivation, increased sick leaves, and in severe cases, accidents, burnouts, and disabilities. This paper describes the main building blocks of AI-based detectors of mental stress and highlights the main challenges and future directions of research., which are expected to be relevant also for HDT learning in other domains because the high degree of individuality is ubiquitous in all human activities.

References

[1]

Michie, S., Causes and management of stress at work, Occupational and Environmental Medicine 2002, 59: 67-72

[2]

Fanny Vainionpää, Marianne Kinnula, Atte Kinnula, Kari Kuutti, and Simo Hosio. 2022. HCI and Digital Twins – A Critical Look: A Literature Review. In Proceedings of the 25th International Academic Mindtrek Conference (Academic Mindtrek '22). Association for Computing Machinery, New York, NY, USA, 75–88. https://doi.org/10.1145/3569219.3569376

Digital Library

[3]

Elizabeth M. Argyle, Adrian Marinescu, Max L. Wilson, Glyn Lawson, Sarah Sharples, Physiological indicators of task demand, fatigue, and cognition in future digital manufacturing environments, International Journal of Human-Computer Studies, Volume 145, 2021, 102522, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2020.102522

[4]

Louise Wright & Stuart Davidson, How to tell the difference between a model and a digital twin, Advanced Modeling and Simulation in Engineering Sciences volume 7, Article number: 13 (2020)

[5]

Naudet, Yannick, Alexandre Baudet, and Margot Risse. "Human Digital Twin in Industry 4.0: Concept and Preliminary Model." IN4PL. 2021.

[6]

Elena Vildjiounaite, Ville Huotari, Johanna Kallio, Vesa Kyllönen, Satu-Marja Mäkelä, Georgy Gimel'farb, Unobtrusive assessment of stress of office workers via analysis of their motion trajectories, Pervasive and Mobile Computing, Volume 58, 2019, 101028, ISSN 1574-1192, https://doi.org/10.1016/j.pmcj.2019.05.009.

Digital Library

[7]

Johanna Kallio, Elena Vildjiounaite, Jani Koivusaari, Pauli Räsänen, Heidi Similä, Vesa Kyllönen, Salla Muuraiskangas, Jussi Ronkainen, Jari Rehu, Kaisa Vehmas, Assessment of perceived indoor environmental quality, stress and productivity based on environmental sensor data and personality categorization, Building and Environment, Volume 175, 2020, 106787, ISSN 0360-1323, https://doi.org/10.1016/j.buildenv.2020.106787.

[8]

Laura P. Jiménez-Mijangos, Jorge Rodríguez-Arce, Rigoberto Martínez-Méndez & José Javier Reyes-Lagos, Advances and challenges in the detection of academic stress and anxiety in the classroom: A literature review and recommendations, Education and Information Technologies (2022)

[9]

Aditi Sharma, Kapil Sharma & Akshi Kumar, Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion, Neural Computing and Applications (2022)

[10]

Hovsepian, K., al'Absi, M., Ertin, E., Kamarck, T., Nakajima, M., Kumar, S., cStress: towards a gold standard for continuous stress assessment in the mobile environment, ACM International Joint Conference on Pervasive and Ubiquitous Computing 2015

[11]

Muaremi, A., Arnrich, B., Tröster, G., Towards Measuring Stress with Smartphones and Wearable Devices During Workday and Sleep, BioNanoScience 2013, Vol. 3, Issue 2, pp 172-183

[12]

Bogomolov, A., Lepri, B., Ferron, M., Pianesi, F., Pentland, A., Daily Stress Recognition from Mobile Phone Data, Weather Conditions and Individual Traits, Proceedings of the 22nd ACM international conference on Multimedia 2014

[13]

Ferdous, R., Osmani, V., Mayora, O., Smartphone app usage as a predictor of perceived stress levels at workplace, 9th International Conference on Pervasive Computing Technologies for Healthcare 2015

Digital Library

[14]

Garcia-Ceja, E., Osmani, V., Mayora, O., Automatic Stress Detection in Working Environments from Smartphones’ Accelerometer Data: A First Step, IEEE Journal of Biomedical and Health Informatics 2016

[15]

Ciman, M. and K. Wac, Individuals' Stress Assessment Using Human-Smartphone Interaction Analysis. IEEE Transactions on Affective Computing, 2018. 9(1): p. 51-65.

[16]

H. Gimpel, C. Regal, M. Schmidt, Mystress: unobtrusive smartphone-based stress detection, European Conference on Information Systems, 2015.

[17]

Maxhuni, A., Hernandez-Leal, P., Sucar, L.E., Osmani, V., Morales, E.F., Mayora, O., Stress modelling and prediction in presence of scarce data, Journal of Biomedical Informatics, 63 (2016) 344-356

Digital Library

[18]

Sergio Muñoz, Carlos Á. Iglesias, Oscar Mayora, Venet Osmani, Prediction of stress levels in the workplace using surrounding stress, Information Processing & Management, Volume 59, Issue 6, 2022 https://doi.org/10.1016/j.ipm.2022.103064.

Digital Library

[19]

Vildjiounaite, E., Unobtrusive stress detection on the basis of smartphone usage data. Personal and Ubiquitous Computing, 2018. 22(4): p. 671-688

Digital Library

[20]

Taylor, S., Personalized Multitask Learning for Predicting Tomorrow's Mood, Stress, and Health. IEEE Transactions on Affective Computing, 2020. 11(2)

[21]

Tervonen, J., Personalized mental stress detection with self-organizing map: From laboratory to the field. Computers in Biology and Medicine, 2020. 124: p. 103935.

[22]

Mara Naegelin, Raphael P. Weibel, Jasmine I. Kerr, Victor R. Schinazi, Roberto La Marca, Florian von Wangenheim, Christoph Hoelscher, Andrea Ferrario, An interpretable machine learning approach to multimodal stress detection in a simulated office environment, Journal of Biomedical Informatics, Volume 139, 2023, 104299, ISSN 1532-0464, https://doi.org/10.1016/j.jbi.2023.104299.

Digital Library

[23]

Carneiro D., Novais P., Pêgo J.M., Sousa N., Neves J., Using Mouse Dynamics to Assess Stress During Online Exams, InInternational Conference on Hybrid Artificial Intelligence Systems 2015, pp. 345-356.

[24]

S. Koldijk, M. A. Neerincx and W. Kraaij, "Detecting Work Stress in Offices by Combining Unobtrusive Sensors," in IEEE Transactions on Affective Computing, vol. 9, no. 2, pp. 227-239, 1 April-June 2018.

[25]

Tazarv, A., Labbaf, S., Reich, S.M., Dutt, N., Rahmani, A.M., Levorato, M., Personalized Stress Monitoring using Wearable Sensors in Everyday Settings, EMBC 2021

[26]

Samriti Sharma, Gurvinder Singh, Manik Sharma, A comprehensive review and analysis of supervised-learning and soft computing techniques for stress diagnosis in humans, Computers in Biology and Medicine, Volume 134, 2021, 104450, ISSN 0010-4825, https://doi.org/10.1016/j.compbiomed.2021.104450.

Digital Library

[27]

Sharma, N., Dhall, A., Gedeon T., Goecke, R., Thermal spatio-temporal data for stress recognition, EURASIP Journal on Image and Video Processing 2014:28

[28]

Pourmohammadi, S. and A. Maleki, Stress detection using ECG and EMG signals: A comprehensive study. Computer Methods and Programs in Biomedicine, 2020. 193: p. 105482.

[29]

Smets E., Velazquez, E. R., Schiavone, G., Chakroun, I., D'Hondt, E., De Raedt, W., Cornelis, J., Janssens, O., Van Hoecke, S., Claes, S., Van Diest, I., Van Hoof, Ch., Large-scale wearable data reveal digital phenotypes for daily-life stress detection, npj Digital Medicine volume 1, Article number: 67 (2018)

[30]

Adams Towards personal stress informatics: comparing minimally invasive techniques for measuring daily stress in the wild, PervasiveHealth 2014, pp. 72-79

[31]

Lamb, S., Kwok, K.C.S., A longitudinal investigation of work environment stressors on the performance and wellbeing of office workers, Applied Ergonomics 52 (2016) pp. 104-111

[32]

Hosseini, S.; Gottumukkala, R.; Katragadda, S.; Bhupatiraju, R.T.; Ashkar, Z.; Borst, C.W.; Cochran, K. A multimodal sensor dataset for continuous stress detection of nurses in a hospital. Sci. Data 2022, 9, 255

[33]

Vos, G., Trinh, K., Sarnyai, Z., & Azghadi, M. R. (2022). Ensemble Machine Learning Model Trained on a New Synthesized Dataset Generalizes Well for Stress Prediction Using Wearable Devices. arXiv preprint arXiv:2209.15146

[34]

van Engelen, J.E., Hoos, H.H. A survey on semi-supervised learning. Mach Learn 109, 373–440 (2020). https://doi.org/10.1007/s10994-019-05855-6

[35]

Alberdi, A., Aztiria, A., Basarab, A., Towards an automatic early stress recognition system for office environments based on multimodal measurements: a review, Journal of Biomedical Informatics 59 (2016), 49-75.

Digital Library

Cited By

Davila-Gonzalez SMartin S(2024)Human Digital Twin in Industry 5.0: A Holistic Approach to Worker Safety and Well-Being through Advanced AI and Emotional AnalyticsSensors10.3390/s2402065524:2(655)Online publication date: 19-Jan-2024
https://doi.org/10.3390/s24020655
Johnson ZSaikia M(2024)Digital Twins for Healthcare Using WearablesBioengineering10.3390/bioengineering1106060611:6(606)Online publication date: 13-Jun-2024
https://doi.org/10.3390/bioengineering11060606
Lauer-Schmaltz MCash PRivera D(2024)ETHICA: Designing Human Digital Twins—A Systematic Review and Proposed MethodologyIEEE Access10.1109/ACCESS.2024.341651712(86947-86973)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3416517
Show More Cited By

Index Terms

Challenges of learning human digital twin: case study of mental wellbeing: Using sensor data and machine learning to create HDT

Index terms have been assigned to the content through auto-classification.

Recommendations

Human Digital Twin in the context of Industry 5.0
Highlights
- Carrying out a comprehensive survey on Human Digital Twin in the context of Industry 5.0.
Abstract
Human-centricity, a core value of Industry 5.0, places humans in the center of production. It leads to the prioritization of human needs, spanning from health and safety to self-actualization and personal growth. The concept of the ...
An Exploratory Case Study on Leadership and Human Resources Perspective on High-Performance Human Resource Practices
Mental Wellbeing at Work: Perspectives of Software Engineers
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Software engineers exhibit higher burnout and suicide rates compared to many other information workers. Consequently, mental wellbeing is a growing concern to technology organizations. To better understand the challenges of supporting mental wellbeing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

July 2023

797 pages

ISBN:9798400700699

DOI:10.1145/3594806

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 August 2023

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

BusinessFinland

Conference

PETRA '23

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

July 5 - 7, 2023

Corfu, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
558
Total Downloads

Downloads (Last 12 months)525
Downloads (Last 6 weeks)66

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Davila-Gonzalez SMartin S(2024)Human Digital Twin in Industry 5.0: A Holistic Approach to Worker Safety and Well-Being through Advanced AI and Emotional AnalyticsSensors10.3390/s2402065524:2(655)Online publication date: 19-Jan-2024
https://doi.org/10.3390/s24020655
Johnson ZSaikia M(2024)Digital Twins for Healthcare Using WearablesBioengineering10.3390/bioengineering1106060611:6(606)Online publication date: 13-Jun-2024
https://doi.org/10.3390/bioengineering11060606
Lauer-Schmaltz MCash PRivera D(2024)ETHICA: Designing Human Digital Twins—A Systematic Review and Proposed MethodologyIEEE Access10.1109/ACCESS.2024.341651712(86947-86973)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3416517
Abilkaiyrkyzy ALaamarti FHamdi MSaddik A(2024)Dialogue System for Early Mental Illness Detection: Toward a Digital Twin SolutionIEEE Access10.1109/ACCESS.2023.334878312(2007-2024)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2023.3348783
Alimam HMazzuto GTozzi NEmanuele Ciarapica FBevilacqua M(2024)The resurrection of digital tripletJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2023.10184635:10Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.jksuci.2023.101846

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents