research-article

Troi: Towards Understanding Users Perspectives to Mobile Automatic Emotion Recognition System in Their Natural Setting

Authors:

Vipula Dissanayake,

Don Samitha Elvitigala,

Suranga NanayakkaraAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue MHCI

Article No.: 203, Pages 1 - 22

https://doi.org/10.1145/3546738

Published: 20 September 2022 Publication History

Abstract

Emotional Self-Awareness (ESA) plays a vital role in physical and mental well-being. Recent advancements in artificial intelligence technologies have shown promising emotion recognition results, opening new opportunities to build systems to support ESA. However, little research has been done to understand users' perspectives on artificial-intelligence-based emotion recognition systems. We introduce Troi, an automatic emotion recognition mobile app using wearable signals. With Troi, we ran a multi-day user study with 12 users to understand user preference parameters, such as perceived accuracy, confidence, preferred emotion representations, effect of self-awareness of emotions, and real-time use cases. Further, we extend our study to evaluate the machine learning model in-the-wild to understand behaviours in-the-wild. We found that users perceived accuracy of the emotion recognition model is higher than the actual model prediction accuracy; there was no strong preference for one specific emotion representation, and users' self-awareness of emotions improved over time.

References

[1]

Alaa Alslaity, Gerry Chan, Rita Orji, and Richard Wilson. 2022. Insights From Longitudinal Evaluation of Moodie Mental Health App. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 308, 8 pages. https: //doi.org/10.1145/3491101.3519851

Digital Library

[2]

Nazanin Andalibi and Justin Buss. 2020. The Human in Emotion Recognition on Social Media: Attitudes, Outcomes, Risks. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--16. https://doi.org/10.1145/3313831.3376680

Digital Library

[3]

Anja Bachmann, Christoph Klebsattel, Andrea Schankin, Till Riedel, Michael Beigl, Markus Reichert, Philip Santangelo, and Ulrich Ebner-Priemer. 2015. Leveraging Smartwatches for Unobtrusive Mobile Ambulatory Mood Assessment. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (Osaka, Japan) (UbiComp/ISWC'15 Adjunct). Association for Computing Machinery, New York, NY, USA, 1057--1062. https://doi.org/10.1145/2800835.2800960

Digital Library

[4]

David Bakker, Nikolaos Kazantzis, Debra Rickwood, and Nikki Rickard. 2016. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR mental health 3, 1 (2016), e4984. https: //doi.org/10.2196/mental.4984

[5]

David Bakker and Nikki Rickard. 2018. Engagement in mobile phone app for self-monitoring of emotional wellbeing predicts changes in mental health: MoodPrism. Journal of affective disorders 227 (2018), 432--442. https://doi.org/10. 1016/j.jad.2017.11.016

[6]

Alberto Betella and Paul FMJ Verschure. 2016. The affective slider: A digital self-assessment scale for the measurement of human emotions. PloS one 11, 2 (2016), e0148037. https://doi.org/10.1371/journal.pone.0148037

[7]

David Bethge, Thomas Kosch, Tobias Grosse-Puppendahl, Lewis L. Chuang, Mohamed Kari, Alexander Jagaciak, and Albrecht Schmidt. 2021. VEmotion: Using Driving Context for Indirect Emotion Prediction in Real-Time. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 638--651. https://doi.org/10.1145/3472749.3474775

Digital Library

[8]

Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49--59. https://doi.org/10.1016/0005- 7916(94)90063--9

[9]

Virginia K Bratton, Nancy G Dodd, and F William Brown. 2011. The impact of emotional intelligence on accuracy of self-awareness and leadership performance. Leadership & Organization Development Journal (2011). https: //doi.org/10.1108/01437731111112971

[10]

Joost Broekens and Willem-Paul Brinkman. 2013. AffectButton: A method for reliable and valid affective self-report. Int. J. Hum. Comput. Stud. 71 (2013), 641--667. https://doi.org/10.1016/j.ijhcs.2013.02.003

Digital Library

[11]

Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem--dimensional models and their implications on emotion representation and metrical evaluation. In ECAI 2016. IOS Press, 1114--1122. https://doi.org/10.3233/978--1- 61499--672--9--1114

[12]

Clara Caldeira, Yu Chen, Lesley Chan, Vivian Pham, Yunan Chen, and Kai Zheng. 2017. Mobile apps for mood tracking: an analysis of features and user reviews. In AMIA Annual Symposium Proceedings, Vol. 2017. American Medical Informatics Association, 495.

[13]

Samantha WT Chan, Shardul Sapkota, Rebecca Mathews, Haimo Zhang, and Suranga Nanayakkara. 2020. Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--23. https://doi.org/10.1145/3432190

Digital Library

[14]

Qinyue Chen, Yuchun Yan, and Hyeon-Jeong Suk. 2021. Bubble Coloring to Visualize the Speech Emotion. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA '21). Association for Computing Machinery, New York, NY, USA, Article 361, 6 pages. https://doi.org/10.1145/3411763. 3451698

Digital Library

[15]

Shih-Chu Chen, Yen-Han Chang, Jing-Han Huang, Ching-Wen Hsu, Chih-Hsuan Lin, and Pei-Yi (Patricia) Kuo. 2022. Exploring the Effect of Emotion Awareness Intervention on Reducing FoMO. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 285, 7 pages. https://doi.org/10.1145/3491101.3519839

Digital Library

[16]

Bo Cheng and Guangyuan Liu. 2008. Emotion recognition from surface EMG signal using wavelet transform and neural network. In Proceedings of the 2nd international conference on bioinformatics and biomedical engineering (ICBBE). 1363--1366. https://doi.org/10.1109/ICBBE.2008.670

[17]

Chun-Wei Chiang and Ming Yin. 2022. Exploring the Effects of Machine Learning Literacy Interventions on Laypeople's Reliance on Machine Learning Models. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI '22). Association for Computing Machinery, New York, NY, USA, 148--161. https://doi.org/10.1145/3490099.3511121

Digital Library

[18]

Karen Church, Eve Hoggan, and Nuria Oliver. 2010. A Study of Mobile Mood Awareness and Communication through MobiMood. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (Reykjavik, Iceland) (NordiCHI '10). Association for Computing Machinery, New York, NY, USA, 128--137. https: //doi.org/10.1145/1868914.1868933

Digital Library

[19]

Ned Cooper, Tiffanie Horne, Gillian R Hayes, Courtney Heldreth, Michal Lahav, Jess Holbrook, and Lauren Wilcox. 2022. A Systematic Review and Thematic Analysis of Community-Collaborative Approaches to Computing Research. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 73, 18 pages. https://doi.org/10.1145/3491102.3517716

Digital Library

[20]

Vipula Dissanayake, Sachith Seneviratne, Rajib Rana, Elliott Wen, Tharindu Kaluarachchi, and Suranga Nanayakkara. 2022. SigRep: Toward Robust Wearable Emotion Recognition With Contrastive Representation Learning. IEEE Access 10 (2022), 18105--18120. https://doi.org/10.1109/ACCESS.2022.3149509

[21]

Vipula Dissanayake, Haimo Zhang, Mark Billinghurst, and Suranga Nanayakkara. 2020. Speech Emotion Recognition "in the Wild' Using an Autoencoder. In Proc. Interspeech 2020. 526--530. https://doi.org/10.21437/Interspeech.2020--1356

[22]

Tara Donker, Katherine Petrie, Judy Proudfoot, Janine Clarke, Mary-Rose Birch, and Helen Christensen. 2013. Smartphones for smarter delivery of mental health programs: a systematic review. Journal of medical Internet research 15, 11 (2013), e2791. https://doi.org/10.2196/jmir.2791

[23]

Andrius Dzedzickis, Arturas Kaklauskas, and Vytautas Bucinskas. 2020. Human emotion recognition: Review of ¯ sensors and methods. Sensors 20, 3 (2020), 592. https://doi.org/10.3390/s20030592

[24]

Alex Egg. 2021. Online Learning for Recommendations at Grubhub. In Fifteenth ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 569--571. https://doi.org/10.1145/3460231.3474599

Digital Library

[25]

Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion 6, 3--4 (1992), 169--200. https://doi.org/10. 1080/02699939208411068

[26]

Don Samitha Elvitigala, Denys JC Matthies, and Suranga Nanayakkara. 2020. StressFoot: Uncovering the potential of the foot for acute stress sensing in sitting posture. Sensors 20, 10 (2020), 2882.

[27]

Don Samitha Elvitigala, Philipp M. Scholl, Hussel Suriyaarachchi, Vipula Dissanayake, and Suranga Nanayakkara. 2021. StressShoe: A DIY Toolkit for Just-in-Time Personalised Stress Interventions for Office Workers Performing Sedentary Tasks. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction. Association for Computing Machinery, New York, NY, USA, Article 38, 14 pages. https://doi.org/10.1145/3447526.3472023

Digital Library

[28]

Surjya Ghosh, Niloy Ganguly, Bivas Mitra, and Pradipta De. 2017. TapSense: Combining Self-Report Patterns and Typing Characteristics for Smartphone Based Emotion Detection. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI '17). Association for Computing Machinery, New York, NY, USA, Article 2, 12 pages. https://doi.org/10.1145/3098279.3098564

Digital Library

[29]

Daniel Goleman. 1996. Emotional intelligence. Why it can matter more than IQ. Learning 24, 6 (1996), 49--50.

[30]

Simeng Gu, Wei Wang, Fushun Wang, and Jason H Huang. 2016. Neuromodulator and emotion biomarker for stress induced mental disorders. Neural plasticity 2016 (2016). https://doi.org/10.1155/2016/2609128

[31]

Kunal Gupta, Jovana Lazarevic, Yun Suen Pai, and Mark Billinghurst. 2020. AffectivelyVR: Towards VR Personalized Emotion Recognition. In 26th ACM Symposium on Virtual Reality Software and Technology. 1--3. https://doi.org/10. 1145/3385956.3422122

[32]

Neska El Haouij, Jean-Michel Poggi, Sylvie Sevestre-Ghalila, Raja Ghozi, and Mériem Jaïdane. 2018. AffectiveROAD system and database to assess driver's attention. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing. 800--803.

Digital Library

[33]

Yu-Liang Hsu, Jeen-Shing Wang, Wei-Chun Chiang, and Chien-Han Hung. 2017. Automatic ECG-based emotion recognition in music listening. IEEE Transactions on Affective Computing 11, 1 (2017), 85--99. https://doi.org/10.1109/ TAFFC.2017.2781732

[34]

Yun Huang, Ying Tang, and Yang Wang. 2015. Emotion Map: A Location-Based Mobile Social System for Improving Emotion Awareness and Regulation (CSCW '15). Association for Computing Machinery, New York, NY, USA, 130--142. https://doi.org/10.1145/2675133.2675173

Digital Library

[35]

Rachael E Jack, Oliver GB Garrod, and Philippe G Schyns. 2014. Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current biology 24, 2 (2014), 187--192. https://doi.org/10.1016/j.cub.2013.11.064

[36]

Longbin Jin. 2020. Emotion Recognition Based BCI Using Channel-Wise Features. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3334480.3381446

Digital Library

[37]

Sukumar Karumuri, Radoslaw Niewiadomski, Gualtiero Volpe, and Antonio Camurri. 2019. From Motions to Emotions: Classification of Affect from Dance Movements Using Deep Learning. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA '19). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3290607.3312910

Digital Library

[38]

Harmanpreet Kaur, Daniel McDuff, Alex C. Williams, Jaime Teevan, and Shamsi T. Iqbal. 2022. ?I Didn't Know I Looked Angry": Characterizing Observed Emotion and Reported Affect at Work. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 199, 18 pages. https://doi.org/10.1145/3491102.3517453

Digital Library

[39]

Rachel Kenny, Barbara Dooley, and Amanda Fitzgerald. 2015. Feasibility of "CopeSmart": a telemental health app for adolescents. JMIR mental health 2, 3 (2015), e4370. https://doi.org/10.2196/mental.4370

[40]

Byoung Chul Ko. 2018. A brief review of facial emotion recognition based on visual information. sensors 18, 2 (2018), 401. https://doi.org/10.3390/s18020401

[41]

Chuanhe Liu, Tianhao Tang, Kui Lv, and Minghao Wang. 2018. Multi-feature based emotion recognition for video clips. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 630--634. https://doi.org/10. 1145/3242969.3264989

Digital Library

[42]

Valentina Markova, Todor Ganchev, and Kalin Kalinkov. 2019. CLAS: A Database for Cognitive Load, Affect and Stress Recognition. In 2019 International Conference on Biomedical Innovations and Applications (BIA). 1--4. https: //doi.org/10.1109/BIA48344.2019.8967457

[43]

Daniel McDuff, Amy Karlson, Ashish Kapoor, Asta Roseway, and Mary Czerwinski. 2012. AffectAura: An Intelligent System for Emotional Memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 849--858. https://doi.org/ 10.1145/2207676.2208525

Digital Library

[44]

Saeid Mohanna and Siavash Talepasand. 2016. The relationship between environmental supports and emotional self-awareness with academic engagement: The mediating role of educational well-being. Iranian Journal of Medical Education 16 (2016), 31--42.

[45]

Margaret E Morris, Qusai Kathawala, Todd K Leen, Ethan E Gorenstein, Farzin Guilak, William DeLeeuw, and Michael Labhard. 2010. Mobile therapy: case study evaluations of a cell phone application for emotional self-awareness. Journal of medical Internet research 12, 2 (2010), e10. https://doi.org/10.2196/jmir.1371

[46]

Mimi Nguyen, Milad Laly, Bum Chul Kwon, Celine Mougenot, and John McNamara. 2022. Moody Man: Improving Creative Teamwork through Dynamic Affective Recognition. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 465, 14 pages. https://doi.org/10.1145/3491101.3519656

Digital Library

[47]

Andrea Papenmeier, Dagmar Kern, Daniel Hienert, Yvonne Kammerer, and Christin Seifert. 2022. How Accurate Does It Feel? -- Human Perception of Different Types of Classification Mistakes. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 180, 13 pages. https://doi.org/10.1145/3491102.3501915

Digital Library

[48]

Cheul Young Park, Narae Cha, Soowon Kang, Auk Kim, Ahsan Habib Khandoker, Leontios Hadjileontiadis, Alice Oh, Yong Jeong, and Uichin Lee. 2020. K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations. Scientific Data 7, 1 (dec 2020), 293. https://doi.org/10.1038/s41597-020-00630-y

[49]

Monica Perusquía-Hernández, Saho Ayabe-Kanamura, Kenji Suzuki, and Shiro Kumano. 2019. The Invisible Potential of Facial Electromyography: A Comparison of EMG and Computer Vision When Distinguishing Posed from Spontaneous Smiles. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--9. https://doi.org/10.1145/3290605.3300379

Digital Library

[50]

Amalia Petrovici and Tatiana Dobrescu. 2014. The role of emotional intelligence in building interpersonal communication skills. Procedia-Social and Behavioral Sciences 116 (2014), 1405--1410. https://doi.org/10.1016/j.sbspro.2014.01.406

[51]

Sarah Miriam Pritz. 2016. Making emotions count: the self-tracking of feelings. In Lifelogging. Springer, 179--187. https://doi.org/10.1007/978--3--658--13137--1_9

[52]

Nora Ptakauskaite, Anna L. Cox, and Nadia Berthouze. 2018. Knowing What You're Doing or Knowing What to Do: How Stress Management Apps Support Reflection and Behaviour Change. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA '18). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3170427.3188648

Digital Library

[53]

Juan Carlos Quiroz, Elena Geangu, and Min Hooi Yong. 2018. Emotion recognition using smart watch sensor data: Mixed-design study. JMIR mental health 5, 3 (2018), e10153. https://doi.org/10.2196/10153

[54]

Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. 2019. Deep PPG: large-scale heart rate estimation with convolutional neural networks. Sensors 19, 14 (2019), 3079. https://doi.org/10.3390/s19143079

[55]

Camilo Rojas, Eugenio Zuccarelli, Alexandra Chin, Gaurav Patekar, David Esquivel, and Pattie Maes. 2022. Towards Enhancing Empathy Through Emotion Augmented Remote Communicationnguyen2022moody. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 454, 9 pages. https://doi.org/10.1145/3491101.3519797

Digital Library

[56]

David C Rubin and Jennifer M Talarico. 2009. A comparison of dimensional models of emotion: Evidence from emotions, prototypical events, autobiographical memories, and words. Memory 17, 8 (2009), 802--808. https://doi.org/ 080/09658210903130764

[57]

Ofir Sadka and Alissa Antle. 2020. Interactive Technologies for Emotion-Regulation Training: Opportunities and Challenges. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3334480.3382894

Digital Library

[58]

Peter Salovey, Alexander J Rothman, Jerusha B Detweiler, and Wayne T Steward. 2000. Emotional states and physical health. American psychologist 55, 1 (2000), 110. https://doi.org/doi/10.1037/0003-066X.55.1.110

[59]

Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. 2018. Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (Boulder, CO, USA) (ICMI '18). Association for Computing Machinery, New York, NY, USA, 400--408. https://doi.org/10.1145/3242969.3242985

Digital Library

[60]

Björn W Schuller. 2018. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Commun. ACM 61, 5 (2018), 90--99. https://doi.org/10.1145/3129340

Digital Library

[61]

Nicola S Schutte, John M Malouff, and Navjot Bhullar. 2009. The assessing emotions scale. In Assessing emotional intelligence. Springer, 119--134. https://doi.org/10.1007/978-0--387--88370-0_7

[62]

Nicola S Schutte, John M Malouff, Lena E Hall, Donald J Haggerty, Joan T Cooper, Charles J Golden, and Liane Dornheim. 1998. Development and validation of a measure of emotional intelligence. Personality and individual differences 25, 2 (1998), 167--177. https://doi.org/10.1016/S0191--8869(98)00001--4

[63]

Sinem Semsioglu, Yagmur Gokce, and Asim Evren Yantac. 2018. Emotionscape: Mediating Spatial Experience for Emotion Awareness and Sharing. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA '18). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3170427.3188691

Digital Library

[64]

Karan Sharma, Claudio Castellini, Egon L van den Broek, Alin Albu-Schaeffer, and Friedhelm Schwenker. 2019. A dataset of continuous affect annotations and physiological signals for emotion analysis. Scientific data 6, 1 (2019), 1--13. https://doi.org/10.1038/s41597-019-0209-0

[65]

Lin Shu, Yang Yu, Wenzhuo Chen, Haoqiang Hua, Qin Li, Jianxiu Jin, and Xiangmin Xu. 2020. Wearable emotion recognition using heart rate data from a smart bracelet. Sensors 20, 3 (2020), 718. https://doi.org/10.3390/s20030718

[66]

Wally Smith, Greg Wadley, Sarah Webber, Benjamin Tag, Vassilis Kostakos, Peter Koval, and James J. Gross. 2022. Digital Emotion Regulation in Everyday Life. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 444, 15 pages. https://doi.org/10.1145/3491102.3517573

Digital Library

[67]

Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. 2018. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing 11, 3 (2018), 532--541. https://doi.org/10. 1109/TAFFC.2018.2817622

[68]

Hussel Suriyaarachchi, Paul Denny, and Suranga Nanayakkara. 2022. Scratch and Sense: Using Real-Time Sensor Data to Motivate Students Learning Scratch. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1 (Providence, RI, USA) (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 983--989. https://doi.org/10.1145/3478431.3499316

Digital Library

[69]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1--9. https://doi.org/10.1109/CVPR.2015.7298594

[70]

Alexander Toet and Jan B F van Erp. 2020. The EmojiGrid as a rating tool for the affective appraisal of touch. PLoS ONE 15 (2020). https://doi.org/10.1371/journal.pone.0237873

[71]

Rafael Wampfler, Severin Klingler, Barbara Solenthaler, Victor R. Schinazi, Markus Gross, and Christian Holz. 2022. Affective State Prediction from Smartphone Touch and Sensor Data in the Wild. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 403, 14 pages. https://doi.org/10.1145/3491102.3501835

Digital Library

[72]

Qiaosi Wang, Shan Jing, David Joyner, Lauren Wilcox, Hong Li, Thomas Plötz, and Betsy Disalvo. 2020. Sensing Affect to Empower Students: Learner Perspectives on Affect-Sensitive Technology in Large Educational Contexts. In Proceedings of the Seventh ACM Conference on Learning @ Scale (Virtual Event, USA) (L@S '20). Association for Computing Machinery, New York, NY, USA, 63--76. https://doi.org/10.1145/3386527.3405917

Digital Library

[73]

Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Understanding the Effect of Accuracy on Trust in Machine Learning Models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/ 3290605.3300509

Digital Library

[74]

Seunghyun Yoon, Seokhyun Byun, and Kyomin Jung. 2018. Multimodal speech emotion recognition using audio and text. In 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 112--118. https://doi.org/10.1109/SLT.2018.8639583

[75]

Tianyi Zhang, Abdallah El Ali, Chen Wang, Alan Hanjalic, and Pablo Cesar. 2021. CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors. Sensors 21, 1 (2021), 52. https://doi.org/10. 3390/s21010052

[76]

Xiao Zhang, Wenzhong Li, Xu Chen, and Sanglu Lu. 2018. MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 176 (jan 2018), 30 pages. https://doi.org/10.1145/3161414

Digital Library

[77]

Mingmin Zhao, Fadel Adib, and Dina Katabi. 2016. Emotion recognition using wireless signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. 95--108. https://doi.org/10.1145/2973750.2973762

Digital Library

Cited By

Montoya MQiao HSasikumar PElvitigala DPell SNanayakkara SMueller F(2024)Exploring an Extended Reality Floatation Tank Experience to Reduce the Fear of Being in WaterProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642285(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642285
Li FLuo JWang LLiu WSang X(2023)GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognitionFrontiers in Neuroscience10.3389/fnins.2023.118313217Online publication date: 4-May-2023
https://doi.org/10.3389/fnins.2023.1183132
Yu ZSherpa MIadarola SShamlian KDaley SLevine GBajorski PZheng Z(2023)Understanding Stress in Children with ASD and Their Caregivers in Daily Life: A Feasibility Study Using Mobile DevicesProceedings of the 25th International Conference on Mobile Human-Computer Interaction10.1145/3565066.3608686(1-7)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1145/3565066.3608686
Show More Cited By

Index Terms

Troi: Towards Understanding Users Perspectives to Mobile Automatic Emotion Recognition System in Their Natural Setting
1. Computing methodologies
  1. Machine learning
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
  2. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

Speech-based recognition of self-reported and observed emotion in a dimensional space

The differences between self-reported and observed emotion have only marginally been investigated in the context of speech-based automatic emotion recognition. We address this issue by comparing self-reported emotion ratings to observed emotion ratings ...
Mediating individual affective experience through the emotional photo frame
Affect Aware Ubiquitous Computing

A photograph is considered a medium with emotional legibility and a means of expressing and exchanging emotional experience. This research presents the interactive emotional photo frame system focusing on mediating individual affective experience among ...
Identifying Mobile Application Design to Enhance the Subjective Wellbeing among Middle-Aged Adults
Proceedings of the 8th International Conference on Universal Access in Human-Computer Interaction. Aging and Assistive Environments - Volume 8515

Faced with life stress and peer competition, middle-aged adults increasingly are lacking happiness and well-being. The address this problem, this research studied current mobile applications for wellbeing and perceptions of Subjective Wellbeing SWB ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue MHCI

MHCI

September 2022

852 pages

EISSN:2573-0142

DOI:10.1145/3564624

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 September 2022

Published in PACMHCI Volume 6, Issue MHCI

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Tertiary Education Commission New Zealand

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
324
Total Downloads

Downloads (Last 12 months)91
Downloads (Last 6 weeks)5

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Montoya MQiao HSasikumar PElvitigala DPell SNanayakkara SMueller F(2024)Exploring an Extended Reality Floatation Tank Experience to Reduce the Fear of Being in WaterProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642285(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642285
Li FLuo JWang LLiu WSang X(2023)GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognitionFrontiers in Neuroscience10.3389/fnins.2023.118313217Online publication date: 4-May-2023
https://doi.org/10.3389/fnins.2023.1183132
Yu ZSherpa MIadarola SShamlian KDaley SLevine GBajorski PZheng Z(2023)Understanding Stress in Children with ASD and Their Caregivers in Daily Life: A Feasibility Study Using Mobile DevicesProceedings of the 25th International Conference on Mobile Human-Computer Interaction10.1145/3565066.3608686(1-7)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1145/3565066.3608686
Li FLuo JLiu W(2023)Speech Emotion Recognition Using Multi-Modal Feature Fusion Network2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI)10.1109/PRAI59366.2023.10332053(884-888)Online publication date: 18-Aug-2023
https://doi.org/10.1109/PRAI59366.2023.10332053
Klimek AMannheim ISchouten GWouters EPeeters M(2023)Wearables measuring electrodermal activity to assess perceived stress in care: a scoping reviewActa Neuropsychiatrica10.1017/neu.2023.19(1-11)Online publication date: 24-Mar-2023
https://doi.org/10.1017/neu.2023.19
Li FLuo J(2023)Speech Emotion Recognition Using Global-Aware Cross-Modal Feature Fusion NetworkAdvanced Intelligent Computing Technology and Applications10.1007/978-981-99-4742-3_17(211-221)Online publication date: 10-Aug-2023
https://dl.acm.org/doi/10.1007/978-981-99-4742-3_17

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents