research-article

MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing

Authors:

Sanglu LuAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 4

Article No.: 176, Pages 1 - 30

https://doi.org/10.1145/3161414

Published: 08 January 2018 Publication History

Abstract

Social psychology and neuroscience had confirmed that emotion state exerts a significant effect on human communication, perception, social behavior and decision making. With the wide availability of smartphones equipped with microphone, accelerometer, GPS, and other source of sensors, it is worthwhile to explore the possibility of automatic emotion detection via smartphone sensing. Particularly, we focus on a novel research problem that tries to detect the compound emotion (a set of multiple dimensional basic emotions) of smartphone users. We observe that users' self-reported emotional states have high correlation with their smartphone usage patterns and sensing data. Based on the observations, we exploit a feature extraction and selection algorithm to find the most significant features. We further adopt a factor graph model to tackle the correlations between features and emotion labels, and propose a machine learning algorithm for compound emotion detection based on the smartphone sensing data. The proposed mechanism is implemented as an APP called MoodExplorer in Android platform. Extensive experiments conducted on the smartphone data collected from 30 university students show that MoodExplorer can recognize users' compound emotions with 76.0% exact match on average.

Supplementary Material

zhang (zhang.zip)

Supplemental movie, appendix, image and software files for, MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing

Download
353.52 KB

References

[1]

J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In INTERSPEECH. Citeseer, 2002.

[2]

S. Aral and D. Walker. Identifying influential and susceptible members of social networks. Science, 337(6092):337--341, 2012.

[3]

A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon. The painful face--pain expression recognition using active appearance models. Image and vision computing, 27(12):1788--1796, 2009.

Digital Library

[4]

X. Bao, S. Fan, A. Varshavsky, K. Li, and R. Roy Choudhury. Your reactions suggest you liked the movie: Automatic content rating via reaction sensing. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 197--206. ACM, 2013.

Digital Library

[5]

C. Beedie, P. Terry, and A. Lane. Distinctions between emotion and mood. Cognition 8 Emotion, 19(6):847--878, 2005.

[6]

A. Bogomolov, B. Lepri, M. Ferron, F. Pianesi, and A. S. Pentland. Daily stress recognition from mobile phone data, weather conditions and individual traits. In Proceedings of the 22nd ACM international conference on Multimedia, pages 477--486. ACM, 2014.

Digital Library

[7]

L. Canzian and M. Musolesi. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing, pages 1293--1304. ACM, 2015.

Digital Library

[8]

B. Cao, L. Zheng, C. Zhang, P. S. Yu, A. Piscitello, J. Zulueta, O. Ajilore, K. Ryan, and A. D. Leow. Deepmood: Modeling mobile phone typing dynamics for mood detection. 2017.

[9]

J. R. Crawford and J. D. Henry. The positive and negative affect schedule (panas): Construct validity, measurement properties and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 43(3):245--265, 2004.

[10]

E. Diener, S. Oishi, and R. E. Lucas. Personality, culture, and subjective well-being: Emotional and cognitive evaluations of life. Annual review of psychology, 54(1):403--425, 2003.

[11]

S. Du, Y. Tao, and A. M. Martinez. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15):E1454--E1462, 2014.

[12]

P. Ekman, W. V. Friesen, M. O'Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W. A. LeCompte, T. Pitcairn, P. E. Ricci-Bitti, et al. Universals and cultural differences in the judgments of facial expressions of emotion. Journal of personality and social psychology, 53(4):712, 1987.

[13]

A. Exler, M. Urschel, A. Schankin, and M. Beigl. Smartphone-based detection of location changes using wifi data. In International Conference on Wireless Mobile Communication and Healthcare, pages 164--167. Springer, 2016.

[14]

J. H. Fowler, N. A. Christakis, et al. Dynamic spread of happiness in a large social network: longitudinal analysis over 20 years in the framingham heart study. British Medical Journal, 337:a2338, 2008.

[15]

B. J. Frey and D. J. MacKay. A revolution: Belief propagation in graphs with cycles. Advances in neural information processing systems, pages 479--485, 1998.

[16]

T. Geller. How do you feel?: Your computer knows. Communications of the ACM, 57(1):24--26, 2014.

Digital Library

[17]

A. Gluhak, M. Presser, L. Zhu, S. Esfandiyari, and S. Kupschick. Towards mood based mobile services and applications. In European Conference on Smart Sensing and Context, pages 159--174. Springer, 2007.

[18]

J. Golbeck, C. Robles, and K. Turner. Predicting personality with social media. In CHI‘11 extended abstracts on human factors in computing systems, pages 253--262. ACM, 2011.

Digital Library

[19]

J. J. Gross and O. P. John. Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. Journal of personality and social psychology, 85(2):348, 2003.

[20]

J. M. Hammersley and P. Clifford. Markov fields on finite graphs and lattices. 1971.

[21]

K. C. Herdem. Reactions: Twitter based mobile application for awareness of friends' emotions. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 796--797. ACM, 2012.

Digital Library

[22]

J. Hernandez, M. E. Hoque, W. Drevo, and R. W. Picard. Mood meter: counting smiles in the wild. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 301--310. ACM, 2012.

Digital Library

[23]

D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.

[24]

F. R. Kschischang, B. J. Frey, and H.-A. Loeliger. Factor graphs and the sum-product algorithm. IEEE Transactions on information theory, 47(2):498--519, 2001.

Digital Library

[25]

J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the eighteenth international conference on machine learning (ICML ‘01), volume 1, pages 282--289, 2001.

[26]

H. Leng, Y. Lin, and L. Zanzi. An experimental study on physiological parameters toward driver emotion recognition. In International Conference on Ergonomics and Health Aspects of Work with Computers, pages 237--246. Springer, 2007.

[27]

M. Levandowsky and D. Winter. Distance between sets. Nature, 234(5323):34--35, 1971.

[28]

S. Li, L. Huang, R. Wang, and G. Zhou. Sentence-level emotion classification with label and context dependence. Proceedings of ACL-2015, pages 1045--1053, 2013.

[29]

W. Li, Y. Hu, X. Fu, S. Lu, and D. Chen. Cooperative positioning and tracking in disruption tolerant networks. IEEE Transactions on Parallel and Distributed Systems, 26(2):382--391, 2015.

[30]

R. LiKamWa, Y. Liu, N. D. Lane, and L. Zhong. Moodscope: building a mood sensor from smartphone usage patterns. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services, pages 389--402. ACM, 2013.

[31]

G. MacKerron and S. Mourato. Happiness is greater in natural environments. Global Environmental Change, 23(5):992--1000, 2013.

[32]

A. Mottelson and K. Hornbæk. An affect detection technique using mobile commodity sensors in the wild. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 781--792. ACM, 2016.

Digital Library

[33]

K. P. Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.

[34]

R. Plutchik. A general psychoevolutionary theory of emotion. Theories of emotion, 1(3-31):4, 1980.

[35]

J. Posner, J. A. Russell, and B. S. Peterson. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, 17(03):715--734, 2005.

[36]

R. Reisenzein, M. Studtmann, and G. Horstmann. Coherence between emotion and facial expression: Evidence from laboratory experiments. Emotion Review, 5(1):16--23, 2013.

[37]

S. Servia-Rodríguez, K. K. Rachuri, C. Mascolo, P. J. Rentfrow, N. Lathia, and G. M. Sandstrom. Mobile sensing at the service of mental well-being: a large-scale longitudinal study. In Proceedings of the 26th International Conference on World Wide Web, pages 103--112. International World Wide Web Conferences Steering Committee, 2017.

Digital Library

[38]

N. SpolaôR, E. A. Cherman, M. C. Monard, and H. D. Lee. A comparison of multi-label feature selection methods using the problem transformation approach. Electronic Notes in Theoretical Computer Science, 292:135--151, 2013.

Digital Library

[39]

N. Spolaôr, E. A. Cherman, M. C. Monard, and H. D. Lee. Relieff for multi-label feature selection. In Intelligent Systems (BRACIS), 2013 Brazilian Conference on, pages 6--11. IEEE, 2013.

Digital Library

[40]

S. A. Stansfeld and M. P. Matheson. Noise pollution: non-auditory effects on health. British medical bulletin, 68(1):243--257, 2003.

[41]

T. Stütz, T. Kowar, M. Kager, M. Tiefengrabner, M. Stuppner, J. Blechert, F. H. Wilhelm, and S. Ginzinger. Smartphone based stress prediction. In International Conference on User Modeling, Adaptation, and Personalization, pages 240--251. Springer, 2015.

[42]

Y. Suhara, Y. Xu, and A. Pentland. Deepmood: Forecasting depressed mood based on self-reported histories via recurrent neural networks. In Proceedings of the 26th International Conference on World Wide Web, pages 715--724. International World Wide Web Conferences Steering Committee, 2017.

Digital Library

[43]

B. Sun, Q. Ma, S. Zhang, K. Liu, and Y. Liu. iself: Towards cold-start emotion labeling using transfer learning with smartphones. In 2015 IEEE Conference on Computer Communications (INFOCOM), pages 1203--1211. IEEE, 2015.

[44]

C. Sutton and A. McCallum. An introduction to conditional random fields. arXiv preprint arXiv:1011.4088, 2010.

[45]

S. S. Tomkins. Affect, imagery, consciousness: Vol. i. the positive affects. 1962.

[46]

G. Tsoumakas, I. Katakis, and I. Vlahavas. Mining multi-label data. In Data mining and knowledge discovery handbook, pages 667--685. Springer, 2009.

[47]

L. Wasserman. All of statistics: a concise course in statistical inference. Springer Science 8 Business Media, 2013.

[48]

Y. Yang, J. Jia, B. Wu, and J. Tang. Social role-aware emotion contagion in image social networks. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.

Digital Library

[49]

Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence, 31(1):39--58, 2009.

Digital Library

[50]

M.-L. Zhang and Z.-H. Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering, 26(8):1819--1837, 2014.

[51]

S. Zhang and P. Hui. A survey on mobile affective computing. ArXiv Prepr. ArXiv14101648, 2014.

[52]

Y. Zhang, J. Tang, J. Sun, Y. Chen, and J. Rao. Moodcast: Emotion prediction via dynamic continuous factor graph model. In Proceedings of the 10th International Conference on Data Mining, pages 1193--1198. IEEE, 2010.

Digital Library

[53]

M. Zhao, F. Adib, and D. Katabi. Emotion recognition using wireless signals. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (MobiCom ‘16), pages 95--108, 2016.

Digital Library

[54]

S. Zhao, H. Yao, and X. Jiang. Predicting continuous probability distribution of image emotions in valence-arousal space. In Proceedings of the 23rd ACM international conference on Multimedia, pages 879--882. ACM, 2015.

Digital Library

[55]

Y. Zhou, H. Xue, and X. Geng. Emotion distribution recognition from facial expressions. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1247--1250. ACM, 2015.

Digital Library

Cited By

Abdrabou YOmelina TDietz FKhamis MAlt FHassib M(2024)Where Do You Look When Unlocking Your Phone? : A Field Study of Gaze Behaviour During Smartphone UnlockExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651094(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651094
Oyebode OSteeves DOrji R(2024)Persuasive strategies and emotional states: towards designing personalized and emotion-adaptive persuasive systemsUser Modeling and User-Adapted Interaction10.1007/s11257-023-09390-xOnline publication date: 5-Mar-2024
https://doi.org/10.1007/s11257-023-09390-x
Li YLi FYang SZhang CLi YLi FYang SZhang C(2024)A Brief IntroductionIncentive Mechanism for Mobile Crowdsensing10.1007/978-981-99-6921-0_1(1-8)Online publication date: 4-Jan-2024
https://doi.org/10.1007/978-981-99-6921-0_1
Show More Cited By

Index Terms

MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools
    2. Ubiquitous and mobile devices
      1. Smartphones

Recommendations

Emotion Detection through Smartphone's Accelerometer and Gyroscope Sensors
UMAP '21: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization

Emotion recognition is essential for assessing human emotional states and predicting user behavior to provide appropriate and personalized feedback. The wide range of Smartphones with accelerometers, microphones, GPSs, gyroscopes, and more motivate ...
TapSense: combining self-report patterns and typing characteristics for smartphone based emotion detection
MobileHCI '17: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services

Typing based communication applications on smartphones, like WhatsApp, can induce emotional exchanges. The effects of an emotion in one session of communication can persist across sessions. In this work, we attempt automatic emotion detection by jointly ...
EmoMix: Building an Emotion Lexicon for Compound Emotion Analysis
Computational Science – ICCS 2019
Abstract
Building a high-quality emotion lexicon is regarded as the foundation of research on emotion analysis. Existing methods have focused on the study of primary categories (i.e., anger, disgust, fear, happiness, sadness, and surprise). However, there ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 4

December 2017

1298 pages

EISSN:2474-9567

DOI:10.1145/3178157

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2018

Accepted: 01 October 2017

Revised: 01 August 2017

Received: 01 May 2017

Published in IMWUT Volume 1, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

72
Total Citations
View Citations
1,653
Total Downloads

Downloads (Last 12 months)165
Downloads (Last 6 weeks)15

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Abdrabou YOmelina TDietz FKhamis MAlt FHassib M(2024)Where Do You Look When Unlocking Your Phone? : A Field Study of Gaze Behaviour During Smartphone UnlockExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651094(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651094
Oyebode OSteeves DOrji R(2024)Persuasive strategies and emotional states: towards designing personalized and emotion-adaptive persuasive systemsUser Modeling and User-Adapted Interaction10.1007/s11257-023-09390-xOnline publication date: 5-Mar-2024
https://doi.org/10.1007/s11257-023-09390-x
Li YLi FYang SZhang CLi YLi FYang SZhang C(2024)A Brief IntroductionIncentive Mechanism for Mobile Crowdsensing10.1007/978-981-99-6921-0_1(1-8)Online publication date: 4-Jan-2024
https://doi.org/10.1007/978-981-99-6921-0_1
Jianhua TJiangtao GNan GSiwei FShan LChun Y(2023)Human-computer interaction for virtual-real fusionJournal of Image and Graphics10.11834/jig.23002028:6(1513-1542)Online publication date: 2023
https://doi.org/10.11834/jig.230020
Deng ZHuang LWang C(2023)Enhanced In-air Signature Verification via Hand Skeleton Tracking to Defeat Robot-level ReplaysProceedings of the 39th Annual Computer Security Applications Conference10.1145/3627106.3627195(451-462)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3627106.3627195
Murali SJin WSivaraman VZhu HJi TLi PLi M(2023)Continuous Authentication Using Human-Induced Electric PotentialProceedings of the 39th Annual Computer Security Applications Conference10.1145/3627106.3627124(409-423)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3627106.3627124
Bhattacharjee APang JLiu AMariakakis AWilliams J(2023)Design Implications for One-Way Text Messaging Services that Support Psychological WellbeingACM Transactions on Computer-Human Interaction10.1145/356988830:3(1-29)Online publication date: 10-Jun-2023
https://dl.acm.org/doi/10.1145/3569888
Yang KZhang ZYouliang TMa J(2023)A Secure Authentication Framework to Guarantee the Traceability of Avatars in MetaverseIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.328868918(3817-3832)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIFS.2023.3288689
Syu JLin JSrivastava GYu K(2023)A Comprehensive Survey on Artificial Intelligence Empowered Edge Computing on Consumer ElectronicsIEEE Transactions on Consumer Electronics10.1109/TCE.2023.331815069:4(1023-1034)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TCE.2023.3318150
Yang KTag BWang CGu YSarsenbayeva ZDingler TWadley GGoncalves J(2023)Survey on Emotion Sensing Using Mobile DevicesIEEE Transactions on Affective Computing10.1109/TAFFC.2022.322048414:4(2678-2696)Online publication date: 1-Oct-2023
https://doi.org/10.1109/TAFFC.2022.3220484
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents