NapTune: Efficient Model Tuning for Mood Classification using Previous Night's Sleep Measures along with Wearable Time-series
Pages 204 - 213
Abstract
Sleep is known to be a key factor in emotional regulation and overall mental health. In this study, we explore the integration of sleep measures from the previous night into wearable-based mood recognition. To this end, we propose NapTune, a novel prompt-tuning framework that utilizes sleep-related measures as additional inputs to a frozen pre-trained wearable time-series encoder by adding and training lightweight prompt parameters to each Transformer layer. Through rigorous empirical evaluation, we demonstrate that the inclusion of sleep data using NapTune not only improves mood recognition performance across different wearable time-series namely ECG, PPG, and EDA, but also makes it more sample-efficient. Our method demonstrates significant improvements over the best baselines and unimodal variants. Furthermore, we analyze the impact of adding sleep-related measures on recognizing different moods as well as the influence of individual sleep-related measures.
References
[1]
Syed Khairul Bashar, Eric Ding, Allan J Walkey, David D McManus, and Ki H Chon. 2019. Noise detection in Electrocardiogram signals for intensive care unit patients. IEEE Access 7 (2019), 88357–88368.
[2]
Behnam Behinaein, Anubhav Bhatti, Dirk Rodenburg, Paul Hungler, and Ali Etemad. 2021. A Transformer Architecture for Stress Detection From ECG. In Proceedings of the ACM International Symposium on Wearable Computers. 132–134.
[3]
Scott A Cairney, Simon J Durrant, Johan Hulleman, and Penelope A Lewis. 2014. Targeted memory reactivation during slow wave sleep facilitates emotional memory consolidation. Sleep 37, 4 (2014), 701–707.
[4]
Scott A Cairney, Simon J Durrant, Rebecca Power, and Penelope A Lewis. 2015. Complementary roles of slow-wave sleep and rapid eye movement sleep in emotional memory consolidation. Cerebral Cortex 25, 6 (2015), 1565–1575.
[5]
Rafael A Calvo and Sidney D’Mello. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing 1, 1 (2010), 18–37.
[6]
Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. 2023. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948 (2023).
[7]
Chiara EG Castiglione-Fontanellaz and Leila Tarokh. 2023. Sleep and Adolescent Depression. Clinical and Translational Neuroscience 8, 1 (2023), 3.
[8]
Nicola Cellini, Katherine A Duggan, and Michela Sarlo. 2017. Perceived sleep quality: The interplay of neuroticism, affect, and hyperarousal. Sleep Health 3, 3 (2017), 184–189.
[9]
Kai-Wei Chang, Yu-Kai Wang, Hua Shen, Iu-thing Kang, Wei-Cheng Tseng, Shang-Wen Li, and Hung-yi Lee. 2023. Speechprompt v2: Prompt tuning for speech classification tasks. arXiv preprint arXiv:2303.00733 (2023).
[10]
Jianzhong Chen, Ke Sun, Yi Sun, and Xinxin Li. 2021. Signal quality assessment of PPG signals using STFT time-frequency spectra and deep learning approaches. In 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. IEEE, 1153–1156.
[11]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning. PMLR, 1597–1607.
[12]
Anja Exler, Andrea Schankin, Christoph Klebsattel, and Michael Beigl. 2016. A wearable system for mood assessment considering smartphone features and data from mobile ECGs. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 1153–1161.
[13]
Hamid Fekri Azgomi, Luciano R F. Branco, Md Rafiul Amin, Saman Khazaei, and Rose T Faghih. 2023. Regulation of Brain Cognitive States Through Auditory, Gustatory, and Olfactory Stimulation With Wearable Monitoring. Scientific reports 13, 1 (2023), 12399.
[14]
Zhilin Gao, Xingran Cui, Wang Wan, Wenming Zheng, and Zhongze Gu. 2021. ECSMP: A dataset on emotion, cognition, sleep, and multi-model physiological signals. Data in Brief 39 (2021), 107660.
[15]
Lisa Genzel, VI Spoormaker, BN Konrad, and M Dresler. 2015. The role of rapid eye movement sleep for amygdala-related memory processing. Neurobiology of learning and memory 122 (2015), 110–121.
[16]
Andrea N Goldstein and Matthew P Walker. 2014. The role of sleep in emotional brain function. Annual review of clinical psychology 10 (2014), 679–708.
[17]
Muhammad Anas Hasnul, Nor Azlina Ab Aziz, Salem Alelyani, Mohamed Mohana, and Azlan Abd Aziz. 2021. Electrocardiogram-based emotion recognition systems and their applications in healthcare—a review. Sensors 21, 15 (2021), 5015.
[18]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on cComputer Vision and Pattern Recognition. 770–778.
[19]
Shenda Hong, Yanbo Xu, Alind Khare, Satria Priambada, Kevin Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov. 2020. Holmes: health online model ensemble serving for deep learning models in intensive care units. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1614–1624.
[20]
Isabel C Hutchison and Shailendra Rathore. 2015. The role of REM sleep theta activity in emotional memory. Frontiers in Psychology 6 (2015), 154468.
[21]
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In European Conference on Computer Vision. Springer, 709–727.
[22]
Wenxiang Jiao, Michael Lyu, and Irwin King. 2020. Real-time emotion recognition via attention gated hierarchical memory network. In Proceedings of the AAAI conference on Artificial Intelligence, Vol. 34. 8002–8009.
[23]
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data 3, 1 (2016), 1–9.
[24]
David A Kalmbach, Vivek Pillai, Thomas Roth, and Christopher L Drake. 2014. The interplay between daily affect and sleep: A 2-week study of young women. Journal of Sleep Research 23, 6 (2014), 636–645.
[25]
Ashish Kapoor and Rosalind W Picard. 2005. Multimodal affect recognition in learning environments. In Proceedings of the 13th Annual ACM International Conference on Multimedia. 677–682.
[26]
Walter Karlen. 2021. CapnoBase IEEE TBME respiratory rate benchmark.
[27]
Harshit Kaushik, Tarun Kumar, and Kriti Bhalla. 2022. iSecureHome: A deep fusion framework for surveillance of smart homes using real-time emotion recognition. Applied Soft Computing 122 (2022), 108788.
[28]
Saskia Koldijk, Maya Sappelli, Suzan Verberne, Mark A Neerincx, and Wessel Kraaij. 2014. The swell knowledge work dataset for stress and user modeling research. In Proceedings of the 16th International Conference on Multimodal Interaction. 291–298.
[29]
Monika Konjarski, Greg Murray, V Vien Lee, and Melinda L Jackson. 2018. Reciprocal relationships between daily sleep and mood: A systematic review of naturalistic prospective studies. Sleep Medicine Reviews 42 (2018), 47–58.
[30]
Zlatan Krizan, Nicholas A Boehm, and Caroline B Strauel. 2023. How emotions impact sleep: A quantitative review of experiments. Sleep Medicine Reviews (2023), 101890.
[31]
Min Seop Lee, Yun Kyu Lee, Dong Sung Pae, Myo Taeg Lim, Dong Won Kim, and Tae Koo Kang. 2019. Fast emotion recognition based on single pulse PPG signal with convolutional neural network. Applied Sciences 9, 16 (2019), 3355.
[32]
Ruth LF Leong and Michael WL Chee. 2023. Understanding the need for sleep to improve cognition. Annual Review of Psychology 74 (2023), 27–57.
[33]
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 3045–3059.
[34]
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 (2021).
[35]
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. 2021. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods 53, 4 (2021), 1689–1696. https://doi.org/10.3758/s13428-020-01516-y
[36]
Peter Mantello and Manh-Tung Ho. 2023. Emotional AI and the future of wellbeing in the post-pandemic workplace. AI & Society (2023), 1–7.
[37]
Juan Abdon Miranda-Correa, Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras. 2018. Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Transactions on Affective Computing 12, 2 (2018), 479–493.
[38]
Jiapu Pan and Willis J Tompkins. 1985. A real-time QRS detection algorithm. IEEE Transactions on Biomedical Engineering3 (1985), 230–236.
[39]
Marco AF Pimentel, Alistair EW Johnson, Peter H Charlton, Drew Birrenkott, Peter J Watkinson, Lionel Tarassenko, and David A Clifton. 2016. Toward a robust estimation of respiratory rate from pulse oximeters. IEEE Transactions on Biomedical Engineering 64, 8 (2016), 1914–1923.
[40]
Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. 2019. Deep PPG: Large-scale heart rate estimation with convolutional neural networks. Sensors 19, 14 (2019), 3079.
[41]
Kyle Ross, Paul Hungler, and Ali Etemad. 2023. Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data. Journal of Ambient Intelligence and Humanized Computing 14, 4 (2023), 3199–3224.
[42]
Akane Sano, Z Yu Amy, Andrew W McHill, Andrew JK Phillips, Sara Taylor, Natasha Jaques, Elizabeth B Klerman, and Rosalind W Picard. 2015. Prediction of happy-sad mood from daily behaviors and previous sleep history. In 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 6796–6799.
[43]
Pritam Sarkar and Ali Etemad. 2020. Self-Supervised Learning for ECG-Based Emotion Recognition. In IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE, 3217–3221.
[44]
Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. 2018. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 400–408.
[45]
Philip Schmidt, Attila Reiss, Robert Dürichen, and Kristof Van Laerhoven. 2019. Wearable-based affect recognition—A review. Sensors 19, 19 (2019), 4079.
[46]
Jainendra Shukla, Miguel Barreda-Angeles, Joan Oliver, Gora Chand Nandi, and Domenec Puig. 2019. Feature extraction and selection for emotion recognition from electrodermal activity. IEEE Transactions on Affective Computing 12, 4 (2019), 857–869.
[47]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[48]
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems 33 (2020), 596–608.
[49]
Sahar Soltanieh, Javad Hashemi, and Ali Etemad. 2023. In-distribution and out-of-distribution self-supervised ecg representation learning for arrhythmia detection. IEEE Journal of Biomedical and Health Informatics (2023).
[50]
Sara Taylor, Natasha Jaques, Ehimwenma Nosakhare, Akane Sano, and Rosalind Picard. 2017. Personalized multitask learning for predicting tomorrow’s mood, stress, and health. IEEE Transactions on Affective Computing 11, 2 (2017), 200–213.
[51]
Maia Ten Brink, Jessica R Dietch, Joshua Tutek, Sooyeon A Suh, James J Gross, and Rachel Manber. 2022. Sleep and affect: A conceptual review. Sleep Medicine Reviews (2022), 101670.
[52]
Robert Joseph Thomas, Joseph E Mietus, Chung-Kang Peng, and Ary L Goldberger. 2005. An electrocardiogram-based technique to assess cardiopulmonary coupling during sleep. Sleep 28, 9 (2005), 1151–1161.
[53]
Giuseppe Romano Tizzano, Matteo Spezialetti, and Silvia Rossi. 2020. A deep learning approach for mood recognition from wearable data. In IEEE International Symposium on Medical Measurements and Applications. IEEE, 1–5.
[54]
Matthew P Walker and Els van Der Helm. 2009. Overnight therapy? The role of sleep in emotional brain processing.Psychological bulletin 135, 5 (2009), 731.
[55]
Junxiang Wang, Guangji Bai, Wei Cheng, Zhengzhang Chen, Liang Zhao, and Haifeng Chen. 2023. Prompt-based Domain Discrimination for Multi-source Time Series Domain Adaptation. arXiv preprint arXiv:2312.12276 (2023).
[56]
Claire Whiting, Nellia Bellaert, Christen Deveney, and Wan-Ling Tseng. 2023. Associations between sleep quality and irritability: testing the mediating role of emotion regulation. Personality and Individual Differences 213 (2023), 112322.
[57]
Christian D Wiesner, Julika Pulst, Fanny Krause, Marike Elsner, Lioba Baving, Anya Pedersen, Alexander Prehn-Kristensen, and Robert Göder. 2015. The effect of selective REM-sleep deprivation on the consolidation and affective evaluation of emotional memories. Neurobiology of Learning and Memory 122 (2015), 131–141.
[58]
Mark Lawrence Wong, Esther Yuet Ying Lau, Jacky Ho Yin Wan, Shu Fai Cheung, C Harry Hui, and Doris Shui Ying Mok. 2013. The interplay between sleep and mood in predicting academic functioning, physical health and psychological health: A longitudinal study. Journal of Psychosomatic Research 74, 4 (2013), 271–277.
[59]
Hao Xue and Flora D Salim. 2023. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering (2023).
[60]
Alexandros Zenonos, Aftab Khan, Georgios Kalogridis, Stefanos Vatsikas, Tim Lewis, and Mahesh Sooriyabandara. 2016. HealthyOffice: Mood recognition at work using smartphones and wearable sensors. In IEEE International Conference on Pervasive Computing and Communication Workshops. IEEE, 1–6.
[61]
Nan Zhang, Simeng Ma, Peilin Wang, Lihua Yao, Lijun Kang, Wei Wang, Zhaowen Nie, Mianmian Chen, Ci Ma, and Zhongchun Liu. 2023. Psychosocial factors of insomnia in depression: a network approach. BMC Psychiatry 23, 1 (2023), 949.
[62]
Lili Zhu, Petros Spachos, and Stefano Gregori. 2023. Electrodermal Activity for Emotion Recognition Using CNN and Bi-GRU Model. In IEEE International Conference on Communications. IEEE, 5533–5538.
[63]
Zack Zhu, Hector F Satizabal, Ulf Blanke, Andres Perez-Uribe, and Gerhard Tröster. 2015. Naturalistic recognition of activities and mood using wearable electronics. IEEE Transactions on Affective Computing 7, 3 (2015), 272–285.
[64]
Martin Zihlmann, Dmytro Perekrestenko, and Michael Tschannen. 2017. Convolutional recurrent neural networks for electrocardiogram classification. In Computing in Cardiology. IEEE, 1–4.
Index Terms
- NapTune: Efficient Model Tuning for Mood Classification using Previous Night's Sleep Measures along with Wearable Time-series
Index terms have been assigned to the content through auto-classification.
Recommendations
Effect of CPAP therapy on daytime cardiovascular regulations in patients with obstructive sleep apnea
Obstructive sleep apnea (OSA) is a sleep disorder with a high prevalence that causes pathological changes in cardiovascular regulation during the night and also during daytime. We investigated whether the treatment of OSA at night by means of continuous ...
Mood recognition in bipolar patients through the PSYCHE platform: Preliminary evaluations and perspectives
Background: Bipolar disorders are characterized by a series of both depressive and manic or hypomanic episodes. Although common and expensive to treat, the clinical assessment of bipolar disorder is still ill-defined. Objective: In the current ...
Time-frequency analysis of heart rate variability for sleep and wake classification
BIBE '12: Proceedings of the 2012 IEEE 12th International Conference on Bioinformatics & Bioengineering (BIBE)This paper describes a method to adapt the spectral features extracted from heart rate variability (HRV) for sleep and wake classification. HRV series can be derived from electrocardiogram (ECG) signals obtained from single-night polysomnography (PSG) ...
Comments
Information & Contributors
Information
Published In
November 2024
725 pages
ISBN:9798400704628
DOI:10.1145/3678957
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICMI '24
ICMI '24: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
November 4 - 8, 2024
San Jose, Costa Rica
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 61Total Downloads
- Downloads (Last 12 months)61
- Downloads (Last 6 weeks)13
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format