Online Multi-level Contrastive Representation Distillation for Cross-Subject fNIRS Emotion Recognition
Pages 29 - 37
Abstract
Utilizing functional near-infrared spectroscopy (fNIRS) signals for emotion recognition is a significant advancement in understanding human emotions. However, due to the lack of artificial intelligence data and algorithms in this field, current research faces the following challenges: 1) The portable wearable devices have higher requirements for lightweight models; 2) The objective differences of physiology and psychology among different subjects aggravate the difficulty of emotion recognition. To address these challenges, we propose a novel cross-subject fNIRS emotion recognition method, called the Online Multi-level Contrastive Representation Distillation framework (OMCRD). Specifically, OMCRD is a framework designed for mutual learning among multiple lightweight student networks. It utilizes multi-level fNIRS feature extractor for each sub-network and conducts multi-view sentimental mining using physiological signals. The proposed Inter-Subject Interaction Contrastive Representation (IS-ICR) facilitates knowledge transfer for interactions between student models, enhancing cross-subject emotion recognition performance. The optimal student network can be selected and deployed on a wearable device. Some experimental results demonstrate that OMCRD achieves state-of-the-art results in emotional perception and affective imagery tasks. The source code is available at https://github.com/Lzhili/fNIRS-OMCRD.
References
[1]
Roddy Cowie, Ellen Douglas-Cowie, Nicolas Tsapatsoulis, George Votsis, Stefanos Kollias, Winfried Fellenz, and John G Taylor. 2001. Emotion recognition in human-computer interaction. IEEE Signal processing magazine, Vol. 18, 1 (2001), 32--80.
[2]
Bea Waelbers, Stefano Bromuri, and Alexander P Henkel. 2022. Comparing neural networks for speech emotion recognition in customer service interactions. In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.
[3]
Deuger Ayata, Yusuf Yaslan, and Mustafa E Kamasak. 2020. Emotion recognition from multimodal physiological signals for emotion aware healthcare systems. Journal of Medical and Biological Engineering, Vol. 40 (2020), 149--157.
[4]
Maochun Huang, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. 2023. Context-Based Adaptive Multimodal Fusion Network for Continuous Frame-Level Sentiment Prediction. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).
[5]
Ruihan Chen, Junpeng Tan, Zhijing Yang, Xiaojun Yang, Qingyun Dai, Yongqiang Cheng, and Liang Lin. 2024. DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization. IEEE Transactions on Multimedia (2024).
[6]
Dongri Yang, Abeer Alsadoon, PW Chandana Prasad, Ashutosh Kumar Singh, and Amr Elchouemi. 2018. An emotion recognition model based on facial recognition in virtual learning environment. Procedia Computer Science, Vol. 125 (2018), 2--10.
[7]
Jinglun Cen, Chunmei Qing, Haochun Ou, Xiangmin Xu, and Junpeng Tan. 2024. MASANet: Multi-Aspect Semantic Auxiliary Network for Visual Sentiment Analysis. IEEE Transactions on Affective Computing (2024).
[8]
Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Junaid Qadir, and Björn Schuller. 2021. Survey of deep representation learning for speech emotion recognition. IEEE Transactions on Affective Computing, Vol. 14, 2 (2021), 1634--1654.
[9]
Peiliang Gong, Ziyu Jia, Pengpai Wang, Yueying Zhou, and Daoqiang Zhang. 2023. ASTDF-Net: Attention-Based Spatial-Temporal Dual-Stream Fusion Network for EEG-Based Emotion Recognition. In Proceedings of the 31st ACM International Conference on Multimedia. 883--892.
[10]
Youngchul Kwak, Woo-Jin Song, and Seong-Eun Kim. 2022. FGANet: fNIRS-guided attention network for hybrid EEG-fNIRS brain-computer interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 30 (2022), 329--339.
[11]
Manasa Kalanadhabhatta, Adrelys Mateo Santana, Deepak Ganesan, Tauhidur Rahman, and Adam Grabell. 2022. Extracting Multimodal Embeddings via Supervised Contrastive Learning for Psychological Screening. In 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1--8.
[12]
Stefano Piana, Alessandra Staglianò, Francesca Odone, and Antonio Camurri. 2016. Adaptive body gesture representation for automatic emotion recognition. ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 6, 1 (2016), 1--31.
[13]
Shuaiqi Liu, Yingying Zhao, Yanling An, Jie Zhao, Shui-Hua Wang, and Jingwen Yan. 2023. GLFANet: A global to local feature aggregation network for EEG emotion recognition. Biomedical Signal Processing and Control, Vol. 85 (2023), 104799.
[14]
Dian Yu and Shouqian Sun. 2020. A systematic exploration of deep neural networks for EDA-based emotion recognition. Information, Vol. 11, 4 (2020), 212.
[15]
Yu-Liang Hsu, Jeen-Shing Wang, Wei-Chun Chiang, and Chien-Han Hung. 2017. Automatic ECG-based emotion recognition in music listening. IEEE Transactions on Affective Computing, Vol. 11, 1 (2017), 85--99.
[16]
Shuo Xiao, Xiaojing Qiu, Chaogang Tang, and Zhenzhen Huang. 2023. A Spatial-Temporal ECG Emotion Recognition Model Based on Dynamic Feature Fusion. In ICASSP 2023--2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1--5.
[17]
Imam Yogie Susanto, Tse-Yu Pan, Chien-Wen Chen, Min-Chun Hu, and Wen-Huang Cheng. 2020. Emotion recognition from galvanic skin response signal based on deep hybrid neural networks. In Proceedings of the 2020 International Conference on Multimedia Retrieval. 341--345.
[18]
Xiaopeng Si, Huang He, Jiayue Yu, and Dong Ming. 2023. Cross-subject emotion recognition brain-computer interface based on fNIRS and DBJNet. Cyborg and Bionic Systems, Vol. 4 (2023), 0045.
[19]
Jiafa Chen, Kaiwei Yu, Fei Wang, Zhengxian Zhou, Yifei Bi, Songlin Zhuang, and Dawei Zhang. 2024. Temporal convolutional network-enhanced real-time implicit emotion recognition with an innovative wearable fNIRS-EEG dual-modal system. Electronics, Vol. 13, 7 (2024), 1310.
[20]
Michiel Spapé, Kalle Mäkelä, and Tuukka Ruotsalo. 2023. NEMO: A Database for Emotion Analysis Using Functional Near-infrared Spectroscopy. IEEE Transactions on Affective Computing (2023).
[21]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
[22]
Tianhao Gu, Zhe Wang, Xinlei Xu, Dongdong Li, Hai Yang, and Wenli Du. 2022. Frame-level teacher--student learning with data privacy for EEG emotion recognition. IEEE transactions on neural networks and learning systems, Vol. 34, 12 (2022), 11021--11028.
[23]
Pai Wang, Chunyong Guo, Shuangqiang Xie, Xiang Qiao, Lili Mao, and Xianyong Fu. 2022. EEG emotion recognition based on knowledge distillation optimized residual networks. In 2022 IEEE 6th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). IEEE, 574--581.
[24]
Yucheng Liu, Ziyu Jia, and Haichao Wang. 2023. Emotionkd: a cross-modal knowledge distillation framework for emotion recognition based on physiological signals. In Proceedings of the 31st ACM International Conference on Multimedia. 6122--6131.
[25]
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4320--4328.
[26]
Xiatian Zhu, Shaogang Gong, et al. 2018. Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems, Vol. 31 (2018).
[27]
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. 2020. Online knowledge distillation with diverse peers. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 3430--3437.
[28]
Guile Wu and Shaogang Gong. 2021. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on artificial intelligence, Vol. 35. 10302--10310.
[29]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597--1607.
[30]
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2020. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659 (2020).
[31]
Xianggen Liu, Yunan Luo, Pengyong Li, Sen Song, and Jian Peng. 2021. Deep geometric representations for modeling effects of mutations on protein-protein binding affinity. PLoS computational biology, Vol. 17, 8 (2021), e1009284.
[32]
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in neural information processing systems, Vol. 33 (2020), 18661--18673.
[33]
Sahar Soltanieh, Ali Etemad, and Javad Hashemi. 2022. Analysis of augmentations for contrastive ecg representation learning. In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--10.
[34]
Xinke Shen, Xianggen Liu, Xin Hu, Dan Zhang, and Sen Song. 2022. Contrastive learning of subject-invariant EEG representations for cross-subject emotion recognition. IEEE Transactions on Affective Computing, Vol. 14, 3 (2022), 2496--2511.
[35]
Jiyang Wang, Ayse Altay, and Senem Velipasalar. 2024. Block-As-Domain Adaptation for Workload Prediction from fNIRS Data. arXiv preprint arXiv:2405.00213 (2024).
[36]
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2023. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.
[37]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106--11115.
[38]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).
[39]
Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013).
[40]
Peter Lang and Margaret M Bradley. 2007. The International Affective Picture System (IAPS) in the study of emotion and attention. Handbook of emotion elicitation and assessment, Vol. 29 (2007), 70--73.
[41]
Ilya Loshchilov and Frank Hutter. 2018. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
[42]
Ilya Loshchilov and Frank Hutter. 2016. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations.
[43]
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? Advances in neural information processing systems, Vol. 32 (2019).
[44]
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. 2020. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11020--11029.
[45]
Chuanguang Yang, Zhulin An, Linhang Cai, and Yongjun Xu. 2022. Mutual contrastive learning for visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3045--3053.
Index Terms
- Online Multi-level Contrastive Representation Distillation for Cross-Subject fNIRS Emotion Recognition
Recommendations
EmotiW 2016: video and group-level emotion recognition challenges
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal InteractionThis paper discusses the baseline for the Emotion Recognition in the Wild (EmotiW) 2016 challenge. Continuing on the theme of automatic affect recognition `in the wild', the EmotiW challenge 2016 consists of two sub-challenges: an audio-video based ...
From individual to group-level emotion recognition: EmotiW 5.0
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal InteractionResearch in automatic affect recognition has come a long way. This paper describes the fifth Emotion Recognition in the Wild (EmotiW) challenge 2017. EmotiW aims at providing a common benchmarking platform for researchers working on different aspects ...
Examining the Neural Correlates of Incidental Facial Emotion Encoding Within the Prefrontal Cortex Using Functional Near-Infrared Spectroscopy
Proceedings, Part I, 10th International Conference on Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience - Volume 9743Previous neuroimaging research has implicated the prefrontal cortex PFC as a region of the brain that is vital for various aspects of emotion processing. The present study sought to examine the neural correlates of incidental facial emotion encoding, ...
Comments
Information & Contributors
Information
Published In
October 2024
67 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Guangdong Provincial Key Laboratory of Human Digital Twin
- National Natural Science Foundation of China
- Natural Science Foundation of Guangdong Province
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 66Total Downloads
- Downloads (Last 12 months)66
- Downloads (Last 6 weeks)9
Reflects downloads up to 30 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in