research-article

A Multi-Domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition

Authors:

Bao-Liang LuAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 5565 - 5573

https://doi.org/10.1145/3474085.3475697

Published: 17 October 2021 Publication History

Abstract

Among all solutions of emotion recognition tasks, electroencephalogram (EEG) is a very effective tool and has received broad attention from researchers. In addition, information across multimedia in EEG often provides a more complete picture of emotions. However, few of the existing studies concurrently incorporate EEG information from temporal domain, frequency domain and functional brain connectivity. In this paper, we propose a Multi-Domain Adaptive Graph Convolutional Network (MD-AGCN), fusing the knowledge of both the frequency domain and the temporal domain to fully utilize the complementary information of EEG signals. MD-AGCN also considers the topology of EEG channels by combining the inter-channel correlations with the intra-channel information, from which the functional brain connectivity can be learned in an adaptive manner. Extensive experimental results demonstrate that our model exceeds state-of-the-art methods in most experimental settings. At the same time, the results show that MD-AGCN could extract complementary domain information and exploit channel relationships for EEG-based emotion recognition effectively.

References

[1]

Fatemeh Bahari and Amin Janghorbani. 2013. EEG-based emotion recognition using recurrence plot analysis and k nearest neighbor classifier. In 2013 20th Iranian Conference on Biomedical Engineering (ICBME). IEEE, 228--233.

[2]

Andrey V Bocharov, Gennady G Knyazev, and Alexander N Savostyanov. 2017. Depression and implicit emotion processing: An EEG study. Neurophysiologie Clinique/Clinical Neurophysiology, Vol. 47, 3 (2017), 225--230.

[3]

Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. 2013. Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 81--84.

[4]

Lester I Goldfischer. 1965. Autocorrelation function and power spectral density of laser-produced speckle patterns. Josa, Vol. 55, 3 (1965), 247--253.

[5]

Matti Hämäläinen, Riitta Hari, Risto J Ilmoniemi, Jukka Knuutila, and Olli V Lounasmaa. 1993. Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain. Reviews of Modern Physics, Vol. 65, 2 (1993), 413.

[6]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and Pattern Recognition. 770--778.

[7]

Tiffany C Ho, Colm G Connolly, Eva Henje Blom, Kaja Z LeWinn, Irina A Strigo, Martin P Paulus, Guido Frank, Jeffrey E Max, Jing Wu, Melanie Chan, et al. 2015. Emotion-dependent functional connectivity of the default mode network in adolescent depression. Biological Psychiatry, Vol. 78, 9 (2015), 635--646.

[8]

Robert Jenke, Angelika Peer, and Martin Buss. 2014. Feature extraction and selection for emotion recognition from EEG. IEEE Transactions on Affective Computing, Vol. 5, 3 (2014), 327--339.

[9]

Ziyu Jia, Youfang Lin, Xiyang Cai, Haobin Chen, Haijun Gou, and Jing Wang. 2020. SST-EmotionNet: Spatial-spectral-temporal based attention 3d dense network for EEG emotion recognition. In Proceedings of the 28th ACM International Conference on Multimedia. 2909--2917.

Digital Library

[10]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, Vol. 521, 7553 (2015), 436--444.

[11]

You-Yun Lee and Shulan Hsieh. 2014. Classifying different emotional states by means of EEG-based functional connectivity patterns. PloS one, Vol. 9, 4 (2014), e95415.

[12]

Tian-Hao Li, Wei Liu, Wei-Long Zheng, and Bao-Liang Lu. 2019 a. Classification of five emotions from EEG and eye movement signals: Discrimination ability and stability over time. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 607--610.

[13]

Yang Li, Lei Wang, Wenming Zheng, Yuan Zong, Lei Qi, Zhen Cui, Tong Zhang, and Tengfei Song. 2020. A novel bi-hemispheric discrepancy model for eeg emotion recognition. IEEE Transactions on Cognitive and Developmental Systems, Vol. 13, 2 (2020), 354--367.

[14]

Yang Li, Wenming Zheng, Zhen Cui, Tong Zhang, and Yuan Zong. 2018. A Novel Neural Network Model based on Cerebral Hemispheric Asymmetry for EEG Emotion Recognition. In IJCAI. 1561--1567.

[15]

Yang Li, Wenming Zheng, Lei Wang, Yuan Zong, and Zhen Cui. 2019 b. From regional to global brain: A novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Transactions on Affective Computing (2019).

[16]

Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. 2021. Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition. IEEE Transactions on Cognitive and Developmental Systems (2021).

[17]

Yifei Lu, Wei-Long Zheng, Binbin Li, and Bao-Liang Lu. 2015. Combining eye movements and EEG to enhance emotion recognition. In Twenty-Fourth International Joint Conference on Artificial Intelligence.

[18]

P MatLab. 2018. 9.7. 0.1190202 (R2019b). MathWorks Inc Natick MA USA (2018).

[19]

Iris B Mauss and Michael D Robinson. 2009. Measures of emotion: A review. Cognition and Emotion, Vol. 23, 2 (2009), 209--237.

[20]

Michael Murias, Sara J Webb, Jessica Greenson, and Geraldine Dawson. 2007. Resting state cortical connectivity reflected in EEG coherence in individuals with autism. Biological Psychiatry, Vol. 62, 3 (2007), 270--273.

[21]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).

[22]

Louis A Schmidt and Laurel J Trainor. 2001. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition & Emotion, Vol. 15, 4 (2001), 487--500.

[23]

Maryam M Shanechi. 2019. Brain--machine interfaces from motor to mood. Nature Neuroscience, Vol. 22, 10 (2019), 1554--1564.

[24]

Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12026--12035.

[25]

Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. 2018. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, Vol. 11, 3 (2018), 532--541.

[26]

Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing, Vol. 129 (2014), 94--106.

Digital Library

[27]

Alexis E Whitton, Stephanie Deccy, Manon L Ironside, Poornima Kumar, Miranda Beltzer, and Diego A Pizzagalli. 2018. Electroencephalography source functional connectivity reveals abnormal high-frequency communication among large-scale functional networks in depression. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, Vol. 3, 1 (2018), 50--58.

[28]

Xun Wu, Wei-Long Zheng, and Bao-Liang Lu. 2019. Identifying functional brain connectivity patterns for EEG-based emotion recognition. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 235--238.

[29]

Yongqiang Yin, Xiangwei Zheng, Bin Hu, Yuang Zhang, and Xinchun Cui. 2021. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Applied Soft Computing, Vol. 100 (2021), 106954.

[30]

Zhongliang Yin, Jun Li, Yun Zhang, Aifeng Ren, Karen M Von Meneen, and Liyu Huang. 2017. Functional brain network analysis of schizophrenic patients with positive and negative syndrome based on mutual information of EEG time series. Biomedical Signal Processing and Control, Vol. 31 (2017), 331--338.

[31]

Guanhua Zhang, Minjing Yu, Yong-Jin Liu, Guozhen Zhao, Dan Zhang, and Wenming Zheng. 2021. SparseDGCNN: Recognizing Emotion from Multichannel EEG Signals. IEEE Transactions on Affective Computing (2021).

[32]

Tong Zhang, Wenming Zheng, Zhen Cui, Yuan Zong, and Yang Li. 2018. Spatial--temporal recurrent neural network for emotion recognition. IEEE Transactions on Cybernetics, Vol. 49, 3 (2018), 839--847.

[33]

Li-Ming Zhao, Rui Li, Wei-Long Zheng, and Bao-Liang Lu. 2019. Classification of five emotions from EEG and eye movement signals: complementary representation properties. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 611--614.

[34]

Wei-Long Zheng, Bo-Nan Dong, and Bao-Liang Lu. 2014. Multimodal emotion recognition using EEG and eye tracking data. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 5040--5043.

[35]

Wei-Long Zheng, Wei Liu, Yifei Lu, Bao-Liang Lu, and Andrzej Cichocki. 2018. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, Vol. 49, 3 (2018), 1110--1122.

[36]

Wei-Long Zheng and Bao-Liang Lu. 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, Vol. 7, 3 (2015), 162--175.

Digital Library

[37]

Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. 2017. Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, Vol. 10, 3 (2017), 417--429.

Digital Library

[38]

Peixiang Zhong, Di Wang, and Chunyan Miao. 2020. EEG-based emotion recognition using regularized graph neural networks. IEEE Transactions on Affective Computing (2020).

Cited By

Wang THuang XXiao ZCai WTai Y(2024)EEG emotion recognition based on differential entropy feature matrix through 2D-CNN-LSTM networkEURASIP Journal on Advances in Signal Processing10.1186/s13634-024-01146-y2024:1Online publication date: 8-Apr-2024
https://doi.org/10.1186/s13634-024-01146-y
Liu HZhang YChen XZhang DLi RQin T(2024)Self-Supervised EEG Representation Learning for Robust Emotion RecognitionACM Transactions on Sensor Networks10.1145/367497520:5(1-22)Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3674975
Li CWang FBian N(2024)Multi-View Hierarchical Attention Graph Convolutional Network with Domain Adaptation for EEG Emotion RecognitionProceedings of the 2024 3rd International Conference on Cryptography, Network Security and Communication Technology10.1145/3673277.3673384(624-630)Online publication date: 19-Jan-2024
https://dl.acm.org/doi/10.1145/3673277.3673384
Show More Cited By

Index Terms

A Multi-Domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Affective Brain-computer Interface has achieved considerable advances that researchers can successfully interpret labeled and flawless EEG data collected in laboratory settings. However, the annotation of EEG data is time-consuming and requires a vast ...
EEG-Based Emotion Recognition Using Convolutional Neural Networks
Computational Science – ICCS 2021
Abstract
In this day and age, Electroencephalography-based methods for Automated Affect Recognition are becoming more and more popular. Owing to the vast amount of information gathered in EEG signals, such methods provide satisfying results in terms of ...
Multi-view domain-adaptive representation learning for EEG-based emotion recognition
Abstract
Current research suggests that there exist certain limitations in EEG emotion recognition, including redundant and meaningless time-frames and channels, as well as inter- and intra-individual differences in EEG signals from different subjects. To ...
Graphical abstract

Display Omitted
Highlights
- Our method can extract discriminative features and minimize individual differences.
- Multi-view learning explores the complementarity between different channels.
- Multi-view cross-attention extracts discriminative spatio-temporal ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
814
Total Downloads

Downloads (Last 12 months)200
Downloads (Last 6 weeks)27

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang THuang XXiao ZCai WTai Y(2024)EEG emotion recognition based on differential entropy feature matrix through 2D-CNN-LSTM networkEURASIP Journal on Advances in Signal Processing10.1186/s13634-024-01146-y2024:1Online publication date: 8-Apr-2024
https://doi.org/10.1186/s13634-024-01146-y
Liu HZhang YChen XZhang DLi RQin T(2024)Self-Supervised EEG Representation Learning for Robust Emotion RecognitionACM Transactions on Sensor Networks10.1145/367497520:5(1-22)Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3674975
Li CWang FBian N(2024)Multi-View Hierarchical Attention Graph Convolutional Network with Domain Adaptation for EEG Emotion RecognitionProceedings of the 2024 3rd International Conference on Cryptography, Network Security and Communication Technology10.1145/3673277.3673384(624-630)Online publication date: 19-Jan-2024
https://dl.acm.org/doi/10.1145/3673277.3673384
Wang YZhang BDi L(2024)Research Progress of EEG-Based Emotion Recognition: A SurveyACM Computing Surveys10.1145/366600256:11(1-49)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3666002
Klepl DWu MHe F(2024)Graph Neural Network-Based EEG Classification: A SurveyIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2024.335575032(493-503)Online publication date: 2024
https://doi.org/10.1109/TNSRE.2024.3355750
Jin MDu CHe HCai TLi J(2024)PGCN: Pyramidal Graph Convolutional Network for EEG Emotion RecognitionIEEE Transactions on Multimedia10.1109/TMM.2024.338567626(9070-9082)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3385676
Liu FYang PShu YLiu NSheng JLuo JWang XLiu Y(2024)Emotion Recognition From Few-Channel EEG Signals by Integrating Deep Feature Aggregation and Transfer LearningIEEE Transactions on Affective Computing10.1109/TAFFC.2023.333653115:3(1315-1330)Online publication date: Jul-2024
https://doi.org/10.1109/TAFFC.2023.3336531
Zeng YWang GRen HCai YLeung HLi QHuang Q(2024)A Knowledge-Enhanced and Topic-Guided Domain Adaptation Model for Aspect-Based Sentiment AnalysisIEEE Transactions on Affective Computing10.1109/TAFFC.2023.329221315:2(709-721)Online publication date: Apr-2024
https://doi.org/10.1109/TAFFC.2023.3292213
Chang YZheng XChen YLi XMiao Q(2024)Spatiotemporal Gated Graph Transformer for EEG-Based Emotion RecognitionIEEE Signal Processing Letters10.1109/LSP.2024.341004431(1630-1634)Online publication date: 2024
https://doi.org/10.1109/LSP.2024.3410044
Tang HXie SXie XCui YLi BZheng DHao YWang XJiang YTian Z(2024)Multi-Domain Based Dynamic Graph Representation Learning for EEG Emotion RecognitionIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.341516328:9(5227-5238)Online publication date: Sep-2024
https://doi.org/10.1109/JBHI.2024.3415163
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents