research-article

SST-EmotionNet: Spatial-Spectral-Temporal based Attention 3D Dense Network for EEG Emotion Recognition

Authors:

Jing WangAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 2909 - 2917

https://doi.org/10.1145/3394171.3413724

Published: 12 October 2020 Publication History

Abstract

Multimedia stimulation of brain activities has not only become an emerging field for intensive research, but also achieves important progress in the electroencephalogram (EEG) emotion classification based on brain activities. However, how to make full use of different EEG features and the discriminative local patterns among the features for different emotions is challenging. Existing models ignore the complementarity among the spatial-spectral-temporal features and discriminative local patterns in all features, which limits the classification ability of the models to a certain extent. In this paper, we propose a novel spatial-spectral-temporal based attention 3D dense network, named SST-EmotionNet, for EEG emotion recognition. The main advantage of the SST-EmotionNet is the simultaneous integration of spatial-spectral-temporal features in a unified network framework. Meanwhile, a 3D attention mechanism is designed to adaptively explore discriminative local patterns. Extensive experiments on two real-world datasets demonstrate that the SST-EmotionNet outperforms the state-of-the-art baselines.

Supplementary Material

MP4 File (3394171.3413724.mp4)

This is our presentation video. In this video, we firstly introduce the research background and the related work of our paper. Then, the motivation and the challenges of our research are presented. After that, our solutions for each challenge and the experiment results are shown. Finally, we summarize the main contributions of this paper.

Download
69.40 MB

References

[1]

Alaa M Al-Kaysi, Ahmed Al-Ani, Colleen K Loo, Tamara Y Powell, Donel M Martin, Michael Breakspear, and Tjeerd W Boonstra. Predicting tdcs treatment outcomes of patients with major depressive disorder using automated eeg classification. Journal of Affective Disorders, 208:597--603, 2017.

[2]

Salma Alhagry, Aly Aly Fahmy, and Reda A El-Khoribi. Emotion recognition based on eeg using lstm recurrent neural network. Emotion, 8(10):355--358, 2017.

[3]

Abeer Alnafjan, Manar Hosny, Areej Al-Wabil, and Yousef Al-Ohali. Classification of human emotions from electroencephalogram (eeg) signal using deep neural network. International Journal of Advanced Computer Science and Applications, 8, 01 2017.

[4]

Fatemeh Bahari and Amin Janghorbani. Eeg-based emotion recognition using recurrence plot analysis and k nearest neighbor classifier. In 2013 20th Iranian Conference on Biomedical Engineering (ICBME), pages 228--233. IEEE, 2013.

[5]

Andrey V Bocharov, Gennady G Knyazev, and Alexander N Savostyanov. Depression and implicit emotion processing: An eeg study. Neurophysiologie Clinique/Clinical Neurophysiology, 47(3):225--230, 2017.

[6]

Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. Differential entropy feature for eeg-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), pages 81--84. IEEE, 2013.

[7]

Rahma Fourati, Boudour Ammar, Chaouki Aouiti, Javier Sanchez-Medina, and Adel M Alimi. Optimized echo state network with intrinsic plasticity for eeg-based emotion recognition. In International Conference on Neural Information Processing, pages 718--727. Springer, 2017.

[8]

Christos A Frantzidis, Charalampos Bratsas, Christos L Papadelis, Evdokimos Konstantinidis, Costas Pappas, and Panagiotis D Bamidis. Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE Transactions on Information Technology in Biomedicine, 14(3):589--597, 2010.

Digital Library

[9]

Lester I Goldfischer. Autocorrelation function and power spectral density of laser-produced speckle patterns. JOSA, 55(3):247--253, 1965.

[10]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700--4708, 2017.

[11]

Ziyu Jia, Youfang Lin, Zehui Jiao, Yan Ma, and Jing Wang. Detecting causality in multivariate time series via non-uniform embedding. Entropy, 21(12):1233, 2019.

[12]

Ziyu Jia, Youfang Lin, Yunxiao Liu, Zehui Jiao, and Jing Wang. Refined nonuniform embedding for coupling detection in multivariate time series. Physical Review E, 101(6):062113, 2020.

[13]

Ziyu Jia, Youfang Lin, Jing Wang, Ronghao Zhou, Xiaojun Ning, Yuanlai He, and Yaoshuai Zhao. Graphsleepnet: Adaptive spatial-temporal graph convolutional networks for sleep stage classification. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 1324--1330. ijcai.org, 2020.

[14]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436--444, 2015.

[15]

Jinpeng Li, Zhaoxiang Zhang, and Huiguang He. Hierarchical convolutional neural networks for eeg-based emotion recognition. Cognitive Computation, 10(2):368--380, 2018.

[16]

Tian-Hao Li, Wei Liu, Wei-Long Zheng, and Bao-Liang Lu. Classification of five emotions from eeg and eye movement signals: Discrimination ability and stability over time. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pages 607--610. IEEE, 2019.

[17]

Yang Li, Wenming Zheng, Zhen Cui, Tong Zhang, and Yuan Zong. A novel neural network model based on cerebral hemispheric asymmetry for eeg emotion recognition. In IJCAI, pages 1561--1567, 2018.

[18]

Yang Li, Wenming Zheng, Lei Wang, Yuan Zong, and Zhen Cui. From regional to global brain: A novel hierarchical spatial-temporal neural network model for eeg emotion recognition. IEEE Transactions on Affective Computing, 2019.

[19]

Yang Li, Wenming Zheng, Lei Wang, Yuan Zong, Lei Qi, Zhen Cui, Tong Zhang, and Tengfei Song. A novel bi-hemispheric discrepancy model for eeg emotion recognition. arXiv preprint arXiv:1906.01704, 2019.

[20]

Yuan-Pin Lin, Chi-Hong Wang, Tzyy-Ping Jung, Tien-Lin Wu, Shyh-Kang Jeng, Jeng-Ren Duann, and Jyh-Horng Chen. Eeg-based emotion recognition in music listening. IEEE Transactions on Biomedical Engineering, 57(7):1798--1806, 2010.

[21]

Yisi Liu and Olga Sourina. Real-time fractal-based valence level recognition from eeg. In Transactions on computational science XVIII, pages 101--120. Springer, 2013.

[22]

Jiaxin Ma, Hao Tang, Wei-Long Zheng, and Bao-Liang Lu. Emotion recognition using multimodal residual lstm network. In Proceedings of the 27th ACM International Conference on Multimedia, pages 176--183, 2019.

Digital Library

[23]

Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533--5541, 2017.

[24]

Siddharth Siddharth, Tzyy-Ping Jung, and Terrence J Sejnowski. Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Transactions on Affective Computing, 2019.

[25]

Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. Eeg emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, 2018.

[26]

Johan AK Suykens and Joos Vandewalle. Least squares support vector machine classifiers. Neural processing letters, 9(3):293--300, 1999.

[27]

Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. Eeg-based emotion recognition using frequency domain features and support vector machines. In International conference on neural information processing, pages 734--743. Springer, 2011.

Digital Library

[28]

Yimin Yang, QM Jonathan Wu, Wei-Long Zheng, and Bao-Liang Lu. Eeg-based emotion recognition using hierarchical network with subnetwork nodes. IEEE Transactions on Cognitive and Developmental Systems, 10(2):408--419, 2017.

[29]

Tong Zhang, Xuehan Wang, Xiangmin Xu, and CL Philip Chen. Gcb-net: Graph convolutional broad network and its application in emotion recognition. IEEE Transactions on Affective Computing, 2019.

[30]

Wei-Long Zheng, Wei Liu, Yifei Lu, Bao-Liang Lu, and Andrzej Cichocki. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE transactions on cybernetics, 49(3):1110--1122, 2018.

[31]

Wei-Long Zheng and Bao-Liang Lu. Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3):162--175, 2015.

Digital Library

[32]

Wei-Long Zheng, Jia-Yi Zhu, Yong Peng, and Bao-Liang Lu. Eeg-based emotion classification using deep belief networks. In 2014 IEEE International Conference on Multimedia and Expo (ICME), pages 1--6. IEEE, 2014.

[33]

Wenming Zheng. Multichannel eeg-based emotion recognition via group sparse canonical correlation analysis. IEEE Transactions on Cognitive and Developmental Systems, 9(3):281--290, 2016.

[34]

Peixiang Zhong, Di Wang, and Chunyan Miao. Eeg-based emotion recognition using regularized graph neural networks. arXiv preprint arXiv:1907.07835, 2019.

[35]

Sheng-hua Zhong, Ahmed Fares, and Jianmin Jiang. An attentional-lstm for improved classification of brain activities evoked by images. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1295--1303, 2019.

Cited By

Cruz-Vazquez JMontiel-Pérez JRomero-Herrera RRubio-Espino E(2025)Emotion Recognition from EEG Signals Using Advanced Transformations and Deep LearningMathematics10.3390/math1302025413:2(254)Online publication date: 14-Jan-2025
https://doi.org/10.3390/math13020254
Chen QMao XSong YWang K(2025)An EEG-based emotion recognition method by fusing multi-frequency-spatial features under multi-frequency bandsJournal of Neuroscience Methods10.1016/j.jneumeth.2025.110360415(110360)Online publication date: Mar-2025
https://doi.org/10.1016/j.jneumeth.2025.110360
Miao MLiang JSheng ZLiu WXu BHu W(2025)ST-SHAP: A hierarchical and explainable attention network for emotional EEG representation learning and decodingJournal of Neuroscience Methods10.1016/j.jneumeth.2024.110317414(110317)Online publication date: Feb-2025
https://doi.org/10.1016/j.jneumeth.2024.110317
Show More Cited By

Index Terms

SST-EmotionNet: Spatial-Spectral-Temporal based Attention 3D Dense Network for EEG Emotion Recognition
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Spatial-temporal Transformers for EEG Emotion Recognition
ICAAI '22: Proceedings of the 6th International Conference on Advances in Artificial Intelligence

Electroencephalography (EEG) is a popular and effective tool for emotion recognition. However, the propagation mechanisms of EEG in the human brain and its intrinsic correlation with emotions are still obscure to researchers. This work proposes four ...
ASTDF-Net: Attention-Based Spatial-Temporal Dual-Stream Fusion Network for EEG-Based Emotion Recognition
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Emotion recognition based on electroencephalography (EEG) has attracted significant attention and achieved considerable advances in the fields of affective computing and human-computer interaction. However, most existing studies ignore the coupling and ...
Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers

A feature-based emotion recognition model is proposed for EEG-based BCI.The approach combines statistical-based feature selection methods and SVM emotion classifiers.The model is based on Valence/Arousal dimensions for emotion classification.Our ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

the Fundamental Research Funds for the Central Universities
National Natural Science Foundation of China

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

113
Total Citations
View Citations
1,970
Total Downloads

Downloads (Last 12 months)397
Downloads (Last 6 weeks)59

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cruz-Vazquez JMontiel-Pérez JRomero-Herrera RRubio-Espino E(2025)Emotion Recognition from EEG Signals Using Advanced Transformations and Deep LearningMathematics10.3390/math1302025413:2(254)Online publication date: 14-Jan-2025
https://doi.org/10.3390/math13020254
Chen QMao XSong YWang K(2025)An EEG-based emotion recognition method by fusing multi-frequency-spatial features under multi-frequency bandsJournal of Neuroscience Methods10.1016/j.jneumeth.2025.110360415(110360)Online publication date: Mar-2025
https://doi.org/10.1016/j.jneumeth.2025.110360
Miao MLiang JSheng ZLiu WXu BHu W(2025)ST-SHAP: A hierarchical and explainable attention network for emotional EEG representation learning and decodingJournal of Neuroscience Methods10.1016/j.jneumeth.2024.110317414(110317)Online publication date: Feb-2025
https://doi.org/10.1016/j.jneumeth.2024.110317
Wu YMeng TLi QXi YZhang H(2025)Study on multidimensional emotion recognition fusing dynamic brain network features in EEG signalsBiomedical Signal Processing and Control10.1016/j.bspc.2024.107054100(107054)Online publication date: Feb-2025
https://doi.org/10.1016/j.bspc.2024.107054
Yu HXiong XZhou JQian RSha K(2024)CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition ModelSensors10.3390/s2415483724:15(4837)Online publication date: 25-Jul-2024
https://doi.org/10.3390/s24154837
Guo YZhang BFan XShen XPeng X(2024)A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion RecognitionMathematics10.3390/math1208118012:8(1180)Online publication date: 15-Apr-2024
https://doi.org/10.3390/math12081180
Gómez-González ICastro-García JMerino-Monge MSánchez-Antón GHamidi FMendoza-Sagrera AMolina-Cantero A(2024)Emotional State Measurement Trial (EMOPROEXE): A Protocol for Promoting Exercise in Adults and Children with Cerebral PalsyJournal of Personalized Medicine10.3390/jpm1405052114:5(521)Online publication date: 14-May-2024
https://doi.org/10.3390/jpm14050521
Mounesi Rad SDanishvar S(2024)Emotion Recognition Using EEG Signals through the Design of a Dry Electrode Based on the Combination of Type 2 Fuzzy Sets and Deep Convolutional Graph NetworksBiomimetics10.3390/biomimetics90905629:9(562)Online publication date: 18-Sep-2024
https://doi.org/10.3390/biomimetics9090562
Lu WZhang XXia LMa HTan T(2024)Domain adaptation spatial feature perception neural network for cross-subject EEG emotion recognitionFrontiers in Human Neuroscience10.3389/fnhum.2024.147163418Online publication date: 17-Dec-2024
https://doi.org/10.3389/fnhum.2024.1471634
Zhang MCui Y(2024)Self supervised learning based emotion recognition using physiological signalsFrontiers in Human Neuroscience10.3389/fnhum.2024.133472118Online publication date: 9-Apr-2024
https://doi.org/10.3389/fnhum.2024.1334721
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents