short-paper

An Attention Model for Group-Level Emotion Recognition

Authors:

Dakshit Agrawal,

Hardik Chauhan,

Marco PedersoliAuthors Info & Claims

ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

Pages 611 - 615

https://doi.org/10.1145/3242969.3264985

Published: 02 October 2018 Publication History

Abstract

In this paper we propose a new approach for classifying the global emotion of images containing groups of people. To achieve this task, we consider two different and complementary sources of information: i) a global representation of the entire image (ii) a local representation where only faces are considered. While the global representation of the image is learned with a convolutional neural network (CNN), the local representation is obtained by merging face features through an attention mechanism. The two representations are first learned independently with two separate CNN branches and then fused through concatenation in order to obtain the final group-emotion classifier. For our submission to the EmotiW 2018 group-level emotion recognition challenge, we combine several variations of the proposed model into an ensemble, obtaining a final accuracy of 64.83% on the test set and ranking 4th among all challenge participants.

References

[1]

Roland Goecke Abhinav Dhall, Amanjot Kaur and Tom Gedeon . 2018. EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction, ACM ICMI 2018. ACM ICMI.

Digital Library

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio . 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR Vol. abs/1409.0473 (2014).

[3]

Kwang-Ho Choi, Junbeom Kim, Oh Sang Kwon, Min Kim, Yeon Hee Ryu, and Ji-Eun Park . 2017. Is heart rate variability (HRV) an adequate tool for evaluating human emotions? -- A focus on the use of the International Affective Picture System (IAPS). Psychiatry Research Vol. 251 (2017), 192--196.

[4]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei . 2009. ImageNet: A Large-Scale Hierarchical Image Database CVPR09.

[5]

A. Dhall, J. Joshi, K. Sikka, R. Goecke, and N. Sebe . 2015. The more the merrier: Analysing the affect of a group of people in images 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. Vol. 1. 1--8.

[6]

Samira Ebrahimi Kahou, Vincent Michalski, Kishore Konda, Roland Memisevic, and Christopher Pal . 2015. Recurrent Neural Networks for Emotion Recognition in Video Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI '15). ACM, New York, NY, USA, 467--474.

Digital Library

[7]

Rohit Girdhar and Deva Ramanan . 2017. Attentional pooling for action recognition. In Advances in Neural Information Processing Systems. 34--45.

[8]

Xin Guo, Luisa F. Polan'ıa, and Kenneth E. Barner . 2017. Group-level Emotion Recognition Using Deep Models on Image Scene, Faces, and Skeletons. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017). ACM, New York, NY, USA, 603--608.

Digital Library

[9]

Gao Huang, Zhuang Liu, and Kilian Q. Weinberger . 2016. Densely Connected Convolutional Networks. CoRR Vol. abs/1608.06993 (2016). showeprint{arxiv}1608.06993deftempurl%http://arxiv.org/abs/1608.06993 tempurl

[10]

Sergey Ioffe and Christian Szegedy . 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. CoRR Vol. abs/1502.03167 (2015). showeprint{arxiv}1502.03167deftempurl%http://arxiv.org/abs/1502.03167 tempurl

[11]

N. Jaques, S. Taylor, A. Azaria, A. Ghandeharioun, A. Sano, and R. Picard . 2015. Predicting students' happiness from physiology, phone, mobility, and behavioral data 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Vol. Vol. 00. 222--228.

Digital Library

[12]

Abubakrelsedik Karali, Ahmad Bassiouny, and Motaz El-Saban . 2016. Facial Expression Recognition in the Wild using Rich Deep Features. CoRR Vol. abs/1601.02487 (2016). showeprint{arxiv}1601.02487deftempurl%http://arxiv.org/abs/1601.02487 tempurl

[13]

Diederik P. Kingma and Jimmy Ba . 2014. Adam: A Method for Stochastic Optimization. CoRR Vol. abs/1412.6980 (2014). showeprint{arxiv}1412.6980deftempurl%http://arxiv.org/abs/1412.6980 tempurl

[14]

Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza . 2017. Emotion recognition in context. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song . 2017. SphereFace: Deep Hypersphere Embedding for Face Recognition. CoRR Vol. abs/1704.08063 (2017). showeprint{arxiv}1704.08063deftempurl%http://arxiv.org/abs/1704.08063 tempurl

[16]

Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang . 2016. Large-margin Softmax Loss for Convolutional Neural Networks Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML'16). JMLR.org, 507--516. deftempurl%http://dl.acm.org/citation.cfm?id=3045390.3045445 tempurl

Digital Library

[17]

Volodymyr Mnih, Nicolas Heess, Alex Graves, et almbox. . 2014. Recurrent models of visual attention. In Advances in neural information processing systems. 2204--2212.

Digital Library

[18]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov . 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research Vol. 15 (2014), 1929--1958. deftempurl%http://jmlr.org/papers/v15/srivastava14a.html tempurl

Digital Library

[19]

Lianzhi Tan, Kaipeng Zhang, Kai Wang, Xiaoxing Zeng, Xiaojiang Peng, and Yu Qiao . 2017. Group Emotion Recognition with Individual Facial Emotion CNNs and Global Image Based CNNs. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017). ACM, New York, NY, USA, 549--552.

Digital Library

[20]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin . 2017. Attention Is All You Need. CoRR Vol. abs/1706.03762 (2017). showeprint{arxiv}1706.03762deftempurl%http://arxiv.org/abs/1706.03762 tempurl

[21]

Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang . 2017. Residual Attention Network for Image Classification Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3156--3164.

[22]

Qinglan Wei, Yijia Zhao, Qihua Xu, Liandong Li, Jun He, Lejun Yu, and Bo Sun . 2017. A New Deep-learning Framework for Group Emotion Recognition Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017). ACM, New York, NY, USA, 587--592.

Digital Library

[23]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio . 2015. Show, attend and tell: Neural image caption generation with visual attention International conference on machine learning. 2048--2057.

Digital Library

[24]

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li . 2014. Learning Face Representation from Scratch. CoRR Vol. abs/1411.7923 (2014). showeprint{arxiv}1411.7923deftempurl%http://arxiv.org/abs/1411.7923 tempurl

[25]

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao . 2016. Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks. CoRR Vol. abs/1604.02878 (2016). showeprint{arxiv}1604.02878deftempurl%http://arxiv.org/abs/1604.02878 tempurl

Cited By

Wang XZhang DLee D(2024)Implementing the Affective Mechanism for Group Emotion Recognition With a New Graph Convolutional Network ArchitectureIEEE Transactions on Affective Computing10.1109/TAFFC.2023.332010115:3(1104-1115)Online publication date: Jul-2024
https://doi.org/10.1109/TAFFC.2023.3320101
Xu JHuang X(2024)Group-Level Emotion Recognition Using Hierarchical Dual-Branch Cross Transformer with Semi-Supervised Learning2024 IEEE 4th International Conference on Software Engineering and Artificial Intelligence (SEAI)10.1109/SEAI62072.2024.10674336(252-256)Online publication date: 21-Jun-2024
https://doi.org/10.1109/SEAI62072.2024.10674336
Augusma AVaufreydaz DLetué F(2023)Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant FeaturesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3616546(750-754)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3616546
Show More Cited By

Index Terms

An Attention Model for Group-Level Emotion Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations

Recommendations

From individual to group-level emotion recognition: EmotiW 5.0
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Research in automatic affect recognition has come a long way. This paper describes the fifth Emotion Recognition in the Wild (EmotiW) challenge 2017. EmotiW aims at providing a common benchmarking platform for researchers working on different aspects ...
Group emotion recognition with individual facial emotion CNNs and global image based CNNs
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

This paper presents our approach for group-level emotion recognition in the Emotion Recognition in the Wild Challenge 2017. The task is to classify an image into one of the group emotion such as positive, neutral or negative. Our approach is based on ...
Cascade Attention Networks For Group Emotion Recognition with Face, Body and Image Cues
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

This paper presents our approach for group-level emotion recognition sub-challenge in the EmotiW 2018. The task is to classify an image into one of the group emotions such as positive, negative, and neutral. Our approach mainly explores three cues, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

October 2018

687 pages

ISBN:9781450356923

DOI:10.1145/3242969

General Chairs:
Sidney K. D'Mello
University of Illinois, USA
,
Panayiotis (Panos) Georgiou
University of Southern California, USA
,
Stefan Scherer
University of Southern California, USA
,
Program Chairs:
Emily Mower Provost
University of Michigan, USA
,
Mohammad Soleymani
University of Southern California, USA
,
Marcelo Worsley
Northwestern University, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: Specialist Interest Group in Computer-Human Interaction of the ACM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Conference

ICMI '18

Sponsor:

SIGCHI

ICMI '18: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 16 - 20, 2018

CO, Boulder, USA

Acceptance Rates

ICMI '18 Paper Acceptance Rate 63 of 149 submissions, 42%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
571
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang XZhang DLee D(2024)Implementing the Affective Mechanism for Group Emotion Recognition With a New Graph Convolutional Network ArchitectureIEEE Transactions on Affective Computing10.1109/TAFFC.2023.332010115:3(1104-1115)Online publication date: Jul-2024
https://doi.org/10.1109/TAFFC.2023.3320101
Xu JHuang X(2024)Group-Level Emotion Recognition Using Hierarchical Dual-Branch Cross Transformer with Semi-Supervised Learning2024 IEEE 4th International Conference on Software Engineering and Artificial Intelligence (SEAI)10.1109/SEAI62072.2024.10674336(252-256)Online publication date: 21-Jun-2024
https://doi.org/10.1109/SEAI62072.2024.10674336
Augusma AVaufreydaz DLetué F(2023)Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant FeaturesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3616546(750-754)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3616546
Wang XZhang DTan HLee D(2023)A Self-Fusion Network Based on Contrastive Learning for Group Emotion RecognitionIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.320224910:2(458-469)Online publication date: Apr-2023
https://doi.org/10.1109/TCSS.2022.3202249
Sharma GDhall ACai J(2023)Audio-Visual Automatic Group Affect AnalysisIEEE Transactions on Affective Computing10.1109/TAFFC.2021.310417014:2(1056-1069)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3104170
Veltmeijer EGerritsen CHindriks K(2023)Automatic Emotion Recognition for Groups: A ReviewIEEE Transactions on Affective Computing10.1109/TAFFC.2021.306572614:1(89-107)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TAFFC.2021.3065726
Nethi AMeda DReddy CKanth Koppala SNandy ABethireddy VSukhija SGupta Y(2023)Cohesive Group Emotion Recognition using Deep Learning2023 26th ACIS International Winter Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD-Winter)10.1109/SNPD-Winter57765.2023.10466291(264-269)Online publication date: 14-Dec-2023
https://doi.org/10.1109/SNPD-Winter57765.2023.10466291
Nethi AMeda DReddy CKanth Koppala SNandy ABethireddy VSukhija SGupta Y(2023)Cohesive Group Emotion Recognition using Deep Learning2023 IEEE/ACIS 8th International Conference on Big Data, Cloud Computing, and Data Science (BCD)10.1109/BCD57833.2023.10466291(264-269)Online publication date: 14-Dec-2023
https://doi.org/10.1109/BCD57833.2023.10466291
Rathod BVanzara RPandya D(2023)A recent survey on perceived group sentiment analysisJournal of Visual Communication and Image Representation10.1016/j.jvcir.2023.10398897(103988)Online publication date: Dec-2023
https://doi.org/10.1016/j.jvcir.2023.103988
Ognibene DDonabauer GTheophilou EBuršić SLomonaco FWilkens RHernández-Leo DKruschwitz U(2023)Moving Beyond Benchmarks and Competitions: Towards Addressing Social Media Challenges in an Educational ContextDatenbank-Spektrum10.1007/s13222-023-00436-323:1(27-39)Online publication date: 24-Feb-2023
https://doi.org/10.1007/s13222-023-00436-3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten