research-article

Group-level emotion recognition using transfer learning from face identification

Authors:

Alexandr Rassadin,

Alexey Gruzdev,

Andrey SavchenkoAuthors Info & Claims

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Pages 544 - 548

https://doi.org/10.1145/3136755.3143007

Published: 03 November 2017 Publication History

Abstract

In this paper, we describe our algorithmic approach, which was used for submissions in the fifth Emotion Recognition in the Wild (EmotiW 2017) group-level emotion recognition sub-challenge. We extracted feature vectors of detected faces using the Convolutional Neural Network trained for face identification task, rather than traditional pre-training on emotion recognition problems. In the final pipeline an ensemble of Random Forest classifiers was learned to predict emotion score using available training set. In case when the faces have not been detected, one member of our ensemble extracts features from the whole image. During our experimental study, the proposed approach showed the lowest error rate when compared to other explored techniques. In particular, we achieved 75.4% accuracy on the validation data, which is 20% higher than the handcrafted feature-based baseline. The source code using Keras framework is publicly available.

References

[1]

E. Sariyanidi, H. Gunes, and A. Cavallaro. 2015. Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Transactions on PAMI, 37(6), pp. 1113–1133

[2]

S. Velusamy, H. Kannan, B. Anand, A. Sharma and B. Navathe. 2011. A method to infer emotions from facial action units. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2028-2031

[3]

Y. Hu, Z. Zeng, L. Yin, X. Wei, J. Tu and T. S. Huang. 2008. A study of nonfrontal-view facial expressions recognition. In Proceedings of International Conference on Pattern Recognition (ICPR), pp. 1-4

[4]

B. Kim, H. Lee, J. Roh and S. Lee. 2015. Hierarchical Committee of Deep CNNs with Exponentially-Weighted Decision Fusion for Static Facial Expression Recognition. In Proceedings of ACM International Conference on Multimodal Interaction (ICMI), pp. 427-434

Digital Library

[5]

G. Levi, and T. Hassner. 2015. Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In Proceedings of ACM on International Conference on Multimodal Interaction (ICMI), pp. 503- 510

Digital Library

[6]

O. M. Parkhi, A. Vedaldi, A. Zisserman. 2015. Deep face recognition. In Proceedings of the British Machine Vision Conference, pp. 1-12

[7]

A. Dhall, J. Joshi, K. Sikka, R. Goecke and N. Sebe. 2015. The more the merrier: Analysing the affect of a group of people in images. In Proceedings of IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1-8

[8]

X. Huang, A. Dhall, X. Liu, G. Zhao, J. Shi, R. Goecke, M. Pietikäinen. 2016. Analyzing the affect of a group of people using multi-modal framework. arXiv preprint arXiv:1610.03640

[9]

X. Huang, A. Dhall, G. Zhao, R. Goecke, M. Pietikäinen. 2015. Riesz-based Volume Local Binary Pattern and A Novel Group Expression Model for Group Happiness Intensity Analysis. In Proceedings of British Machine Vision Conference, pp. 34.1-34.13

[10]

A. Dhall, R. Goecke and T. Gedeon. 2015. Automatic Group Happiness Intensity Analysis. IEEE Transactions on Affective Computing, vol. 6, no. 1, pp. 13-26

Digital Library

[11]

J. Li, S. Roy, J. Feng and T. Sim. 2016. Happiness level prediction with sequential inputs via multiple regressions. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), pp. 487-493

Digital Library

[12]

V. Vonikakis, Y. Yazici, V. D. Nguyen and S. Winkler. 2016. Group happiness assessment using geometric features and dataset balancing. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), pp. 479- 486

Digital Library

[13]

B. Sun, Q.Wei, L. Li, Q. Xu, J. He and L.Yu. 2016. LSTM for dynamic emotion and group emotion recognition in the wild. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), pp. 451-457

Digital Library

[14]

A. Dhall, R. Goecke, S. Ghosh, J. Joshi, J. Hoey and T. Gedeon. 2017. From Individual to Group-level Emotion Recognition: EmotiW 5.0. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI)

Digital Library

[15]

A.V. Savchenko. 2016. Search Techniques in Intelligent Classification Systems. Springer, ISBN: 978-3-319-30515-8

Digital Library

[16]

A.V. Savchenko. 2017. Maximum-likelihood approximate nearest neighbor method in real-time image recognition. Pattern Recognition, vol. 61, pp. 459- 469

Digital Library

[17]

P. Viola and M. Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511- 518.

[18]

N. Dalal and B. Triggs. 2005. Histograms of oriented gradients for human detection. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 886-893

Digital Library

[19]

P. Hu and D. Ramanan. 2017. Finding Tiny Faces. arXiv preprint arXiv:1612.04402

[20]

K. He, X. Zhang, S. Ren and J. Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778

[21]

P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus and Y. LeCun. 2013. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229.

[22]

A. V. Savchenko. 2017. Deep Convolutional Neural Networks and Maximum-Likelihood Principle in Approximate Nearest Neighbor Search. In Proceedings of Iberian Conference on Pattern Recognition and Image Analysis, L.A. Alexandre et al. (Eds.). Lecture Notes in Computer Science, vol. 10255.

[23]

Springer, pp. 42–49.

[24]

Ian J. Goodfellow et al. 2013. Challenges in Representation Learning: A report on three machine learning contests. Neural Networks, vol. 64, pp. 59-63

Digital Library

[25]

H. Kaya, F. Gürpınar, A. A. Salah. 2017. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image and Vision Computing,

Digital Library

[26]

V. Kazemi and J. Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1867- 1874

Digital Library

[27]

K. Simonyan and A. Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

[28]

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), vol. 115, pp. 211-252,

Digital Library

[29]

F. Chollet. 2016. Xception: Deep Learning with Depthwise Separable Convolutions. arXiv preprint arXiv:1610.02357.

Cited By

Gong WWang YWu YGao SVasilakos AZhang P(2025)A Hybrid Fusion Model for Group-Level Emotion Recognition in Complex ScenariosInformation Sciences10.1016/j.ins.2025.121968(121968)Online publication date: Feb-2025
https://doi.org/10.1016/j.ins.2025.121968
Xu JHuang X(2024)Group-Level Emotion Recognition Using Hierarchical Dual-Branch Cross Transformer with Semi-Supervised Learning2024 IEEE 4th International Conference on Software Engineering and Artificial Intelligence (SEAI)10.1109/SEAI62072.2024.10674336(252-256)Online publication date: 21-Jun-2024
https://doi.org/10.1109/SEAI62072.2024.10674336
Mallegowda MKumaran SAditya Raj VKumar SGowda R(2024)Advancing Road Safety: Deep Learning-Powered Real-Time Driver State Assessment and R-CNN for Proximity Vehicle MonitoringProceedings of Fifth Doctoral Symposium on Computational Intelligence10.1007/978-981-97-6036-7_38(463-477)Online publication date: 4-Oct-2024
https://doi.org/10.1007/978-981-97-6036-7_38
Show More Cited By

Index Terms

Group-level emotion recognition using transfer learning from face identification
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
  2. Machine learning

Recommendations

EmotiW 2016: video and group-level emotion recognition challenges
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

This paper discusses the baseline for the Emotion Recognition in the Wild (EmotiW) 2016 challenge. Continuing on the theme of automatic affect recognition `in the wild', the EmotiW challenge 2016 consists of two sub-challenges: an audio-video based ...
From individual to group-level emotion recognition: EmotiW 5.0
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Research in automatic affect recognition has come a long way. This paper describes the fifth Emotion Recognition in the Wild (EmotiW) challenge 2017. EmotiW aims at providing a common benchmarking platform for researchers working on different aspects ...
Group emotion recognition with individual facial emotion CNNs and global image based CNNs
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

This paper presents our approach for group-level emotion recognition in the Emotion Recognition in the Wild Challenge 2017. The task is to classify an image into one of the group emotion such as positive, neutral or negative. Our approach is based on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

November 2017

676 pages

ISBN:9781450355438

DOI:10.1145/3136755

General Chairs:
Edward Lank
University of Waterloo, Canada
,
Alessandro Vinciarelli
University of Glasgow, UK
,
Program Chairs:
Eve Hoggan
Aarhus University, Denmark
,
Sriram Subramanian
University of Sussex, UK
,
Stephen A. Brewster
University of Glasgow, UK

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13 - 17, 2017

Glasgow, UK

Acceptance Rates

ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
533
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)5

Reflects downloads up to 20 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gong WWang YWu YGao SVasilakos AZhang P(2025)A Hybrid Fusion Model for Group-Level Emotion Recognition in Complex ScenariosInformation Sciences10.1016/j.ins.2025.121968(121968)Online publication date: Feb-2025
https://doi.org/10.1016/j.ins.2025.121968
Xu JHuang X(2024)Group-Level Emotion Recognition Using Hierarchical Dual-Branch Cross Transformer with Semi-Supervised Learning2024 IEEE 4th International Conference on Software Engineering and Artificial Intelligence (SEAI)10.1109/SEAI62072.2024.10674336(252-256)Online publication date: 21-Jun-2024
https://doi.org/10.1109/SEAI62072.2024.10674336
Mallegowda MKumaran SAditya Raj VKumar SGowda R(2024)Advancing Road Safety: Deep Learning-Powered Real-Time Driver State Assessment and R-CNN for Proximity Vehicle MonitoringProceedings of Fifth Doctoral Symposium on Computational Intelligence10.1007/978-981-97-6036-7_38(463-477)Online publication date: 4-Oct-2024
https://doi.org/10.1007/978-981-97-6036-7_38
Dhall ASingh MGoecke RGedeon TZeng DWang YIkeda K(2023)EmotiW 2023: Emotion Recognition in the Wild ChallengeProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3616545(746-749)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3616545
Sharma GDhall ACai J(2023)Audio-Visual Automatic Group Affect AnalysisIEEE Transactions on Affective Computing10.1109/TAFFC.2021.310417014:2(1056-1069)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3104170
Veltmeijer EGerritsen CHindriks K(2023)Automatic Emotion Recognition for Groups: A ReviewIEEE Transactions on Affective Computing10.1109/TAFFC.2021.306572614:1(89-107)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TAFFC.2021.3065726
Nethi AMeda DReddy CKanth Koppala SNandy ABethireddy VSukhija SGupta Y(2023)Cohesive Group Emotion Recognition using Deep Learning2023 26th ACIS International Winter Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD-Winter)10.1109/SNPD-Winter57765.2023.10466291(264-269)Online publication date: 14-Dec-2023
https://doi.org/10.1109/SNPD-Winter57765.2023.10466291
Rathod BVanazara RPandya D(2023)Improved Group Facial Expression Recognition Using Super-Resolved Local Facial Multi Scale Features2023 11th International Conference on Intelligent Systems and Embedded Design (ISED)10.1109/ISED59382.2023.10444587(1-6)Online publication date: 15-Dec-2023
https://doi.org/10.1109/ISED59382.2023.10444587
Nethi AMeda DReddy CKanth Koppala SNandy ABethireddy VSukhija SGupta Y(2023)Cohesive Group Emotion Recognition using Deep Learning2023 IEEE/ACIS 8th International Conference on Big Data, Cloud Computing, and Data Science (BCD)10.1109/BCD57833.2023.10466291(264-269)Online publication date: 14-Dec-2023
https://doi.org/10.1109/BCD57833.2023.10466291
Malhotra ASharma GKumar RDhall AHoey J(2023)Social Event Context and Affect Prediction in Group Videos2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)10.1109/ACIIW59127.2023.10388162(1-8)Online publication date: 10-Sep-2023
https://doi.org/10.1109/ACIIW59127.2023.10388162
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten