Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation

  • Conference paper
  • First Online:
Pattern Recognition (ACPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13188))

Included in the following conference series:

Abstract

Facial expression recognition (FER) is a topic attracting significant research in both psychology and machine learning with a wide range of applications. Despite a wealth of research on human FER and considerable progress in computational FER made possible by deep neural networks (DNNs), comparatively less work has been done on comparing the degree to which DNNs may be comparable to human performance. In this work, we compared the recognition performance and attention patterns of humans and machines during a two-alternative forced-choice FER task. Human attention was here gathered through click data that progressively uncovered a face, whereas model attention was obtained using three different popular techniques from explainable AI: CAM, GradCAM and Extremal Perturbation. In both cases, performance was gathered as percent correct. For this task, we found that humans outperformed machines quite significantly. In terms of attention patterns, we found that Extremal Perturbation had the best overall fit with the human attention map during the task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Publ. Interest 20(1), 1–68 (2019)

    Article  Google Scholar 

  2. Barsoum, E., Zhang, C., Ferrer, C.C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 279–283 (2016)

    Google Scholar 

  3. Buciluǎ, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541 (2006)

    Google Scholar 

  4. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. Ieee (2009)

    Google Scholar 

  5. Egner, S., Reimann, S., Hoeger, R., Zangemeister, W.H.: Attention and information acquisition: comparison of mouse-click with eye-movement attention tracking. J. Eye Move. Res. 11(6), 1–27 (2018)

    Google Scholar 

  6. Ekman, P., Keltner, D.: Universal facial expressions of emotion. In: Segerstrale U.P., Molnar P. (eds.) Nonverbal Communication: Where Nature Meets Culture, pp. 27–46 (1997)

    Google Scholar 

  7. Fan, X., Deng, Z., Wang, K., Peng, X., Qiao, Y.: Learning discriminative representation for facial expression recognition from uncertainties. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 903–907. IEEE (2020)

    Google Scholar 

  8. Fong, R., Patrick, M., Vedaldi, A.: Understanding deep networks via extremal perturbations and smooth masks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2950–2958 (2019)

    Google Scholar 

  9. Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F.: An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes. Pattern Recogn. 44(8), 1761–1776 (2011)

    Article  Google Scholar 

  10. Gendron, M., Roberson, D., van der Vyver, J.M., Barrett, L.F.: Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. Emotion 14(2), 251 (2014)

    Article  Google Scholar 

  11. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

  12. Jack, R.E., Blais, C., Scheepers, C., Schyns, P.G., Caldara, R.: Cultural confusions show that facial expressions are not universal. Curr. Biol. 19(18), 1543–1548 (2009)

    Article  Google Scholar 

  13. Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. 109(19), 7241–7244 (2012)

    Article  Google Scholar 

  14. Kim, N.W., et al.: Bubbleview: an interface for crowdsourcing image importance maps and tracking visual attention. ACM Trans. Comput.-Hum. Interact. (TOCHI) 24(5), 1–40 (2017)

    Article  Google Scholar 

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  16. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020)

    Google Scholar 

  17. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  18. Meng, D., Peng, X., Wang, K., Qiao, Y.: Frame attention networks for facial expression recognition in videos. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 3866–3870. IEEE (2019)

    Google Scholar 

  19. Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  20. Moon, H.J.: Facial expression processing with deep neural networks: from implementation to comparison with humans. Master’s thesis, Korea University, Seoul, Korea (2019)

    Google Scholar 

  21. Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 443–449 (2015)

    Google Scholar 

  22. Nusseck, M., Cunningham, D.W., Wallraven, C., Bülthoff, H.H.: The contribution of different facial regions to the recognition of conversational expressions. J. Vis. 8(8), 1–1 (2008)

    Article  Google Scholar 

  23. Pham, L., Vu, T.H., Tran, T.A.: Facial expression recognition using residual masking network. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4513–4519. IEEE (2021)

    Google Scholar 

  24. Pourmirzaei, M., Esmaili, F., Montazer, G.A.: Using self-supervised co-training to improve facial representation. arXiv preprint arXiv:2105.06421 (2021)

  25. Savchenko, A.V.: Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. arXiv preprint arXiv:2103.17107 (2021)

  26. Schiller, D., Huber, T., Dietz, M., André, E.: Relevance-based data masking: a model-agnostic transfer learning approach for facial expression recognition. Front. Comput. Sci. 2(6) (2020)

    Google Scholar 

  27. Schoneveld, L., Othmani, A., Abdelkawy, H.: Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recogn. Lett. (2021)

    Google Scholar 

  28. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

    Google Scholar 

  29. Vo, T.H., Lee, G.S., Yang, H.J., Kim, S.H.: Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8, 131988–132001 (2020)

    Article  Google Scholar 

  30. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)

    Google Scholar 

Download references

Acknowledgments

This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT).

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence, Korea University, Seoul, Korea

    Serin Park

  2. Department of Artificial Intelligence & Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea

    Christian Wallraven

Authors
  1. Serin Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Wallraven
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Wallraven .

Editor information

Editors and Affiliations

  1. Korea University, Seoul, Korea (Republic of)

    Christian Wallraven

  2. Nanjing University, Nanjing, China

    Qingshan Liu

  3. Osaka University, Osaka, Japan

    Hajime Nagahara

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Park, S., Wallraven, C. (2022). Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation. In: Wallraven, C., Liu, Q., Nagahara, H. (eds) Pattern Recognition. ACPR 2021. Lecture Notes in Computer Science, vol 13188. Springer, Cham. https://doi.org/10.1007/978-3-031-02375-0_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-02375-0_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-02374-3

  • Online ISBN: 978-3-031-02375-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation