research-article

Multi-View Facial Expression Recognition with Multi-View Facial Expression Light Weight Network

Authors:

Qian YongshengAuthors Info & Claims

Pattern Recognition and Image Analysis, Volume 30, Issue 4

Pages 805 - 814

https://doi.org/10.1134/S1054661820040197

Published: 01 October 2020 Publication History

Abstract

Abstract

Facial expression recognition for frontal faces has become a well-established research area in the last two decades. However, non-frontal facial expression recognition hasn’t been paid much attention until recently. In this paper, we propose an MVFE-LightNet (Multi-View Facial Expression Light Weight Network) for multi-view facial expression recognition. To this end, we first applied MTCNN for facial detection and alignment and then did preprocessing like normalization and data augmentation. Finally, we put the images into MVFE-LightNet to extract sub-space features of facial expressions with various poses. A depthwise separable residual convolution module architecture was designed to reduce the parameters of the model and lessen the chance of overfitting. Experiments were implemented on Radboud Faces Database and BU-3DFE dataset. We demonstrated that our method could effectively improve the recognition accuracy, and achieved the accuracy of 95.6% and 88.7% respectively for the Radboud and BU-3DFE.

References

[1]

H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for facial expression recognition”, in Proceedings of the IEEE International Conference on Computer Vision (Santiago, 2015), pp. 2983–2991.

[2]

N. Hesse, T. Gehrig, H. Gao, and H. K. Ekenel, “Multi-view facial expression recognition using local appearance features,” in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012) (Tsukuba, 2012), pp. 3533–3536.

[3]

O. Rudovic, I. Patras, and M. Pantic, “Coupled Gaussian process regression for pose-invariant facial expression recognition,” in Computer Vision–ECCV 2010 (Crete, 2010), pp. 350–363.

[4]

Moore S. and Bowden R. Local binary patterns for multiview facial expression recognition Comput. Vision Image Understanding 2011 115 541-558

[5]

Shan C., Gong S., and McOwan P. W. Facial expression recognition based on local binary patterns: A comprehensive study Image Vision Comput. 2009 27 803-816

[6]

G. Fanelli, A. Yao, P.-L. Noel, J. Gall, and L. Van Gool, “Hough forest-based facial expression recognition from video sequences,” in European Conference on Computer Vision (Crete, 2010), pp. 195–206.

[7]

Ding Y., Zhao Q., Li B., and Yuan X. Facial expression recognition from image sequence based on LBP and Taylor expansion IEEE Access 2017 5 19409-19419

[8]

Ryu B., Rivera A. R., Kim J., and Chae O. Local directional ternary pattern for facial expression recognition IEEE Trans. Image Process. 2017 26 6006-6018

[9]

O. Rudovic, I. Patras, and M. Pantic, “Regression-based multi-view facial expression recognition,” in 2010 20th International Conference on Pattern Recognition (Istanbul, 2010), pp. 4121–4124.

[10]

Mao Q., Rao Q., Yu Y., and Dong M. Hierarchical Bayesian theme models for multi-pose facial expression recognition IEEE Trans. Multimedia 2017 19 861-873

[11]

Zhang T., Zheng W., Cui Z., Zong Y., Yan J., and Yan K. A deep neural network-driven feature learning method for multi-view facial expression recognition IEEE Trans. Multimedia 2016 18 2528-2536

[12]

A. Mollahosseini, D. Chan, and M. H. Mahoor, “Going deeper in facial expression recognition using deep neural networks,” in IEEE Winter Conference on Applications of Computer Vision (2016), pp. 1–10.

[13]

D. V. Sang, L. T. B. Cuong, and P. T. Ha, “Discriminative deep feature learning for facial emotion recognition,” in 2018 1st International Conference on Multimedia Analysis and Pattern Recognition (MAPR) (Ho Chi Minh City, 2018), pp. 1–6.

[14]

K. Shan, J. Guo, W. You, D. Lu, and R. Bie, “Automatic facial expression recognition based on a deep convolutional-neural-network structure,” in 2017IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA) (London 2017), pp. 123–128.

[15]

W. Ge and Y. Yu, “Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (Honolulu, HI, 2017).

[16]

Y. Zhou and B. E. Shi, “Action unit selective feature maps in deep networks for facial expression recognition,” in 2017 International Joint Conference on Neural Networks (IJCNN) (Anchorage, AK, 2017), pp. 2031–2038.

[17]

J. Luttrell et al., “A deep transfer learning approach to fine-tuning facial recognition models,” in 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA) (Wuhan, 2018), pp. 2671–2676.

[18]

J. Xiang and G. Zhu, “Joint face detection and facial expression recognition with MTCNN,” in 2017 4th International Conference on Information Science and Control Engineering (Changsha, 2017), pp. 424–427.

[19]

Halevy A., Norvig P., and Pereira F. The unreasonable effectiveness of data Intell. Syst. 2009 24 8-12

[20]

X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” in International Conference on Artificial Intelligence and Statistics (Ft. Lauderdale, 2012), pp. 315–323.

[21]

S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International Conference on Machine Learning (Lille, 2015), pp. 448–456.

[22]

Langner O., Dotsch R., Bijlstra G., Wigboldus D., Hawk S., and Knippenberg A. Presentation and validation of the Radboud Faces Database Cognit. Emotion 2010 24 1377-1388

[23]

L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato, “A 3D facial expression database for facial behavior research,” in Automatic face and Gesture Recognition 2006. FGR 2006. 7th International Conference on (Southampton, 2006), pp. 211–216.

[24]

F. Chollet, Keras (2015). https://github.com/fchollet/keras.

[25]

W. Zheng, H. Tang, Z. Lin, et al., “A novel approach to expression recognition from non-frontal face images,” in IEEE International Conference on Computer Vision (Kyoto, 2011), pp. 1901–1908.

[26]

Zheng Wenming Multi-view facial expression recognition based on group sparse IEEE Trans. Affective Comput. 2014 5 71-85

Cited By

Roy SEtemad A(2023)Contrastive Learning of View-invariant Representations for Facial Expressions RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363296020:4(1-22)Online publication date: 14-Nov-2023
https://dl.acm.org/doi/10.1145/3632960
Chaudhari ABhatt CNguyen TPatel NChavda KSarda K(2023)Emotion Recognition System via Facial Expressions and Speech Using Machine Learning and Deep Learning TechniquesSN Computer Science10.1007/s42979-022-01633-94:4Online publication date: 28-Apr-2023
https://dl.acm.org/doi/10.1007/s42979-022-01633-9
Savchenko ASavchenko L(2022)Audio-Visual Continuous Recognition of Emotional State in a Multi-User System Based on Personalized Representation of Facial Expressions and VoicePattern Recognition and Image Analysis10.1134/S105466182203039732:3(665-671)Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1134/S1054661822030397

Index Terms

Multi-View Facial Expression Recognition with Multi-View Facial Expression Light Weight Network
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Local binary patterns for multi-view facial expression recognition

Research into facial expression recognition has predominantly been applied to face images at frontal view only. Some attempts have been made to produce pose invariant facial expression classifiers. However, most of these attempts have only considered ...
Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...

Comments

Information & Contributors

Information

Published In

cover image Pattern Recognition and Image Analysis

Pattern Recognition and Image Analysis Volume 30, Issue 4

Oct 2020

271 pages

ISSN:1054-6618

Issue’s Table of Contents

© Pleiades Publishing, Ltd. 2020. ISSN 1054-6618, Pattern Recognition and Image Analysis, 2020, Vol. 30, No. 4, pp. 805–814. © Pleiades Publishing, Ltd., 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 October 2020

Accepted: 27 July 2020

Revision received: 26 July 2020

Received: 26 April 2020

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Roy SEtemad A(2023)Contrastive Learning of View-invariant Representations for Facial Expressions RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363296020:4(1-22)Online publication date: 14-Nov-2023
https://dl.acm.org/doi/10.1145/3632960
Chaudhari ABhatt CNguyen TPatel NChavda KSarda K(2023)Emotion Recognition System via Facial Expressions and Speech Using Machine Learning and Deep Learning TechniquesSN Computer Science10.1007/s42979-022-01633-94:4Online publication date: 28-Apr-2023
https://dl.acm.org/doi/10.1007/s42979-022-01633-9
Savchenko ASavchenko L(2022)Audio-Visual Continuous Recognition of Emotional State in a Multi-User System Based on Personalized Representation of Facial Expressions and VoicePattern Recognition and Image Analysis10.1134/S105466182203039732:3(665-671)Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1134/S1054661822030397

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents