research-article

Fast and Light Manifold CNN based 3D Facial Expression Recognition across Pose Variations

Authors:

Liming ChenAuthors Info & Claims

MM '18: Proceedings of the 26th ACM international conference on Multimedia

Pages 229 - 238

https://doi.org/10.1145/3240508.3240568

Published: 15 October 2018 Publication History

Abstract

This paper proposes a novel approach to 3D Facial Expression Recognition (FER), and it is based on a Fast and Light Manifold CNN model, namely FLM-CNN. Different from current manifold CNNs, FLM-CNN adopts a human vision inspired pooling structure and a multi-scale encoding strategy to enhance geometry representation, which highlights shape characteristics of expressions and runs efficiently. Furthermore, a sampling tree based preprocessing method is presented, and it sharply saves memory when applied to 3D facial surfaces, without much information loss of original data. More importantly, due to the property of manifold CNN features of being rotation-invariant, the proposed method shows a high robustness to pose variations. Extensive experiments are conducted on BU-3DFE, and state-of-the-art results are achieved, indicating its effectiveness.

References

[1]

Amal Azazi, Syaheerah Lebai Lutfi, Ibrahim Venkat, and Fernando Fernández-Martínez. 2015. Towards a robust affect recognition: Automatic facial expression recognition in 3D faces. Expert Syst. Appl. Vol. 42, 6 (2015), 3056--3066.

Digital Library

[2]

Stefano Berretti, Alberto Del Bimbo, Pietro Pala, Boulbaba Ben Amor, and Mohamed Daoudi. 2010. A Set of Selected SIFT Features for 3D Facial Expression Recognition> In IEEE International Conference on Pattern Recognition. 4125--4128.

Digital Library

[3]

Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael M. Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In Advances in Neural Information Processing Systems. 3189--3197.

Digital Library

[4]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In International Conference on Learning Representations.

[5]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In Advances in Neural Information Processing Systems. 3837--3845.

Digital Library

[6]

Dmytro Derkach and Federico M. Sukno. 2017. Local Shape Spectrum Analysis for 3D Facial Expression Recognition. In 12th IEEE International Conference on Automatic Face and Gesture Recognition. 41--47.

[7]

Boqing Gong, Yueming Wang, Jianzhuang Liu, and Xiaoou Tang. 2009. Automatic facial expression recognition on a single 3D face by exploring shape deformation ACM International Conference on Multimedia. 569--572.

Digital Library

[8]

Di Huang, Chao Zhu, Yunhong Wang, and Liming Chen. 2014. HSOG: A Novel Local Image Descriptor Based on Histograms of the Second-Order Gradients. IEEE Trans. Image Processing Vol. 23, 11 (2014), 4680--4695.

[9]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning. 448--456.

Digital Library

[10]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.

[11]

Jan J. Koenderink and Andrea J. van Doorn. 1992. Surface shape and curvature scales. Image Vision Comput. Vol. 10, 8 (1992), 557--564.

Digital Library

[12]

Iasonas Kokkinos, Michael M. Bronstein, Roee Litman, and Alexander M. Bronstein. 2012. Intrinsic shape context descriptors for deformable shapes. In IEEE Conference on Computer Vision and Pattern Recognition. 159--166.

Digital Library

[13]

Yann Lécun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE Vol. 86, 11 (1998), 2278--2324.

[14]

Pierre Lemaire, Boulbaba Ben Amor, Mohsen Ardabilian, Liming Chen, and Mohamed Daoudi. 2011. Fully automatic 3D facial expression recognition using a region-based approach. In ACM Workshop on Human Gesture and Behavior Understanding. 53--58.

Digital Library

[15]

Pierre Lemaire, Mohsen Ardabilian, Liming Chen, and Mohamed Daoudi. 2013. Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients. In IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. 1--7.

[16]

Huibin Li, Liming Chen, Di Huang, Yunhong Wang, and Jean-Marie Morvan. 2012. 3D facial expression recognition via multiple kernel learning of Multi-Scale Local Normal Patterns. In IEEE International Conference on Pattern Recognition. 2577--2580.

[17]

Huibin Li, Huaxiong Ding, Di Huang, Yunhong Wang, Xi Zhao, Jean-Marie Morvan, and Liming Chen. 2015. An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition. Computer Vision and Image Understanding Vol. 140 (2015), 83--92.

Digital Library

[18]

Huibin Li, Jian Sun, Dong Wang, Zongben Xu, and Liming Chen. 2015. Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition. CoRR Vol. abs/1511.03015 (2015).

[19]

Huibin Li, Jian Sun, Zongben Xu, and Liming Chen. 2017. Multimodal 2D + 3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network. IEEE Trans. Multimedia Vol. 19, 12 (2017), 2816--2831.

[20]

Yangyan Li, Rui Bu, Mingchao Sun, and Baoquan Chen. 2018. PointCNN. CoRR Vol. abs/1801.07791 (2018).

[21]

Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network In Network. CoRR Vol. abs/1312.4400 (2013).

[22]

Ahmed Maalej, Boulbaba Ben Amor, Mohamed Daoudi, Anuj Srivastava, and Stefano Berretti. 2011. Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognition Vol. 44, 8 (2011), 1581--1589.

Digital Library

[23]

Jonathan Masci, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. 2015. Geodesic Convolutional Neural Networks on Riemannian Manifolds. In IEEE International Conference on Computer Vision Workshops. 832--840.

Digital Library

[24]

Daniel Maturana and Sebastian Scherer. 2015. VoxNet: A 3D Convolutional Neural Network for real-time object recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems. 922--928.

[25]

Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. 2017. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In IEEE Conference on Computer Vision and Pattern Recognition. 5425--5434.

[26]

Iordanis Mpiperis, Sotiris Malassiotis, and Michael G. Strintzis. 2008. Bilinear Models for 3-D Face and Facial Expression Recognition. IEEE Transactions on Information Forensics and Security Vol. 3, 3 (2008), 498--511.

Digital Library

[27]

Oyebade K. Oyedotun, Girum G. Demisse, Abd El Rahman Shabayek, Djamila Aouada, and Björn E. Ottersten. 2017. Facial Expression Recognition via Joint Deep Learning of RGB-Depth Map Latent Representations. In IEEE International Conference on Computer Vision Workshops. 3161--3168.

[28]

Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition. 77--85.

[29]

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In Advances in Neural Information Processing Systems. 5105--5114.

[30]

Yipeng Qin, Xiaoguang Han, Hongchuan Yu, Yizhou Yu, and Jian-Jun Zhang. 2016. Fast and exact discrete geodesic computation based on triangle-oriented wavefront propagation. ACM Trans. Graph. Vol. 35, 4 (2016), 125:1--125:13.

Digital Library

[31]

Subramanian Ramanathan, Ashraf A. Kassim, Y. V. Venkatesh, and Wu Sin Wah. 2006. Human Facial Expression Recognition using a 3D Morphable Model. In IEEE International Conference on Image Processing. 661--664.

[32]

Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. 2017. OctNet: Learning Deep 3D Representations at High Resolutions. In IEEE Conference on Computer Vision and Pattern Recognition. 6620--6629.

[33]

Patrice Y. Simard, David Steinkraus, and John C. Platt. 2003. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In International Conference on Document Analysis and Recognition. 958--962.

Digital Library

[34]

Hamit Soyel and Hasan Demirel. 2007. Facial Expression Recognition Using 3D Facial Feature Distances. In International Conference on Image Analysis and Recognition. 831--838.

Digital Library

[35]

Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. 2015. Multi-view Convolutional Neural Networks for 3D Shape Recognition. In IEEE International Conference on Computer Vision. 945--953.

Digital Library

[36]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition. 1--9.

[37]

Hao Tang and Thomas S. Huang. 2008. 3D facial expression recognition based on automatically selected features. In IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1--8.

[38]

Engin Tola, Vincent Lepetit, and Pascal Fua. 2010. DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo. IEEE Trans. Pattern Anal. Mach. Intell. Vol. 32, 5 (2010), 815--830.

Digital Library

[39]

Jun Wang, Lijun Yin, Xiaozhou Wei, and Yi Sun. 2006. 3D Facial Expression Recognition Based on Primitive Surface Feature Distribution. In IEEE Conference on Computer Vision and Pattern Recognition. 1399--1406.

Digital Library

[40]

Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017. O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. Vol. 36, 4 (2017), 72:1--72:11.

Digital Library

[41]

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition. 1912--1920.

[42]

Huiyuan Yang and Lijun Yin. 2017. CNN based 3D facial expression recognition using masking and landmark features. In International Conference on Affective Computing and Intelligent Interaction. 556--560.

[43]

Xudong Yang, Di Huang, Yunhong Wang, and Liming Chen. 2015. Automatic 3D facial expression recognition using geometric scattering representation. In IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. 1--6.

[44]

Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J. Rosato. 2006. A 3D Facial Expression Database For Facial Behavior Research. In IEEE International Conference on Automatic Face and Gesture Recognition. 211--216.

Digital Library

[45]

Wei Zeng, Huibin Li, Liming Chen, Jean-Marie Morvan, and Xianfeng David Gu. 2013. An automatic 3D expression recognition framework based on sparse representation of conformal images. In IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. 1--8.

[46]

Xi Zhao, Di Huang, Emmanuel Dellandréa, and Liming Chen. 2010. Automatic 3D Facial Expression Recognition Based on a Bayesian Belief Net and a Statistical Facial Feature Model. In IEEE International Conference on Pattern Recognition. 3724--3727.

Digital Library

[47]

Qingkai Zhen, Di Huang, Yunhong Wang, and Liming Chen. 2015. Muscular Movement Model Based Automatic 3D Facial Expression Recognition. In International Conference on MultiMedia Modeling. 522--533.

Cited By

Li HYang HHuang D(2024)DrFER: Learning Disentangled Representations for 3D Facial Expression Recognition2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581989(1-8)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581989
Jin ZZhang XWang JXu XXiao J(2023)Fine-Grained Facial Expression Recognition in Multiple SmilesElectronics10.3390/electronics1205108912:5(1089)Online publication date: 22-Feb-2023
https://doi.org/10.3390/electronics12051089
Ni RYang BZhou XCangelosi ALiu X(2023)Facial Expression Recognition Through Cross-Modality Attention FusionIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2022.315001915:1(175-185)Online publication date: Mar-2023
https://doi.org/10.1109/TCDS.2022.3150019
Show More Cited By

Index Terms

Fast and Light Manifold CNN based 3D Facial Expression Recognition across Pose Variations
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns

3D facial expression recognition has been greatly promoted for overcoming the inherent drawbacks of 2D facial expression recognition and has achieved superior recognition accuracy to the 2D. In this paper, a novel holistic, full-automatic approach for ...
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...
Pose-Invariant Facial Expression Recognition Based on 3D Face Morphable Model and Domain Adversarial Learning
Image and Graphics
Abstract
Pose is one of the most important factors affecting performance of face related recognition algorithms including facial expression recognition (FER). Traditionally, non-frontal FER is conducted by either performing face formalization or designing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '18: Proceedings of the 26th ACM international conference on Multimedia

October 2018

2167 pages

ISBN:9781450356657

DOI:10.1145/3240508

General Chairs:
Susanne Boll
University of Oldenburg, Germany
,
Kyoung Mu Lee
Seoul National University, Korea
,
Jiebo Luo
University of Rochester, USA
,
Wenwu Zhu
Tsinghua University, China
,
Program Chairs:
Hyeran Byun
Yonsei University, Korea
,
Chang Wen Chen
State Univ. Of New York at Buffalo, USA
,
Rainer Lienhart
University of Augsburg, Germany
,
Tao Mei
JD AI, China

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

The Microsoft Research Asia Collaborative Program
The National Natural Science Foundation of China

Conference

MM '18

Sponsor:

SIGMM

MM '18: ACM Multimedia Conference

October 22 - 26, 2018

Seoul, Republic of Korea

Acceptance Rates

MM '18 Paper Acceptance Rate 209 of 757 submissions, 28%;

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
601
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)5

Reflects downloads up to 14 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li HYang HHuang D(2024)DrFER: Learning Disentangled Representations for 3D Facial Expression Recognition2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581989(1-8)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581989
Jin ZZhang XWang JXu XXiao J(2023)Fine-Grained Facial Expression Recognition in Multiple SmilesElectronics10.3390/electronics1205108912:5(1089)Online publication date: 22-Feb-2023
https://doi.org/10.3390/electronics12051089
Ni RYang BZhou XCangelosi ALiu X(2023)Facial Expression Recognition Through Cross-Modality Attention FusionIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2022.315001915:1(175-185)Online publication date: Mar-2023
https://doi.org/10.1109/TCDS.2022.3150019
Lutfi SLahasan BLuna-Jiménez CBamasood ZAkhtar Z(2023)Effects of Facial Expressions and Gestures on the Trustworthiness of a PersonIEEE Access10.1109/ACCESS.2023.333427011(133891-133902)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3334270
R. VNag P(2023)Text-based emotion recognition using contextual phrase embedding modelMultimedia Tools and Applications10.1007/s11042-023-14524-982:23(35329-35355)Online publication date: 16-Mar-2023
https://doi.org/10.1007/s11042-023-14524-9
Huang DYang H(2023)3D Face RecognitionHandbook of Face Recognition10.1007/978-3-031-43567-6_15(433-468)Online publication date: 30-Dec-2023
https://doi.org/10.1007/978-3-031-43567-6_15
Owusu EAppati JOkae P(2022)Robust facial expression recognition system in higher posesVisual Computing for Industry, Biomedicine, and Art10.1186/s42492-022-00109-05:1Online publication date: 16-May-2022
https://doi.org/10.1186/s42492-022-00109-0
Li SDeng W(2022)Deep Facial Expression Recognition: A SurveyIEEE Transactions on Affective Computing10.1109/TAFFC.2020.298144613:3(1195-1215)Online publication date: 1-Jul-2022
https://doi.org/10.1109/TAFFC.2020.2981446
Zhu ZSui MLi HZhao F(2022)CMANET: Curvature-Aware Soft Mask Guided Attention Fusion Network for 2D+3D Facial Expression Recognition2022 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME52920.2022.9859837(1-6)Online publication date: 18-Jul-2022
https://doi.org/10.1109/ICME52920.2022.9859837
Fu YDeng YZhang YYang ZQi R(2022)Low Rank Tucker Decomposition for 2D+3D Facial Expression RecognitionProcedia Computer Science10.1016/j.procs.2021.12.276198(499-504)Online publication date: 2022
https://doi.org/10.1016/j.procs.2021.12.276
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents