Abstract
Facial Expression Recognition has gained considerable attention in the field of affective computing, but only a few works considered the intensity of emotion embedded in the expression. Even the available studies on expression intensity estimation successfully assigned a nominal/regression value or classified emotion in a range of intervals. The approaches from multiclass and its extensions do not conform to man heuristic manner of recognising emotion with the respective intensity. This work is presenting a Multi-label CNN-based model which could simultaneously recognise emotion and also provide ordinal metrics as the intensity of the emotion. In the experiments conducted on BU-3DFE and Cohn Kanade (CK+) datasets, we check how well our model could adapt and generalise. Our model gives promising results with multilabel evaluation metrics and generalise well when trained on BU-3DFE and evaluated on CK+.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chang, K.Y., Chen, C.S., Hung, Y.P.: Intensity rank estimation of facial expressions based on a single image (2013). https://doi.org/10.1109/SMC.2013.538
Dhall, A., Goecke, R., Gedeon, T.: Automatic group happiness intensity analysis. IEEE Trans. Affect. Comput. 6(1), 13–26 (2015). https://doi.org/10.1109/TAFFC.2015.2397456
Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Social Psychol. 17(2), 124–129 (1971). https://doi.org/10.1037/h0030377
Georgescu, M.I., Ionescu, R.T., Popescu, M.: Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7, 64827–64836 (2019). https://doi.org/10.1109/ACCESS.2019.2917266
Ghayoumi, M.: A quick review of deep learning in facial expression. J. Commun. Comput. 14, 34–38 (2017). https://doi.org/10.17265/1548-7709/2017.01.004
Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensor 13, 7714–7734 (2013). https://doi.org/10.3390/s130607714
Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 551, 541–551 (1989). https://doi.org/10.1162/neco.1989.1.4.541
Huang, Y., Chen, F., Lv, S., Wang, X.: Facial expression recognition: a survey. SS Symmetry 11, 1189 (2019). https://doi.org/10.3390/sym11101189
Khairuni, S., Kamarol, A., Hisham, M., Kälviäinen, H., Parkkinen, J., Parthiban, R.: Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recogn. Lett. 92, 25–32 (2017). https://doi.org/10.1016/j.patrec.2017.04.003
Li, S., Deng, W.: Blended emotion in-the-wild: multi-label facial expression recognition using crowdsourced annotations and deep locality feature learning. Int. J. Comput. Vis. 127, 884–906 (2018). https://doi.org/10.1007/s11263-018-1131-1
Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 3045(c), 1–20 (2020). https://doi.org/10.1109/TAFFC.2020.2981446
Liu, H., Lu, J., Feng, J., Zhou, J.: Ordinal deep feature learning for facial age estimation. In: Proceedings - 12th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2017–1st International Workshop on Adaptive Shot Learning for Gesture Understanding and Production, ASL4GUP 2017, Biometrics in the Wild, Bwild 2017, Heteroge, pp. 157–164 (2017). https://doi.org/10.1109/FG.2017.28
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, July, pp. 94–101 (2010). https://doi.org/10.1109/CVPRW.2010.5543262
Mahmood, Z., Muhammad, N.: A review on state-of-the-art face recognition approaches. Fractals 25(2), 1–19 (2017). https://doi.org/10.1142/S0218348X17500256
Martinez, B., Valstar, M.F.: Advances, challenges, and opportunities in automatic facial expression recognition. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 63–100. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-25958-1_4
Mayya, V., Pai, R.M., Pai, M.M.: Automatic facial expression recognition using DCNN. Proc. - Proc. Comput. Sci. 93(September), 453–461 (2016). https://doi.org/10.1016/j.procs.2016.07.233
Xia, X.: Facial expression recognition based on monogenic binary coding. Appl. Mech. Mater. 512, 437–440 (2014). https://doi.org/10.4028/www.scientific.net/AMM.511-512.437. Scientific Net
Nomiya, H., Sakaue, S., Hochin, T.: Recognition and intensity estimation of facial expression using ensemble classifiers. In: 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS), pp. 1–6 (2016). https://doi.org/10.1109/ICIS.2016.7550861
Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognising Emotion from Facial Clue. Malor Books, San Jose (2003)
Plutchik, R., Bering, J.M., Descriptions, B.: Contents a phylogenetic approach to religious origins on the subcortical sources of basic human emotions and emergence of a unified mind science, vol. 08191 (2001)
Ramzan, M., Khan, H.U., Awan, S.M., Ismail, A., Ilyas, M., Mahmood, A.: A survey on state-of-the-art drowsiness detection techniques. IEEE Access 7, 61904–61919 (2019). https://doi.org/10.1109/ACCESS.2019.2914373
Rudovic, O., Pavlovic, V., Pantic, M.: Multi-output Laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2634–2641 (2012). https://doi.org/10.1109/CVPR.2012.6247983
Kimura, S., Yachida, M.: Facial expression recognition and its degree estimation, pp. 295–300 (1997). https://doi.org/10.1109/CVPR.1997.609338
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pp. 1–14 (2015)
Yannakakis, G.N., Cowie, R., Busso, C.: The ordinal nature of emotions, pp. 248–255 (2017). https://doi.org/10.1109/ACII.2017.8273608
Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3D facial expression database for facial behavior research. In: FGR 2006: Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, pp. 211–216 (2006). https://doi.org/10.1109/FGR.2006.6
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ekundayo, O., Viriri, S. (2020). Facial Expression Recognition and Ordinal Intensity Estimation: A Multilabel Learning Approach. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science(), vol 12510. Springer, Cham. https://doi.org/10.1007/978-3-030-64559-5_46
Download citation
DOI: https://doi.org/10.1007/978-3-030-64559-5_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64558-8
Online ISBN: 978-3-030-64559-5
eBook Packages: Computer ScienceComputer Science (R0)