Abstract
People use various nonverbal communicative channels to convey emotions, among which facial expressions are considered the most important ones. Consequently, automatic Facial Expression Recognition (FER) is a crucial task for enhancing computers’ perceptive abilities, particularly in human-computer interaction. Although state-of-the-art FER systems can identify emotions from the entire face, situations may arise where occlusions prevent the entire face from being visible. During the COVID-19 pandemic, many FER systems have been developed for recognizing emotions from the eye region due to the obligation to wear a mask. However, in many situations, the eyes may be covered, for instance, by sunglasses or virtual reality devices. In this paper, we faced the problem of developing a FER system that solely considers the mouth region and classifies emotions using only the lower part of the face. We tested the effectiveness of this FER system in recognizing emotions from the lower part of the face and compared the results to a FER system trained on the same datasets using the same approach on the entire face. As expected, emotions primarily associated with the mouth region (e.g., happiness, surprise) were recognized with minimal loss compared to the entire face. Nevertheless, even though most negative emotions were not accurately detected using only the mouth region, in cases where the face is partially covered, this area may still provide some information about the displayed emotion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Encyclopedia of Human Behavior, 2nd edition. V. S. Ramachandran (2012)
Akbar, M.T., Ilmi, M.N., Rumayar, I.V., Moniaga, J., Chen, T.K., Chowanda, A.: Enhancing game experience with facial expression recognition as dynamic balancing. Procedia Computer Science 157, 388–395 (2019). doi: https://doi.org/10.1016/j.procs.2019.08.230,the 4th International Conference on Computer Science and Computational Intelligence (ICCSCI 2019) : Enabling Collaboration to Escalate Impact of Research Results for Society
Assari, M.A., Rahmati, M.: Driver drowsiness detection using face expression recognition. 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA) pp. 337–341 (2011)
Biondi, G., Franzoni, V., Gervasi, O., Perri, D.: An approach for improving automatic mouth emotion recognition. In: Computational Science and Its Applications-ICCSA 2019: 19th International Conference, Saint Petersburg, Russia, July 1–4, 2019, Proceedings, Part I 19. pp. 649–664. Springer (2019)
Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: A dataset for recognising faces across pose and age. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) pp. 67–74 (2018)
Carbon, C.C.: Wearing face masks strongly confuses counterparts in reading emotions. Frontiers in Psychology 11, 2526 (2020)
Castellano, G., De Carolis, B., Macchiarulo, N.: Automatic facial emotion recognition at the covid-19 pandemic time. Multimedia Tools and Applications 82(9), 12751–12769 (2023)
Ekman, P.: Basic emotions. Handbook of cognition and emotion 98(45–60), 16 (1999)
Franzoni, V., Biondi, G., Perri, D., Gervasi, O.: Enhancing mouth-based emotion recognition using transfer learning. Sensors 20(18), 5222 (2020)
González-Lozoya, S.M., de la Calleja, J., Pellegrin, L., Escalante, H.J., Medina, M.A., Benitez-Ruiz, A.: Recognition of facial expressions based on cnn features. Multimedia tools and applications 79, 13987–14007 (2020)
Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, L., Xu, B., Chuang, Z., Bengio, Y.: Challenges in representation learning: A report on three machine learning contests (2013)
Greco, A., Saggese, A., Vento, M., Vigilante, V.: Performance assessment of face analysis algorithms with occluded faces. In: Del Bimbo, A., Cucchiara, R., Sclaroff, S., Farinella, G.M., Mei, T., Bertini, M., Escalante, H.J., Vezzani, R. (eds.) Pattern Recognition. ICPR International Workshops and Challenges. pp. 472–486. Springer International Publishing, Cham (2021)
Grundmann, F., Epstude, K., Scheibe, S.: Face masks reduce emotion-recognition accuracy and perceived closeness. PloS one 16(4), e0249792 (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016). DOI: 10.1109/CVPR.2016.90
Kim, G., Seong, S.H., Hong, S.S., Choi, E.: Impact of face masks and sunglasses on emotion recognition in south koreans. PLoS One 17(2), e0263466 (2022)
King, D.E.: Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research 10, 1755–1758 (2009)
Lankes, M., Riegler, S., Weiss, A., Mirlacher, T., Pirker, M., Tscheligi, M.: Facial expressions as game input with different emotional feedback conditions. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology. p. 253–256. ACE ’08, Association for Computing Machinery, New York, NY, USA (2008). DOI: 10.1145/1501750.1501809
Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2584–2593. IEEE (2017)
Mehrabian, A.: Nonverbal Communication. Aldine-Atherton, New York (1972)
Park, J., Woo, S., Lee, J.Y., Kweon, I.S.: Bam: Bottleneck attention module. In: BMVC (2018)
Santana, O.J., Freire-Obregón, D., Hernández-Sosa, D., Lorenzo-Navarro, J., Sánchez-Nielsen, E., Castrillón-Santana, M.: Facial expression analysis in a wild sporting environment. Multimedia Tools and Applications 82(8), 11395–11415 (2023)
Saxena, S., Tripathi, S., Sudarshan, T.: Deep facial emotion recognition system under facial mask occlusion. In: International Conference on Computer Vision and Image Processing. pp. 381–393. Springer (2020)
Tkalčič, M., Maleki, N., Pesek, M., Elahi, M., Ricci, F., Marolt, M.: Prediction of music pairwise preferences from facial expressions. In: Proceedings of the 24th International Conference on Intelligent User Interfaces. p. 150–159. IUI ’19, Association for Computing Machinery, New York, NY, USA (2019). DOI: 10.1145/3301275.3302266
Yang, B., Wu, J., Hattori, G.: Facial expression recognition with the advent of human beings all behind face masks. MUM2020, Association for Computing Machinery, Essen, Germany (2020)
Zakka, B.E., Vadapalli, H.: Detecting learning affect in e-learning platform using facial emotion expression. In: Abraham, A., Jabbar, M.A., Tiwari, S., Jesus, I.M.S. (eds.) Proceedings of the 11th International Conference on Soft Computing and Pattern Recognition (SoCPaR 2019). pp. 217–225. Springer International Publishing, Cham (2021)
Zhang, L., Verma, B., Tjondronegoro, D., Chandran, V.: Facial expression analysis under partial occlusion: A survey. ACM Computing Surveys (CSUR) 51(2), 1–49 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
De Carolis, B., Macchiarulo, N., Palestra, G., De Matteis, A.P., Lippolis, A. (2023). FERMOUTH: Facial Emotion Recognition from the MOUTH Region. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-43148-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43147-0
Online ISBN: 978-3-031-43148-7
eBook Packages: Computer ScienceComputer Science (R0)