Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Lite general network and MagFace CNN for micro-expression spotting in long videos

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Facial expressions, especially spontaneous micro-expressions, as an intuitive reflection of human emotions, have come through much concern along with rapid advances in computer vision recently. Micro-expressions are small in amplitude and short in duration and often appear together with macro-expressions, making micro-expression spotting in long videos a challenging task. In this article, we propose intersection over minimum labelling method combined with a Lite General Network and MagFace CNN (LGNMNet) model to predict the possibility of video frames belonging to a micro-expression interval, which balances easy and difficult samples to improve the learning effect of training process. Experimental results show that our method achieves state-of-the-art performance in spotting micro-expressions in long videos of both the CAS(ME)2 and SAMM-LV datasets (with F1-scores of 0.2474 and 0.2555, respectively). Additionally, a new pair-merge way of combining nearby detected apex frames to construct micro-expression intervals in post-processing stage has been devised and analysed, providing a feasible solution for the task of macro- and micro-spotting in long videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets including SAMM and CAS(ME)2 that support the findings of this study are available at http://www2.docm.mmu.ac.uk/STAFF/M.Yap/dataset.php and http://fu.psych.ac.cn/CASME/casme2-en.php respectively.

References

  1. Ekman, P.: Telling lies: clues to deceit in the marketplace, politics, and marriage, revised WW Norton & Company (2009)

    Google Scholar 

  2. Martin, C.W. (ed.): The philosophy of deception. Oxford university press on demand (2009)

    Google Scholar 

  3. Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., Liu, Y.-J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 5826–5846 (2021)

    Google Scholar 

  4. Liu, J., Li, K., Song, B., Zhao, L.: A multi-stream convolutional neural network for micro-expression recognition using optical flow and evm. arXiv preprint arXiv:2011.03756. (2020)

  5. Yuhong, H.: Research on micro-expression spotting method based on optical flow features. In: Proceedings of the 29th ACM International Conference on Multimedia (MM '21), New York (2021)

  6. Wang, S.-J., Wu, S., Qian, X., Li, J., Fu, X.: A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing 230, 382–389 (2017)

    Article  Google Scholar 

  7. Liu, Y.J., Zhang, J.K., Yan, W.J., Wang, S.J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2016)

    Article  Google Scholar 

  8. Zhou, Y., Song, Y., Chen, L., Chen, Y., Ben, X., Cao, Y.: A novel micro-expression detection algorithm based on BERT and 3DCNN. Image Vis. Comput 119, 104378 (2022)

    Article  Google Scholar 

  9. Sun, B., Cao, S., He, J., Yu, L.: Two-stream attention-aware network for spontaneous micro-expression movement spotting. IEEE 10th International Conference on Software Engineering and Service Science (ICSESS). Beijing, China (2019)

    Book  Google Scholar 

  10. Tran, T.-K., Vo, Q.-N., Hong, X., Zhao, G.J.E.I.: Dense prediction for micro-expression spotting based on deep sequence model. Electronic Imaging, pp. 401–406 (2019)

  11. Verburg, M., Menkovski, V.: Micro-expression detection in long videos using optical flow and recurrent neural networks. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), New York (2019)

    Book  Google Scholar 

  12. Yang, B., Wu, J., Zhou, Z., Komiya, M., Kishimoto, K., Xu, J., Nonaka, K., Horiuchi, T., Komorita, S. and Hattori, G.: Facial action unit-based deep learning framework for spotting macro-and micro-expressions in long video sequences. City (2021)

  13. Liong, G.-B., See, J., Wong, L.-K.: Shallow optical flow three-stream CNN for macro-and micro-expression spotting from long videos. In: IEEE International Conference on Image Processing (ICIP), New York (2021)

    Book  Google Scholar 

  14. Yu, W.-W., Jiang, J., Li, Y.-J.: LSSNet: A two-stream convolutional neural network for spotting macro-and micro-expression in long videos. In: Proceedings of the 29th ACM International Conference on Multimedia (MM '21), New York (2021)

  15. Zhao, S., Tao, H., Zhang, Y., Xu, T., Zhang, K., Hao, Z., Chen, E.J.N.: A two-stage 3D CNN based learning method for spontaneous micro-expression recognition. Neurocomputing 448, 276–289 (2021)

    Article  Google Scholar 

  16. Nummenmaa, L., Saarimäkia, H., Glereana, E., Gotsopoulosa, A., Jääskeläinena, I. P., Harib, R., Samsa, M., Glerean, E., Hari, R., Hietanen, J. K. Ekman, Paul: Emotions revealed. Recognizing faces and feelings to improve communication and emotional life. New York: holt paper-back, montgomery, arlene (2013) Neurobiology essentials for clinicians. what every therapist needs to know, New York, London (2007)

  17. Bhushan, B.: Study of facial micro-expressions in psychology. In: Understanding facial expressions in communication: cross-cultural and multidisciplinary perspectives, pp. 265–286. Springer (2015)

    Chapter  Google Scholar 

  18. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikäinen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2017)

    Article  Google Scholar 

  19. Li, J., Soladie, C., Seguier, R.: Ltp-ml: micro-expression detection by recognition of local temporal pattern of facial movements. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi'an, China (2018)

    Google Scholar 

  20. Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, Barcelona, Spain (2011)

    Book  Google Scholar 

  21. Chen, M., Ma, H.T., Li, J., Wang, H.: Emotion recognition using fixed length micro-expressions sequence and weighting method. In: IEEE International Conference on Real-time Computing and Robotics (RCAR), Angkor Wat, Cambodia (2016)

    Book  Google Scholar 

  22. Xu, F., Zhang, J., Wang, J.Z.: Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8(2), 254–267 (2017)

    Article  Google Scholar 

  23. Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro- and micro-expression spotting in long videos using spatio-temporal strain. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, USA (2011)

    Book  Google Scholar 

  24. Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., Sarkar, S.: Automatic expression spotting in videos. Image Vis Comput 32(8), 476–486 (2014)

    Article  Google Scholar 

  25. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA (2013)

  26. Guo, Y., Li, B., Ben, X., Ren, Y., Zhang, J., Yan, R., Li, Y.: A Magnitude and angle combined optical flow feature for microexpression spotting. IEEE Multimedia 28(2), 29–39 (2021)

    Article  Google Scholar 

  27. Tran, T.-K., Hong, X., Zhao, G.: Sliding window based micro-expression spotting: a benchmark. In: Advanced Concepts for Intelligent Vision Systems. ACIVS 2017. Lecture Notes in Computer Science(), vol 10617, Springer, Cham (2017)

    Book  Google Scholar 

  28. Xia, Z., Feng, X., Peng, J., Peng, X., Zhao, G.: Spontaneous micro-expression spotting via geometric deformation modeling. Comput. Vis. Image Underst. 147, 87–94 (2016)

    Article  Google Scholar 

  29. Wang, S.-J., He, Y., Li, J., Fu, X.: MESNet: a convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Trans Image Process 30, 3956–3969 (2021)

    Article  Google Scholar 

  30. Kim, D. H., Baddar, W. J., Ro, Y. M.: Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM international conference on Multimedia (MM '16), New York (2016)

  31. Liong, S.-T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.-H., Phan, R.: Automatic apex frame spotting in micro-expression database. 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 665–669, New York (2015)

    Book  Google Scholar 

  32. Pan, H., Xie, L., Wang, Z.: Local bilinear convolutional neural network for spotting macro-and micro-expression intervals in long video sequences. IEEE, Buenos Aires, Argentina (2020)

    Book  Google Scholar 

  33. Li, Y., Huang, X., Zhao, G.: Micro-expression action unit detection with spatial and channel attention. Neurocomputing 436, 221–231 (2021)

    Article  Google Scholar 

  34. Khor, H.-Q., See, J., Phan, R.C.W., Lin, W.: Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi'an, China (2018)

    Book  Google Scholar 

  35. Yap, C. H., Yap, M. H., Davison, A. K., Cunningham, R.: 3d-cnn for facial micro-and macro-expression spotting on long video sequences using temporal oriented reference frame. In: Proceedings of the 30th ACM International Conference on Multimedia (MM '22). New York, (2021)

  36. Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S. Retinaface: Single-shot multi-level face localisation in the wild. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, (2020)

  37. Mohamed, M.A., Mertsching, B.: TV-L1 optical flow estimation with image details recovering based on modified census transform. Springer, Berlin (2012)

    Book  Google Scholar 

  38. Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Gool, L.V.: Temporal segment networks: Towards good practices for deep action recognition. Springer, Cham (2016)

    Google Scholar 

  39. Howard, A., Zhmoginov, A., Chen, L.-C., Sandler, M. and Zhu, M.: Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation. (2018)

  40. Yap, C.H., Kendrick, C., Yap, M.H.: Samm long videos: a spontaneous facial micro-and macro-expressions dataset. In: 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina (2020)

    Google Scholar 

  41. Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., Fu, X.: CAS (ME) $^ 2$: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affective. Comput. 9(4), 424–436 (2017)

    Article  Google Scholar 

  42. Davison, A.K., Merghani, W., Yap, M.H.: Objective classes for micro-facial expression recognition. J Imaging 4(10), 119 (2018)

    Article  Google Scholar 

  43. Liong, G.B., Liong, S.-T., See, J., Chan, C.-S.: MTSN: a Multi-Temporal Stream Network for Spotting Facial Macro-and Micro-Expression with Hard and Soft Pseudo-labels. In: Proceedings of the 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis (FME '22), New York (2022)

  44. Wang, J., Liu, Y., Hu, Y., Shi, H., Mei, T.: Facex-zoo: a pytorch toolbox for face recognition. In: Proceedings of the 29th ACM International Conference on Multimedia (MM '21), New York (2021)

Download references

Author information

Authors and Affiliations

  1. Wuxi Xishang Bank, Wuxi, 214000, China

    Quan-Lin Gu, Sai Yang & Tianxing Yu

  2. China Center for Internet Economy Research, Central University of Finance and Economics, Beijing, 100098, China

    Quan-Lin Gu

Authors
  1. Quan-Lin Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tianxing Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SY and TY: wrote the main manuscript text. Q-LG: discussed key methods and experiments. All authors reviewed the manuscript.

Corresponding author

Correspondence to Sai Yang.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Communicated by B. Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, QL., Yang, S. & Yu, T. Lite general network and MagFace CNN for micro-expression spotting in long videos. Multimedia Systems 29, 3521–3530 (2023). https://doi.org/10.1007/s00530-023-01145-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01145-3

Search

Navigation