Abstract
Understanding students’ educational emotion is important for learning process, however, it is challenging to detect in an online learning environment. Deep learning architectures show excellent performance for emotion detection from facial expressions; however, their complexity and computational requirements limit their deployment on students’ edge devices. Additionally, the availability and the size of the datasets for detecting educational emotions are scarce. In this study, we propose a lightweight deep learning model based on MobileNet architecture which is deployable in students’ edge devices for educational emotion detection. We also propose a generative adversarial network based synthetic image generation technique to address the challenges of scarcity of the dataset. This framework is compared with the state-of-the-art models, where it demonstrated competitive performance while making it suitable for the edge devices for the educational emotion detection, and additionally, the use of the synthetic dataset further contributes to improve the performance of the proposed model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhang, Z., Maeda, Y., Newby, T.: Individual differences in preservice teachers’ online self-regulated learning capacity: a multilevel analysis. Comput. Educ. 207, 1–13 (2023)
Dewan, M.A.A., Murshed, M., Lin, F.: Engagement detection in online learning: a review. Smart Learn. Environ. 6(1), 1–20 (2019)
Li, T., Chan, K.-L., Tjahjadi, T.: Multi-Scale correlation module for video-based facial expression recognition in the wild. Pattern Recogn. 142, 1–10 (2023)
Boulanger, D., Dewan, M.A.A., Kumar, V.S., Lin, F.: Lightweight and interpretable detection of affective engagement for online learners. In: Proceedings of IEEE PICom 2021, pp. 176–184 (2021)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications, pp. 1–9. arXiv preprint arXiv:1704.04861 (2017)
Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)
Bian, C., Zhang, Y., Yang, F., Bi, W., Lu, W.: Spontaneous facial expression database for academic emotion inference in online learning. IET Comput. Vision 13, 329–337 (2019)
Anderson, A.R., Christenson, S.L., Sinclair, M.F., Lehr, C.A.: Check and connect: the importance of relationships for promoting engagement with school. J. School Psychol. 42, 95–113 (2004)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
Singh, S., Nasoz, F.: Facial expression recognition with convolutional neural networks. In: Annual Computing and Communication Workshop and Conference, pp. 324–328. Las Vegas, USA (2020)
Thai, L.H., Nguyen, N.D.T., Hai, T.S.: A facial expression classification system integrating canny, principal component analysis and artificial neural network. Int. J. Mach. Learn. Comput. 1(4), 388–393 (2011)
Murthy, G.R.S., Jadon, R.S.: Recognizing facial expressions using eigenspaces. In: IEEE International Conference on Computational Intelligence and Multimedia Applications, Sivakasi, India (2007)
Aung, D.M., Aye, N.: Facial expression classification using histogram based method. In: Proceedings of the International Conference on Signal Processing Systems (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Iandola, F.N., et al.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016)
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)
Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans. Image Process. 28(5), 2439–2450 (2018)
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)
Rana, S.P., Dey, M., Siarry, P.: Boosting content based image retrieval performance through integration of parametric & nonparametric approaches. J. Vis. Commun. Image Represent. 58, 205–219 (2019)
Deng, J., Pang, G., Zhang, Z., Pang, Z., Yang, H., Yang, G.: cGAN based facial expression recognition for human-robot interaction. IEEE Access 7, 9848–9859 (2019)
Zhang, H., et al.: Stackgan: text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5908–5916 (2017)
Zhao, J., Mathieu, M., LeCun, Y.: Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126 (2016)
Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition (2001)
Hsu, H., Lachenbruch, P.A.: Paired t-test. In: Wiley StatsRef: Statistics (2014)
Acknowledgements
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and Alberta Innovates, Canada.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kabir, M.R., Dewan, M.A.A., Lin, F. (2024). Emotion Detection from Facial Expression in Online Learning Through Using Synthetic Image Generation. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2024. Lecture Notes in Computer Science(), vol 14735. Springer, Cham. https://doi.org/10.1007/978-3-031-60611-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-60611-3_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-60613-7
Online ISBN: 978-3-031-60611-3
eBook Packages: Computer ScienceComputer Science (R0)