MSP-GEO Corpus: A Multimodal Database for Understanding Video-Learning Experience
Authors: 
Ali N Salman, 
Ning Wang, Luz Maria Martinez-Lucas, Andrea Vidal, Carlos BussoAuthors Info & Claims
Ali N Salman, Ning Wang, Luz Maria Martinez-Lucas, Andrea Vidal, Carlos BussoAuthors Info & ClaimsPages 488 - 497
Abstract
Video-based learning has become a popular, scalable, and effective approach for students to learn new skills. Many of the challenges for video-based learning can be addressed with machine learning models. However, the available datasets often lack the rich source of data that is needed to accurately predict students’ learning experiences and outcomes. To address this limitation, we introduce the MSP-GEO corpus, a new multimodal database that contains detailed demographic and educational data, recordings of the students and their screens, and meta-data about the lecture during the learning experience. The MSP-GEO corpus was collected using a quasi-experimental pre-test/post-test design. It consists of more than 39,600 seconds (11 hours) of continuous facial footage from 76 participants watching one of three experimental videos on the topic of fossil formation, resulting in over one million facial images. The data collected includes 21 gaze synchronization points, webcam and monitor recordings, and metadata for pauses, plays, and timeline navigation. Additionally, we annotated the recordings for engagement, boredom, and confusion using human evaluators. The MSP-GEO corpus has the potential to improve the accuracy of video-based learning outcomes and experience predictions, facilitate research on the psychological processes of video-based learning, inform the design of instructional videos, and advance the development of learning analytics methods.
References
[1]
Eric Araka, Elizaphan Maina, Rhoda Gitonga, and Robert Oboko. 2020. Research trends in measurement and intervention tools for self-regulated learning for e-learning environments—systematic review (2008–2018). Research and Practice in Technology Enhanced Learning 15 (12 2020). https://doi.org/10.1186/s41039-020-00129-5
[2]
Rachel Baker, Di Xu, Jihyun Park, Renzhe Yu, Qiujie Li, Bianca Cung, Christian Fischer, Fernando Rodriguez, Mark Warschauer, and Padhraic Smyth. 2020. The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes. International Journal of Educational Technology in Higher Education volume 17 (04 2020). https://doi.org/10.1186/s41239-020-00187-1
[3]
L.J. Cronbach. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16, 3 (September 1951), 297–334.
[4]
Radek Danecek, Michael J. Black, and Timo Bolkart. 2022. EMOCA: Emotion Driven Monocular Face Capture and Animation. In Conference on Computer Vision and Pattern Recognition (CVPR).
[5]
Kevin Delgado, Juan Manuel Origgi, Tania Hasanpoor, Hao Yu, Danielle Allessio, Ivon Arroyo, William Lee, Margrit Betke, Beverly Woolf, and Sarah Adel Bargal. 2021. Student Engagement Dataset. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). 3621–3629. https://doi.org/10.1109/ICCVW54120.2021.00405
[6]
Blaženka Divjak, Bart Rienties, Francisco Iniesto, Petra Vondra, and Mirza Žižak. 2022;19. Flipped classrooms in higher education during the COVID-19 pandemic: findings and future research recommendations.Research and Practice in Technology Enhanced Learning (2022;19). https://doi.org/10.1186/s41239-021-00316-4
[7]
Nancy Falchikov and David Boud. 1989. Student self-assessment in higher education: A meta-analysis. Review of educational research 59, 4 (1989), 395–430.
[8]
Christian Fischer, Zachary A. Pardos, Ryan Shaun Baker, Joseph Jay Williams, Padhraic Smyth, Renzhe Yu, Stefan Slater, Rachel Baker, and Mark Warschauer. 2020. Mining Big Data in Education: Affordances and Challenges. Review of Research in Education 44, 1 (2020), 130–160. https://doi.org/10.3102/0091732X20903304 arXiv:https://doi.org/10.3102/0091732X20903304
[9]
Jeffrey M Girard. 2014. CARMA: Software for continuous affect rating and media annotation. Journal of Open Research Software 2, 1 (2014), e5. https://doi.org/10.5334/jors.ar
[10]
Abhay Gupta, Richik Jaiswal, Sagar Adhikari, and Vineeth Balasubramanian. 2016. DAISEE: Dataset for Affective States in E-Learning Environments. CoRR abs/1609.01885 (2016). arXiv:1609.01885http://arxiv.org/abs/1609.01885
[11]
R. Gupta, K. Audhkhasi, Z. Jacokes, A. Rozga, and S.S. Narayanan. 2018. Modeling multiple time series annotations based on ground truth inference and distortion. IEEE Transactions on Affective Computing 9, 1 (January-March 2018), 76–89. https://doi.org/10.1109/TAFFC.2016.2592918
[12]
Carolina Guzmán-Valenzuela, Carolina Gómez-González, Andrés Rojas-Murphy Tagle, and Alejandro Lorca-Vyhmeister. 2021. Learning analytics in higher education: a preponderance of analytics but very little learning?International Journal of Educational Technology in Higher Education 18 (2021).
[13]
Paul A. Harris, Robert Taylor, Brenda L. Minor, Veida Elliott, Michelle Fernandez, Lindsay O’Neal, Laura McLeod, Giovanni Delacqua, Francesco Delacqua, Jacqueline Kirby, and Stephany N. Duda. 2019. The REDCap consortium: Building an international community of software platform partners. Journal of Biomedical Informatics 95 (2019), 103208. https://doi.org/10.1016/j.jbi.2019.103208
[14]
Kintu Justice and Chang Zhu. 2017. Blended learning effectiveness: the relationship between student characteristics, design features and outcomes. International Journal of Educational Technology in Higher Education 14 (02 2017). https://doi.org/10.1186/s41239-017-0043-4
[15]
Amanjot Kaur, Aamir Mustafa, Love Mehta, and Abhinav Dhall. 2018. Prediction and Localization of Student Engagement in the Wild. In 2018 Digital Image Computing: Techniques and Applications (DICTA). 1–8. https://doi.org/10.1109/DICTA.2018.8615851
[16]
D.E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (July 2009), 1755–1758.
[17]
Nicolas Legewie and Anne Nassauer. 2018. YouTube, Google, Facebook: 21st Century Online Video Research and Research Ethics. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research Vol 19 (2018), No 3 (2018): Research Ethics in Qualitative Research. https://doi.org/10.17169/FQS-19.3.3130
[18]
N. Li and C. Busso. 2014. User-Independent Gaze Estimation by Exploiting Similarity Measures in the Eye Pair Appearance Eigenspace. In International conference on multimodal interaction (ICMI 2014). Istanbul, Turkey, 335–338. https://doi.org/10.1145/2663204.2663250
[19]
N. Li and C. Busso. 2018. Calibration Free, User Independent Gaze Estimation with Tensor Analysis. Image and Vision Computing 74 (June 2018), 10–20. https://doi.org/10.1016/j.imavis.2018.04.001
[20]
Julie C Libarkin and Steven W Anderson. 2005. Assessment of learning in entry-level geoscience courses: Results from the Geoscience Concept Inventory. Journal of Geoscience Education 53, 4 (2005), 394–401.
[21]
Tiecheng Liu and John R Kender. 2004. Lecture videos for e-learning: Current research and challenges. In IEEE Sixth International Symposium on Multimedia Software Engineering. IEEE, 574–578.
[22]
S. Mariooryad and C. Busso. 2013. Analysis and Compensation of the Reaction Lag of Evaluators in Continuous Emotional Annotations. In Affective Computing and Intelligent Interaction (ACII 2013). Geneva, Switzerland, 85–90. https://doi.org/10.1109/ACII.2013.21
[23]
S. Mariooryad and C. Busso. 2015. Correcting Time-Continuous Emotional Labels by Modeling the Reaction Lag of Evaluators. IEEE Transactions on Affective Computing 6, 2 (April-June 2015), 97–108. https://doi.org/10.1109/TAFFC.2014.2334294 Special Issue Best of ACII.
[24]
L. Martinez-Lucas, Mohammed Abdelwahab, and Carlos Busso. 2020. The MSP-Conversation Corpus. In Interspeech 2020. Shanghai, China, 1823–1827. https://doi.org/10.21437/Interspeech.2020-2444
[25]
Richard E Mayer. 2005. Cognitive theory of multimedia learning. The Cambridge handbook of multimedia learning 41, 1 (2005), 31–48.
[26]
Daniel A. McFarland, Saurabh Khanna, Benjamin W. Domingue, and Zachary A. Pardos. 2021. Education Data Science: Past, Present, Future. AERA Open 7 (2021), 23328584211052055. https://doi.org/10.1177/23328584211052055 arXiv:https://doi.org/10.1177/23328584211052055
[27]
G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schröder. 2012. The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent. IEEE Transactions on Affective Computing 3, 1 (January-March 2012), 5–17. https://doi.org/10.1109/T-AFFC.2011.20
[28]
A. Mollahosseini, B. Hasani, and M. H. Mahoor. 2019. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Transactions on Affective Computing 10, 1 (January-March 2019), 18–31. https://doi.org/10.1109/TAFFC.2017.2740923
[29]
Christian Mühl, Brendan Allison, Anton Nijholt, and Guillaume Chanel. 2014. A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain-Computer Interfaces 1, 2 (2014), 66–84. https://doi.org/10.1080/2326263X.2014.912881 arXiv:https://doi.org/10.1080/2326263X.2014.912881
[30]
Larian M. Nkomo, Ben Kei Daniel, and Russell Butson. 2021. Synthesis of student engagement with digital technologies: a systematic review of the literature. International Journal of Educational Technology in Higher Education 18 (2021).
[31]
Ekene Francis Okagbue, Ujunwa Perpetua Ezeachikulo, Esther Onyinye Nwigwe, and Amina Abedi Juma. 2022. Machine Learning and Artificial Intelligence in Education Research: A Comprehensive Overview of 22 Years of Research indexed in the Scopus Database. Social Sciences & Humanities Open (07 2022).
[32]
Md. Mustafizur Rahman, Ajay Krishno Sarkar, Md. Amzad Hossain, Md. Selim Hossain, Md. Rabiul Islam, Md. Biplob Hossain, Julian M.W. Quinn, and Mohammad Ali Moni. 2021. Recognition of human emotions using EEG signals: A review. Computers in Biology and Medicine 136 (2021), 104696. https://doi.org/10.1016/j.compbiomed.2021.104696
[33]
F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne. 2013. Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions. In 2nd International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space (EmoSPACE 2013). Shanghai, China, 1–8. https://doi.org/10.1109/FG.2013.6553805
[34]
David Sharek and Eric Wiebe. 2014. Measuring Video Game Engagement Through the Cognitive and Affective Dimensions. Simulation & Gaming 45, 4-5 (2014), 569–592. https://doi.org/10.1177/1046878114554176 arXiv:https://doi.org/10.1177/1046878114554176
[35]
Jieun Shin. 2020. How do partisans consume news on social media? A comparison of self-reports with digital trace measures among Twitter users. Social Media+ Society 6, 4 (2020), 2056305120981039.
[36]
Ilaria Tiddi, Mathieu d’Aquin, and Enrico Motta. 2015. An ontology design pattern to define explanations. In Proceedings of the 8th International Conference on Knowledge Capture. 1–8.
[37]
Mark Urban-Lurain, Diane Ebert-May, Jennifer Momsen, Ryan McFall, Matthew B Jones, Ben Leinfelder, and Jon Sticklen. 2009. An assessment database for supporting educational research. In 2009 39th IEEE Frontiers in Education Conference. IEEE, 1–6.
[38]
Ning Wang, Zachary Clowdus, Alessandra Sealander, and Robert Stern. 2022. Geonews: Timely geoscience educational YouTube videos about recent geologic events. Geoscience Communication 5, 2 (2022), 125–142.
[39]
Ning Wang, Robert J Stern, Mary L Urquhart, and Katherine M Seals. 2022. Google earth geoscience video library (GEGVL): Organizing geoscience videos in a google earth environment to support fieldwork teaching methodology in earth science. Geosciences 12, 6 (2022), 250.
[40]
Ning Wang, Robert J. Stern, and Lowell Waite. 2023. Workflow for designing instructional videos to support place-based geoscience education for geoscience majors. Journal of Geoscience Education 71, 1 (2023), 107–125. https://doi.org/10.1080/10899995.2022.2093543 arXiv:https://doi.org/10.1080/10899995.2022.2093543
[41]
Jacob Whitehill, Zewelanji Serpell, Yi-Ching Lin, Aysha Foster, and Javier R. Movellan. 2014. The Faces of Engagement: Automatic Recognition of Student Engagement from Facial Expressions. IEEE Transactions on Affective Computing 5, 1 (2014), 86–98. https://doi.org/10.1109/TAFFC.2014.2316163
[42]
Olaf Zawacki-Richter, Victoria Marín, Melissa Bond, and Franziska Gouverneur. 2019. Systematic review of research on artificial intelligence applications in higher education -where are the educators?International Journal of Educational Technology in Higher Education 16 (10 2019), 1–27. https://doi.org/10.1186/s41239-019-0171-0
Index Terms
- MSP-GEO Corpus: A Multimodal Database for Understanding Video-Learning Experience
Recommendations
Emotional Design Tutoring System Based on Multimodal Affective Computing Techniques
In a traditional class, the role of the teacher is to teach and that of the students is to learn. However, the constant and rapid technological advancements have transformed education in numerous ways. For instance, in addition to traditional, face to ...
The effects of web-based learning experience, perceived-initiative, and perceived-performance on learners' attitudes toward mobile learning
This study explored the effects of prior web-based learning experience, perceived-initiative, and perceived-performance on learners' attitudes toward a mobile learning system designed to enhance novices' reflective thinking and problem-solving through ...
Multimethod assessment of affective experience and expression during deep learning
Inquiries into the link between affect and learning require robust methodologies to measure the learner's affective states. We describe two studies that utilised either an online or offline methodology to detect the affective states of a learner during ...
Comments
Information & Contributors
Information
Published In
November 2024
725 pages
ISBN:9798400704628
DOI:10.1145/3678957
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICMI '24
ICMI '24: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
November 4 - 8, 2024
San Jose, Costa Rica
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 72Total Downloads
- Downloads (Last 12 months)72
- Downloads (Last 6 weeks)47
Reflects downloads up to 21 Dec 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in