Abstract
In the realm of digital education, the growing use of short-form online videos, coupled with innovative generative AI methods, has dramatically expanded the production of didactic academic videos. This shift, however, underscores a critical question - how to ascertain the "effectiveness" of these videos for student learning? It is essential to devise a classification mechanism that filters videos for clarity, comprehensibility, and their capacity to meet student learning objectives. The automated evaluation of these learning videos holds substantial implications for student academic performance. Accordingly, this paper presents a novel supervised-learning-based approach, predicated on video feature analysis, to predict the effectiveness of K-12 science and mathematics videos. Our method integrates diverse features such as image, spoken text, and audio, among other hand-crafted elements, to accurately assess video effectiveness. We conduct an evaluation of our approach using a comprehensive dataset comprised of 3,134 short-form academic videos. The results demonstrate robust performance, with the system achieving an accuracy of 76.1% and an F1 score of 80.6%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agrawal, A., Paepcke, A.: The stanford moocposts dataset. Accessed: Dec 15, 2020 (2014)
Ali, M.: PyCaret: An open source, low-code machine learning library in Python (April 2020), https://www.pycaret.org, pyCaret version 1.0.0
Bawden, D., Robinson, L.: Information overload: An overview (2020)
Bhanji, F., Gottesman, R., de Grave, W., Steinert, Y., Winer, L.R.: The retrospective pre-post: A practical method to evaluate learning from an educational program. Acad. Emerg. Med. 19(2), 189–194 (2012)
Boateng, R., Boateng, S.L., Awuah, R.B., Ansong, E., Anderson, A.B.: Videos in learning in higher education: assessing perceptions and attitudes of students at the university of ghana. Smart Learning Environments 3, 1–13 (2016)
Brame, C.J., et al.: Effective educational videos (2015)
Chassiakos, Y., Radesky, J., Christakis, D., Moreno, M., Cross, C., Hill, D., et al.: Children and adolescents and digital media. pediatrics [internet]. 2016 nov 1 [cited 2021 jun 9]; 138 (5)
Chung, D., Chen, Y., Meng, Y.: Perceived information overload and intention to discontinue use of short-form video: The mediating roles of cognitive and psychological factors. Behav. Sci. 13(1), 50 (2023)
Clavié, B., Gal, K.: Edubert: Pretrained deep language models for learning analytics. arXiv preprint arXiv:1912.00690 (2019)
Cohen, I., Huang, Y., Chen, J., Benesty, J., Benesty, J., Chen, J., Huang, Y., Cohen, I.: Pearson correlation coefficient. Noise reduction in speech processing pp. 1–4 (2009)
Davis, G.A.: Using a retrospective pre-post questionnaire to determine program impact. (2002)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Flesch, R.: A new readability yardstick. J. Appl. Psychol. 32(3), 221 (1948)
Garzotto, F.: Investigating the educational effectiveness of multiplayer online games for children. In: Proceedings of the 6th international conference on Interaction design and children. pp. 29–36 (2007)
Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 776–780. IEEE (2017)
Gunning, R.: The fog index after twenty years. J. Bus. Commun. 6(2), 3–13 (1969)
Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., et al.: Cnn architectures for large-scale audio classification. In: 2017 ieee international conference on acoustics, speech and signal processing (icassp). pp. 131–135. IEEE (2017)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017)
Killen, R.: Differences between students’ and lecturers’ perceptions of factors influencing students’ academic success at university. High. Educ. Res. Dev. 13(2), 199–211 (1994)
Kincaid, J.P., Fishburne Jr, R.P., Rogers, R.L., Chissom, B.S.: Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975)
Lee, S.g., Kim, H., Shin, C., Tan, X., Liu, C., Meng, Q., Qin, T., Chen, W., Yoon, S., Liu, T.Y.: Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. arXiv preprint arXiv:2106.06406 (2021)
Madeni, F., Horiuchi, S., Iida, M.: Evaluation of a reproductive health awareness program for adolescence in urban tanzania-a quasi-experimental pre-test post-test research. Reprod. Health 8(1), 1–9 (2011)
Marsden, E., Torgerson, C.J.: Single group, pre-and post-test research designs: Some methodological concerns. Oxf. Rev. Educ. 38(5), 583–616 (2012)
Mc Laughlin, G.H.: Smog grading-a new readability formula. J. Read. 12(8), 639–646 (1969)
Michelazzo, M.B., Pastorino, R., Mazzucco, W., Boccia, S.: Distance learning training in genetics and genomics testing for italian health professionals: results of a pre and post-test evaluation. Epidemiology, Biostatistics and Public Health 12(3) (2015)
Pardos, Z., Bergner, Y., Seaton, D., Pritchard, D.: Adapting bayesian knowledge tracing to a massive open online course in edx. In: Educational Data Mining 2013. Citeseer (2013)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
Rosen, L., Samuel, A.: Conquering digital distraction. Harv. Bus. Rev. 93(6), 110–113 (2015)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)
Shukor, N.A., Tasir, Z., Van der Meijden, H.: An examination of online learning effectiveness using data mining. Procedia. Soc. Behav. Sci. 172, 555–562 (2015)
Ssemugabi, S., De Villiers, M.: Effectiveness of heuristic evaluation in usability evaluation of e-learning applications in higher education. South African computer journal 2010(45), 26–39 (2010)
Stockwell, B.R., Stockwell, M.S., Cennamo, M., Jiang, E.: Blended learning improves science education. Cell 162(5), 933–936 (2015)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
Webster, J.G., Ksiazek, T.B.: The dynamics of audience fragmentation: Public attention in an age of digital media. J. Commun. 62(1), 39–56 (2012)
Acknowledgment
The primary author would like to extend thanks to the NLP department at MBZUAI and the department chair Professor Preslav Nakov, for their support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Thareja, R., Dwivedi, D., Garg, R., Baghel, S., Shukla, J., Mohania, M. (2025). Video Analysis Engine for Predicting Effectiveness. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15322. Springer, Cham. https://doi.org/10.1007/978-3-031-78312-8_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-78312-8_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-78311-1
Online ISBN: 978-3-031-78312-8
eBook Packages: Computer ScienceComputer Science (R0)
