Abstract
Providing personalised feedback to a large student cohort is a longstanding challenge in education. Recent work in prescriptive learning analytics (PLA) demonstrated a promising approach by augmenting predictive models with prescriptive capabilities of explainable artificial intelligence (XAI). Although theoretically sound, in practice, not all predictive features can be leveraged by XAI to prescribe useful feedback. It remains under-explored as to how to engineer such predictive features that can be used to prescribe personalised and actionable feedback. To address this, we proposed a learning activity-based approach to design features that are informative to both predictive and prescriptive performance in PLA. We conducted empirical evaluations of the quality of PLA-generated feedback compared to feedback written by experienced teachers in a large-scale university course. Four rubric criteria, including Readily Applicablility, Readability, Relational, and Specificity, were designed based on previous research. We found that: (i) By adopting learning activity-based features, PLA generates high quality feedback without sacrificing predictive performance; (ii) Most experienced teaching staff rated PLA-generated feedback as readily applicable to the course; and (iii) Compared to teacher-written feedback, the quality of PLA-generated feedback is consistently rated higher (with statistical significance) in all four rubric criteria by experienced teachers. All code is available via our GitHub repository (https://github.com/CoLAMZP/AIED-2024-AutoFeedback).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
Scikit-learn package (2023), https://scikit-learn.org/stable/.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Ajjawi, R., Boud, D., Henderson, M., Molloy, E.: Improving feedback research in naturalistic settings. In: Henderson, M., Ajjawi, R., Boud, D., Molloy, E. (eds.) The Impact of Feedback in Higher Education, pp. 245–265. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25112-3_14
Bergstra, J., Yamins, D., Cox, D.: Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: International Conference on Machine Learning, pp. 115–123. PMLR (2013)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Capuano, N., Rossi, D., Ströele, V., Caballé, S.: Explainable prediction of student performance in online courses. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) TLIC 2023. LNCS, vol. 767, pp. 639–652. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-41637-8_52
Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)
Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)
Dawson, P., et al.: What makes for effective feedback: staff and student perspectives. Assess. Eval. High. Educ. 44(1), 25–36 (2019)
Dawson, P., et al.: Technology and feedback design. In: Spector, M., Lockee, B., Childress, M. (eds.) Learning, Design, and Technology, pp. 1–45. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-17727-4_124-1
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., De Weerdt, J.: A review of automated feedback systems for learners: classification framework, challenges and opportunities. Comput. Educ. 162, 104094 (2021)
Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29, 131–163 (1997)
Jang, Y., Choi, S., Jung, H., Kim, H.: Practical early prediction of students’ performance using machine learning and explainable AI. Educ. Inf. Technol. 27(9), 12855–12889 (2022)
Jia, Q., Cui, J., Xiao, Y., Liu, C., Rashid, P., Gehringer, E.F.: All-in-one: multi-task learning bert models for evaluating peer assessments. arXiv preprint arXiv:2110.03895 (2021)
Jia, Q., et al.: Insta-reviewer: a data-driven approach for generating instant feedback on students’ project reports. International Educational Data Mining Society (2022)
Karalar, H., Kapucu, C., Gürüler, H.: Predicting students at risk of academic failure using ensemble model during pandemic in a distance learning system. Int. J. Educ. Technol. High. Educ. 18(1), 63 (2021)
Kasneci, E., et al.: Chatgpt for good? on opportunities and challenges of large language models for education. Learn. Individ. Differ. 103, 102274 (2023)
van der Lee, C., Gatt, A., van Miltenburg, E., Krahmer, E.: Human evaluation of automatically generated text: current trends and best practice guidelines. Comput. Speech Lang. 67, 101151 (2021)
Maier, U., Klotz, C.: Personalized feedback in digital learning environments: classification framework and literature review. Comput. Educ. Artif. Intell. 3, 100080 (2022)
Mothilal, R.K., Sharma, A., Tan, C.: Explaining machine learning classifiers through diverse counterfactual explanations. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607–617 (2020)
Pardo, A., et al.: Ontask: delivering data-informed, personalized learning support actions (2018)
Patel, C.R., Pandya, S.K., Sojitra, B.M.: Perspectives of chatgpt in pharmacology education, and research in health care: a narrative review. J. Pharmacol. Pharmacotherapeut. 0976500X231210427 (2023)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Pinger, P., Rakoczy, K., Besser, M., Klieme, E.: Implementation of formative assessment-effects of quality of programme delivery on students’ mathematics achievement and interest. Assess. Educ. Principles Policy Pract. 25(2), 160–182 (2018)
Ramaswami, G., Susnjak, T., Mathrani, A.: On developing generic models for predicting student outcomes in educational data mining. Big Data Cogn. Comput. 6(1), 6 (2022)
Ramaswami, G., Susnjak, T., Mathrani, A.: Supporting students’ academic performance using explainable machine learning with automated prescriptive analytics. Big Data Cogn. Comput. 6(4), 105 (2022)
Ramaswami, G., Susnjak, T., Mathrani, A.: Effectiveness of a learning analytics dashboard for increasing student engagement levels. J. Learn. Anal. 10(3), 115–134 (2023)
Ryan, T., Henderson, M., Ryan, K., Kennedy, G.: Designing learner-centred text-based feedback: a rapid review and qualitative synthesis. Assess. Eval. High. Educ. 46(6), 894–912 (2021)
Ryan, T., Henderson, M., Ryan, K., Kennedy, G.: Identifying the components of effective learner-centred feedback information. Teach. High. Educ. 28(7), 1565–1582 (2023)
Sarsa, S., Denny, P., Hellas, A., Leinonen, J.: Automatic generation of programming exercises and code explanations using large language models. In: Proceedings of the 2022 ACM Conference on International Computing Education Research, vol. 1, pp. 27–43 (2022)
Shapley, L.S., et al.: A value for n-person games (1953)
Smith, B.I., Chimedza, C., Bührmann, J.H.: Individualized help for at-risk students using model-agnostic and counterfactual explanations. In: Education and Information Technologies, pp. 1–20 (2022)
Stasaski, K., Ramanarayanan, V.: Automatic feedback generation for dialog-based language tutors using transformer models and active learning. In: 34th Conference on Neural Information Processing Systems, Vancouver (2020)
Susnjak, T.: Beyond predictive learning analytics modelling and onto explainable artificial intelligence with prescriptive analytics and chatgpt. Int. J. Artif. Intell. Educ. 1–31 (2023)
Troussas, C., Papakostas, C., Krouska, A., Mylonas, P., Sgouropoulou, C.: Personalized feedback enhanced by natural language processing in intelligent tutoring systems. In: Frasson, C., Mylonas, P., Troussas, C. (eds.) ITS 2023. LNCS, pp. 667–677. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-32883-1_58
Van Petegem, C., et al.: Pass/fail prediction in programming courses. J. Educ. Comput. Res. 61(1), 68–95 (2023)
Varank, İ, et al.: Effectiveness of an online automated evaluation and feedback system in an introductory computer literacy course. Eurasia J. Math. Sci. Technol. Educ. 10(5), 395–404 (2014)
Widyahastuti, F., Tjhin, V.U.: Predicting students performance in final examination using linear regression and multilayer perceptron. In: 2017 10th International Conference on Human System Interactions (HSI), pp. 188–192. IEEE (2017)
Winstone, N., Boud, D., Dawson, P., Heron, M.: From feedback-as-information to feedback-as-process: a linguistic analysis of the feedback literature. Assess. Eval. High. Educ. 47(2), 213–230 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liang, Z., Sha, L., Tsai, YS., Gašević, D., Chen, G. (2024). Towards the Automated Generation of Readily Applicable Personalised Feedback in Education. In: Olney, A.M., Chounta, IA., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds) Artificial Intelligence in Education. AIED 2024. Lecture Notes in Computer Science(), vol 14830. Springer, Cham. https://doi.org/10.1007/978-3-031-64299-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-64299-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-64298-2
Online ISBN: 978-3-031-64299-9
eBook Packages: Computer ScienceComputer Science (R0)