Abstract
Online Judges (OJs) have gained substantial traction in programming education due to their ability to simultaneously present problem-solving challenges to students while offering instant feedback and correction. Such technologies are also essential to allow students in remote areas to access quality and equitable education. Nonetheless, OJ systems often lack sufficient amounts of annotated data (i.e., labelled data) about the topics of the problems that they aim to support, which makes choosing appropriate problems hard. Topic annotations hold significant value for instructors when selecting problems for assignments and for novice students seeking independent use of OJ systems. In this work, we propose and evaluate a pre-trained deep learning architecture and an active learning methodology to automatically annotate OJ problems in the context of introductory programming. Our results show that, when using a smaller amount of data, the methodology demonstrates performance comparable to those of the existing state-of-the-art methods for the identical task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)
Athavale, V., Naik, A., Vanjape, R., Shrivastava, M.: Predicting algorithm classes for programming word problems. arXiv preprint arXiv:1903.00830 (2019)
Bachman, P., Sordoni, A., Trischler, A.: Learning algorithms for active learning. In: International Conference on Machine Learning, pp. 301–310. PMLR (2017)
Bez, J.L., Tonin, N.A., Rodegheri, P.R.: URI online judge academic: a tool for algorithms and programming classes. In: 2014 9th International Conference on Computer Science & Education, pp. 149–152. IEEE (2014)
Denny, P., Sarsa, S., Hellas, A., Leinonen, J.: Robosourcing educational resources–leveraging large language models for learnersourcing. arXiv preprint arXiv:2211.04715 (2022)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Farrow, E., Moore, J., Gašević, D.: Analysing discussion forum data: a replication study avoiding data contamination. In: Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe, Arizona, pp. 170–179, 4–8 March 2019. https://doi.org/10.1145/3303772.3303779
Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., Romero, C.: Text mining in education. Wiley Interdisc. Rev. Data Mining Knowl. Discov. 9(6), e1332 (2019)
Hastings, P., Hughes, S., Britt, M.A.: Active learning for improving machine learning of student explanatory essays. In: Penstein Rosé, C., et al. (eds.) AIED 2018. LNCS (LNAI), vol. 10947, pp. 140–153. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93843-1_11
Hogg, C., Jump, M.: Designing autograders for novice programmers. In: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education, vol. 2, p. 1200 (2022)
Ihantola, P., et al.: Educational data mining and learning analytics in programming: Literature review and case studies. In: Proceedings of the 2015 ITiCSE on Working Group Reports, pp. 41–63 (2015)
Intisar, C.M., Watanobe, Y.: Cluster analysis to estimate the difficulty of programming problems. In: Proceedings of the 3rd International Conference on Applications in Information Technology, pp. 23–28 (2018)
Intisar, C.M., Watanobe, Y., Poudel, M., Bhalla, S.: Classification of programming problems based on topic modeling. In: Proceedings of the 2019 7th International Conference on Information and Education Technology, pp. 275–283 (2019)
Jiang, H., He, P., Chen, W., Liu, X., Gao, J., Zhao, T.: Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv:1911.03437 (2019)
Kalyuga, S.: Expertise reversal effect and its implications for learner-tailored instruction. Educ. Psychol. Rev. 19, 509–539 (2007)
Koroteev, M.: BERT: a review of applications in natural language processing and understanding. arXiv preprint arXiv:2103.11943 (2021)
Kostopoulos, G., Kotsiantis, S., Ragos, O., Grapsa, T.N.: Early dropout prediction in distance higher education using active learning. In: 2017 8th International Conference on Information, Intelligence, Systems & Applications (IISA), pp. 1–6. IEEE (2017)
Kurt, V.: Intelligent tutoring systems for continuous, embedded assessment. In: The Future of Assessment, pp. 113–138. Routledge (2005)
Lang, C., Siemens, G., Wise, A., Gasevic, D.: Handbook of Learning Analytics. SOLAR, Society for Learning Analytics and Research, New York (2017)
Lin, J., et al.: Enhancing educational dialogue act classification with discourse context and sample informativeness. IEEE Trans. Learn. Technol. p. (2023, in press). https://doi.org/10.1109/TLT.2023.3302573
Mead, J., et al.: A cognitive approach to identifying measurable milestones for programming skill acquisition. ACM SIGCSE Bull. 38(4), 182–194 (2006)
Pereira, F.D., et al.: Towards supporting CS1 instructors and learners with fine-grained topic detection in online judges (2022)
Pereira, F.D., et al.: Towards supporting CS1 instructors and learners with fine-grained topic detection in online judges. IEEE Access 11, 22513–22525 (2023)
Pereira, F.D., et al.: Towards a human-AI hybrid system for categorising programming problems. In: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, pp. 94–100 (2021)
Rolim, V., Mello, R.F., Nascimento, A., Lins, R.D., Gašević, D.: Reducing the size of training datasets in the classification of online discussions. In: 2021 International Conference on Advanced Learning Technologies (ICALT), pp. 179–183. IEEE (2021)
Sarsa, S., Leinonen, J., Hellas, A., et al.: Empirical evaluation of deep learning models for knowledge tracing: of hyperparameters and metrics on performance and replicability. J. Educ. Data Mining 14(2) (2022)
Schröder, C., Müller, L., Niekler, A., Potthast, M.: Small-text: active learning for text classification in Python. arXiv preprint arXiv:2107.10314 (2021)
Settles, B.: From theories to queries: active learning in practice. In: Active Learning and Experimental Design Workshop in Conjunction with AISTATS 2010, pp. 1–18. JMLR Workshop and Conference Proceedings (2011)
Sharma, M., Bilgic, M.: Evidence-based uncertainty sampling for active learning. Data Min. Knowl. Disc. 31, 164–202 (2017)
Swaray, R.: An evaluation of a group project designed to reduce free-riding and promote active learning. Assess. Eval. Higher Educ. 37(3), 285–292 (2012)
Tan, W., et al.: Does informativeness matter? Active learning for educational dialogue act classification. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds.) AIED 2023. Lecture Notes in Computer Science, vol. 13916, pp. 176–188. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-36272-9_15
Teague, D., Lister, R.: Longitudinal think aloud study of a novice programmer. In: Conferences in Research and Practice in Information Technology Series (2014)
Wasik, S., Antczak, M., Badura, J., Laskowski, A., Sternal, T.: A survey on online judge systems and their applications. ACM Comput. Surv. (CSUR) 51(1), 1–34 (2018)
Yang, T.Y., Baker, R.S., Studer, C., Heffernan, N., Lan, A.S.: Active learning for student affect detection. In: Proceedings of the 12th International Conference on Educational Data Mining, EDM 2019, Montréal, Canada, 2–5 July 2019. International Educational Data Mining Society (IEDMS) 2019, pp. 208–217. Université du Québec; Polytechnique Montréal (2019)
Zhao, W.X., Zhang, W., He, Y., Xie, X., Wen, J.R.: Automatically learning topics and difficulty levels of problems in online judge systems. ACM Trans. Inf. Syst. (TOIS) 36(3), 1–33 (2018)
Acknowledgments
This research, carried out within the scope of the Samsung-UFAM Project for Education and Research (SUPER), according to Article 39 of Decree \(\textrm{n}^\circ \)10.521/2020, was funded by Samsung Electronics of Amazonia Ltda., under the terms of Federal Law \(\textrm{n}^\circ \)8.387/1991 through agreement 001/2020, signed with UFAM and FAEPI, Brazil. This study was financed in part by Conselho Nacional de Desenvolvimento Científico e Tecnológico- Brasil- CNPq (Process 303443/2023-5) and Fundação de Amparo a Pesquisa do Estado do Amazonas-FAPEAM (Process 01.02.016301.02770/2021-63). This study was financed in part by the Acuity Insights under the Alo Grant program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pereira, F.D. et al. (2024). From Sparse to Smart: Leveraging AI for Effective Online Judge Problem Classification in Programming Education. In: Ferreira Mello, R., Rummel, N., Jivet, I., Pishtari, G., Ruipérez Valiente, J.A. (eds) Technology Enhanced Learning for Inclusive and Equitable Quality Education. EC-TEL 2024. Lecture Notes in Computer Science, vol 15159. Springer, Cham. https://doi.org/10.1007/978-3-031-72315-5_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-72315-5_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72314-8
Online ISBN: 978-3-031-72315-5
eBook Packages: Computer ScienceComputer Science (R0)