research-article

An automatic grading system for a high school-level computational thinking course

Authors:

Sirazum Munira Tisha,

Rufino A. Oregon,

Gerald Baumgartner,

Fernando Alegre,

Juana MorenoAuthors Info & Claims

SEENG '22: Proceedings of the 4th International Workshop on Software Engineering Education for the Next Generation

Pages 20 - 27

https://doi.org/10.1145/3528231.3528357

Published: 07 February 2023 Publication History

Abstract

Automatic grading systems help lessen the load of manual grading. Most existent autograders are based on unit testing, which focuses on the correctness of the code, but has limited scope for judging code quality. Moreover, it is cumbersome to implement unit testing for evaluating graphical output code. We propose an autograder that can effectively judge the code quality of the visual output codes created by students enrolled in a high school-level computational thinking course. We aim to provide suggestions to teachers on an essential aspect of their grading, namely the level of student competency in using abstraction within their codes. A dataset from five different assignments, including open-ended problems, is used to evaluate the effectiveness of our autograder. Our initial experiments show that our method can classify the students' submissions even for open-ended problems, where existing autograders fail to do so. Additionally, survey responses from course teachers support the importance of our work.

References

[1]

Fernando Alegre, John Underwoood, Juana Moreno, and Mario Alegre. 2020. Introduction to Computational Thinking: A New High School Curriculum Using CodeWorld. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE '20). Association for Computing Machinery, New York, NY, USA, 992--998.

Digital Library

[2]

Sara Mernissi Arifi, Ismail Nait Abdellah, Azeddine Zahi, and Rachid Benabbou. 2015. Automatic program assessment using static and dynamic analysis. In 2015 Third World Conference on Complex Systems (WCCS). 1--6.

[3]

David Arthur and Sergei Vassilvitskii. 2006. k-means++: The advantages of careful seeding. Technical Report. Stanford.

[4]

Vera Barstad, Morten Goodwin, and Terje Gjøsæter. 2014. Predicting source code quality with static analysis and machine learning. In Norsk IKT-konferanse for forskning og utdanning.

[5]

Clara Benac Earle, Lars-Åke Fredlund, and John Hughes. 2016. Automatic Grading of Programming Exercises Using Property-Based Testing. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (Arequipa, Peru) (ITiCSE '16). Association for Computing Machinery, New York, NY, USA, 47--52.

Digital Library

[6]

Guillaume Bouchard and Bill Triggs. 2004. The tradeoff between generative and discriminative classifiers. In 16th IASC International Symposium on Computational Statistics (COMPSTAT'04). 721--728.

[7]

Georg Brandl, Matthäus Chajdas, Armin Ronacher, Pocoo, and Tim Hatch. 2006. Python syntax highlighter. https://pygments.org

[8]

Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD '16). Association for Computing Machinery, New York, NY, USA, 785--794.

Digital Library

[9]

CodeGrade. 2020. Deliver engaging feedback on code. https://www.codegrade.com

[10]

CodeWorld. 2020. Educational computer programming environment using Haskell. https://github.com/google/codeworld

[11]

CSforALL. 2021. CSforAll. https://www.csforall.org/.

[12]

Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018. CatBoost: gradient boosting with categorical features support. ArXiv abs/1810.11363 (2018).

[13]

Stephen H Edwards and Manuel A Perez-Quinones. 2008. Web-CAT: automatically grading programming assignments. In Proceedings of the 13th annual conference on Innovation and technology in computer science education. 328--328.

Digital Library

[14]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 [cs.CL]

[15]

Chase Geigle, ChengXiang Zhai, and Duncan C Ferguson. 2016. An exploration of automated grading of complex assignments. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. 351--360.

Digital Library

[16]

Gradescope. 2020. Online grading platform. https://www.gradescope.com

[17]

Anshul Gupta and Neel Sundaresan. 2018. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'18) Deep Learning Day.

[18]

MH Halstead and T McCabe. 1976. A software complexity measure. IEEE Trans. Software Engineering 2, 12 (1976), 308--320.

[19]

HLint. 2020. Linter for Haskell. https://github.com/ndmitchell/hlint

[20]

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli calling international conference on computing education research. 86--93.

Digital Library

[21]

Florian Klapproth, Lisa Federkeil, Franziska Heinschke, and Tanja Jungmann. 2020. Teachers' Experiences of Stress and Their Coping Strategies during COVID-19 Induced Distance Teaching. Journal of Pedagogical Research 4, 4 (2020), 444--452.

[22]

LAB.COMPUTER. 2020. Put your computer laboratory on the browser. https://lab.computer

[23]

Xiao Liu, Shuai Wang, Pei Wang, and Dinghao Wu. 2019. Automatic Grading of Programming Assignments: A Formal Semantics Based Approach. In Proceedings 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training ICSE-SEET 2019, 25--31 May 2019, Montreal, Canada.

Digital Library

[24]

Simon Marlow et al. 2010. Haskell 2010 language report. Available online http://www.haskell.org/(May 2011) (2010).

[25]

Dhendra Marutho, Sunarna Hendra Handaka, Ekaprana Wijaya, et al. 2018. The determination of cluster number at k-mean using elbow method and purity evaluation on headline news. In 2018 International Seminar on Application for Technology of Information and Communication. IEEE, 533--538.

[26]

Carnegie Mellon University's School of Computer Science (SCS). 2020. Carnegie Mellon University Computer Science Academy. https://academy.cs.cmu.edu/

[27]

Gabriella Oliveira, Jorge Grenha Teixeira, Ana Torres, and Carla Morais. 2021. An exploratory study on the emergency remote education experience of higher education students and teachers during the COVID-19 pandemic. British Journal of Educational Technology (2021).

[28]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[29]

Christopher Piech, Ali Malik, Kylie Jue, and Mehran Sahami. 2021. Code in Place: Online Section Leading for Scalable Human-Centered Learning. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 973--979.

Digital Library

[30]

Joachim Schwieren, Gottfried Vossen, and Peter Westerkamp. 2006. Using Software Testing Techniques for Efficient Handling of Programming Exercises in an e-Learning Platform. Electronic Journal of e-Learning 4, 1 (2006), 87--94.

[31]

Gursimran Singh, Shashank Srikant, and Varun Aggarwal. 2016. Question independent grading using machine learning: The case of computer program grading. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 263--272.

Digital Library

[32]

Shashank Srikant and Varun Aggarwal. 2013. Automatic grading of computer programs: A machine learning approach. In 2013 12th International Conference on Machine Learning and Applications, Vol. 1. IEEE, 85--92.

Digital Library

[33]

Shashank Srikant and Varun Aggarwal. 2014. A System to Grade Computer Programming Skills Using Machine Learning. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, New York, USA) (KDD '14). Association for Computing Machinery, New York, NY, USA, 1887--1896.

Digital Library

[34]

Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. 2021. ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback. arXiv preprint arXiv:2107.14035 (2021).

Cited By

Abubakari MShafik WHidayatullah A(2024)Evaluating the Potential of Artificial Intelligence in Islamic Religious EducationAI-Enhanced Teaching Methods10.4018/979-8-3693-2728-9.ch010(216-239)Online publication date: 14-Jun-2024
https://doi.org/10.4018/979-8-3693-2728-9.ch010
Krusche STenbergen B(2023)Software Engineering Education for the Next Generation SEENG 2022 Workshop ReportACM SIGSOFT Software Engineering Notes10.1145/3573074.357310448:1(112-116)Online publication date: 17-Jan-2023
https://dl.acm.org/doi/10.1145/3573074.3573104

Recommendations

Computational thinking in high school courses
SIGCSE '10: Proceedings of the 41st ACM technical symposium on Computer science education

The number of undergraduates entering computer science has declined in recent years. This is paralleled by a drop in the number of high school students taking the CS AP exam and the number of high schools offering computer science courses. The declines ...
Changing perceptions of computer science and computational thinking among high school teachers

Interest in specific college majors begins in high school. The influence of high school math and science teachers is important, both for high school preparation of future computer science undergraduates, and for the understanding and interest in ...
Teaching Code Quality in High School Programming Courses - Understanding Teachers’ Needs
ACE '22: Proceedings of the 24th Australasian Computing Education Conference

The introduction of computer programming into schools has meant that many teachers who are not trained in Computer Science are expected to become familiar with computing concepts and techniques. For senior high school programming students studying text-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SEENG '22: Proceedings of the 4th International Workshop on Software Engineering Education for the Next Generation

May 2022

70 pages

ISBN:9781450393362

DOI:10.1145/3528231

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 17, 2022

Pennsylvania, Pittsburgh

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
95
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)5

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Abubakari MShafik WHidayatullah A(2024)Evaluating the Potential of Artificial Intelligence in Islamic Religious EducationAI-Enhanced Teaching Methods10.4018/979-8-3693-2728-9.ch010(216-239)Online publication date: 14-Jun-2024
https://doi.org/10.4018/979-8-3693-2728-9.ch010
Krusche STenbergen B(2023)Software Engineering Education for the Next Generation SEENG 2022 Workshop ReportACM SIGSOFT Software Engineering Notes10.1145/3573074.357310448:1(112-116)Online publication date: 17-Jan-2023
https://dl.acm.org/doi/10.1145/3573074.3573104

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents