Advancing Automated Assessment Tools - Opportunities for Innovations in Upper-level Computing Courses: A Position Paper
Authors: 
Steffan Hooper, 
Burkhard C. Wünsche, 
Andrew Luxton-Reilly, 
Paul Denny, 
Tony Haoran FengAuthors Info & Claims
Steffan Hooper, Burkhard C. Wünsche, Andrew Luxton-Reilly, Paul Denny, Tony Haoran FengAuthors Info & ClaimsPages 519 - 525
Abstract
Teaching large cohorts in upper-level computing courses is challenging, as providing rapid feedback and marking at scale is difficult without significant resources. Many institutions lack funds to employ a large number of skilled markers or such markers are simply not available.
Automated Assessment Tools (AATs) can be used to grade and provide feedback on student work, and they support incremental learning by enabling students to make multiple submissions and to correct automatically detected mistakes. Despite their advantages, the application of AATs in certain fields is limited, especially where the nature of the assessments is complex. For example, in computer graphics (CG) courses, assessing and providing feedback requires both objective source code review and subjective visual inspection of graphical output. This makes developing AATs for these courses difficult. Although various attempts have been made, we lack an overview of the field that synthesises the findings from a disparate set of papers.
In this position paper, we assert that AATs should be used more frequently for CG courses, and if carefully designed with a focus on learners' needs, are useful tools to not only reduce workload for educators, but also improve learning. We undertake a literature review focusing on AATs in CG courses to gain an understanding of what has been achieved, the challenges that exist, and which areas can be developed. With this paper we want to initiate a discussion about the challenges and opportunities of AATs in CG and other computing fields that require a diverse set of skills.
References
[1]
Kirsti M Ala-Mutka. 2005. A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education, Vol. 15, 2 (2005), 83--102. https://doi.org/10.1080/08993400500150747
[2]
Carlos Andujar, Antonio Chica, Marta Fairén, and Alvar Vinacua. 2018. GL-Socket: A CG Plugin-based Framework for Teaching and Assessment. In EG 2018 - Education Papers, Frits Post and Jirí ?ára (Eds.). The Eurographics Association. https://doi.org/10.2312/eged.20181003
[3]
Carlos Andujar, Marc Comino Trinidad, Marta Fairén, and Àlvar Vinacua. 2020a. A Tool for N-way Analysis of Programming Exercises. In EuroVis 2020 - Posters, Jan By?ka and Stefan Jänicke (Eds.). The Eurographics Association. https://doi.org/10.2312/eurp.20201122
[4]
Carlos Andujar, Cristina Raluca Vijulie, and Alvar Vinacua. 2019. A Parser-based Tool to Assist Instructors in Grading Computer Graphics Assignments. In Eurographics 2019 - Education Papers, Marco Tarini and Eric Galin (Eds.). The Eurographics Association. https://doi.org/10.2312/eged.20191025
[5]
Carlos Andujar, Cristina R. Vijulie, and Àlvar Vinacua. 2020b. Syntactic and semantic analysis for extended feedback on computer graphics assignments. IEEE computer graphics and applications, Vol. 40, 3 (2020), 105--111.
[6]
Prince Steven Annor, Edwin Kayang, Samuel Boateng, and George Boateng. 2022. AutoGrad: Automated Grading Software for Mobile Game Assignments in SuaCode Courses. In Proceedings of the 10th Computer Science Education Research Conference (Virtual Event, Netherlands) (CSERC '21). Association for Computing Machinery, New York, NY, USA, 79--85. https://doi.org/10.1145/3507923.3507954
[7]
Dennis G. Balreira, Marcelo Walter, and Dieter W. Fellner. 2018. A survey of the contents in introductory Computer Graphics courses. Computers & Graphics, Vol. 77 (2018), 88--96. https://doi.org/10.1016/j.cag.2018.10.001
[8]
Jeremiah Blanchard, John R. Hott, Vincent Berry, Rebecca Carroll, Bob Edmison, Richard Glassey, Oscar Karnalim, Brian Plancher, and Seán Russell. 2022. Stop Reinventing the Wheel! Promoting Community Software in Computing Education. In Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (Dublin, Ireland) (ITiCSE-WGR '22). Association for Computing Machinery, New York, NY, USA, 261--292. https://doi.org/10.1145/3571785.3574129
[9]
Daniel Bruzual, Maria L. Montoya Freire, and Mario Di Francesco. 2020. Automated Assessment of Android Exercises with Cloud-Native Technologies. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE '20). Association for Computing Machinery, New York, NY, USA, 40--46. https://doi.org/10.1145/3341525.3387430
[10]
Andrea di Filippo, Sara Antinozzi, Nicola Cappetti, and Francesco Villecco. 2023. Methodologies for assessing the quality of 3D models obtained using close-range photogrammetry. International Journal on Interactive Design and Manufacturing (2023), 1--8. https://doi.org/10.1007/s12008-023-01428-z
[11]
Christopher Douce, David Livingstone, and James Orwell. 2005. Automatic Test-Based Assessment of Programming: A Review. J. Educ. Resour. Comput., Vol. 5, 3 (Sept. 2005), 4--es. https://doi.org/10.1145/1163405.1163409
[12]
Daniel Galan, Ruben Heradio, Hector Vargas, Ismael Abad, and Jose A. Cerrada. 2019. Automated Assessment of Computer Programming Practices: The 8-Years UNED Experience. IEEE Access, Vol. 7 (2019), 130113--130119. https://doi.org/10.1109/ACCESS.2019.2938391
[13]
Blake Hodgkinson, Christof Lutteroth, and Burkhard Wünsche. 2016. glGetFeedback - Towards automatic feedback and assessment for OpenGL 3D modelling assignments. In 2016 International Conference on Image and Vision Computing New Zealand (IVCNZ). 1--6. https://doi.org/10.1109/IVCNZ.2016.7804418
[14]
Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Sepp"al"a. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '10). Association for Computing Machinery, New York, NY, USA, 86--93. https://doi.org/10.1145/1930464.1930480
[15]
International Game Developers Association. 2008. IGDA Curriculum Framework The Study of Games and Game Development. Retrieved August 14, 2023 from https://web.archive.org/web/20090419214800/http://www.igda.org/wiki/images/e/ee/Igda2008cf.pdf.
[16]
HuaQiang Jiang, Liang Zhang, and WangQiong Ye. 2009. The Automatic Evaluation Strategies and Methods of Multimedia Work Assignments. In 2009 International Conference on Computational Intelligence and Software Engineering. 1--5. https://doi.org/10.1109/CISE.2009.5365271
[17]
Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. ACM Trans. Comput. Educ., Vol. 19, 1, Article 3 (Sept. 2018), bibinfonumpages43 pages. https://doi.org/10.1145/3231711
[18]
Fabrizio Lamberti, Andrea Sanna, Gianluca Paravati, and Gilles Carlevaris. 2014. Automatic Grading of 3D Computer Animation Laboratory Assignments. IEEE Transactions on Learning Technologies, Vol. 7, 3 (2014), 280--290. https://doi.org/10.1109/TLT.2014.2340861
[19]
Andrew Luxton-Reilly, Ewan Tempero, Nalin Arachchilage, Angela Chang, Paul Denny, Allan Fowler, Nasser Giacaman, Igor Kontorovich, Danielle Lottridge, Sathiamoorthy Manoharan, Shyamli Sindhwani, Paramvir Singh, Ulrich Speidel, Sudeep Stephen, Valerio Terragni, Jacqueline Whalley, Burkhard Wuensche, and Xinfeng Ye. 2023. Automated Assessment: Experiences From the Trenches. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE '23). Association for Computing Machinery, New York, NY, USA, 1--10. https://doi.org/10.1145/3576123.3576124
[20]
Evan Maicus, Matthew Peveler, Andrew Aikens, and Barbara Cutler. 2020. Autograding Interactive Computer Graphics Applications. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE '20). Association for Computing Machinery, New York, NY, USA, 1145--1151. https://doi.org/10.1145/3328778.3366954
[21]
Evan E. Maicus. 2021. Automated Grading for Advanced Topics Courses. Ph.,D. Dissertation. Rensselaer Polytechnic Institute, USA. AAI28544209.
[22]
Swapneel Mehta, Chirag Raman, Nitin Ayer, and Sameer Sahasrabudhe. 2018. Auto-Grading for 3D Modeling Assignments in MOOCs. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT). 51--53. https://doi.org/10.1109/ICALT.2018.00012
[23]
Eerik Muuli, Kaspar Papli, Eno T onisson, Marina Lepp, Tauno Palts, Reelika Suviste, Merilin S"ade, and Piret Luik. 2017. Automatic Assessment of Programming Assignments Using Image Recognition. In Data Driven Approaches in Digital Education, Élise Lavoué, Hendrik Drachsler, Katrien Verbert, Julien Broisin, and Mar Pérez-Sanagustín (Eds.). Springer International Publishing, Cham, 153--163.
[24]
Eerik Muuli, Eno T onisson, Marina Lepp, Piret Luik, Tauno Palts, Reelika Suviste, Kaspar Papli, and Merilin S"ade. 2020. Using Image Recognition to Automatically Assess Programming Tasks with Graphical Output. Education and Inf. Technologies, Vol. 25, 6 (Nov. 2020), 5185--5203. https://doi.org/10.1007/s10639-020--10218-z
[25]
José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Trans. Comput. Educ., Vol. 22, 3, Article 34 (June 2022), bibinfonumpages40 pages. https://doi.org/10.1145/3513140
[26]
Gianluca Paravati, Fabrizio Lamberti, Valentina Gatteschi, Claudio Demartini, and Paolo Montuschi. 2017. Point Cloud-Based Automatic Assessment of 3D Computer Animation Courseworks. IEEE Transactions on Learning Technologies, Vol. 10, 4 (2017), 532--543. https://doi.org/10.1109/TLT.2016.2638811
[27]
Vreda Pieterse. 2013. Automated Assessment of Programming Assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research (Arnhem, Netherlands) (CSERC '13). Open Universiteit, Heerlen, Heerlen, NLD, 45--56.
[28]
Vreda Pieterse and Christoph Stallmann. 2014. Managing a Large Tertiary Computer Science Class. In Proceedings of the Computer Science Education Research Conference (Berlin, Germany) (CSERC '14). Association for Computing Machinery, New York, NY, USA, 79--90. https://doi.org/10.1145/2691352.2691359
[29]
Luka Fürst, and Viljan Mahni?. 2015. Introduction of the automated assessment of homework assignments in a university-level programming course. In 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). 761--766. https://doi.org/10.1109/MIPRO.2015.7160373
[30]
Lindomar Rocha and Rui Rodrigues. 2021. GraphJudge: A System for Assisted Assessment of Computer Graphics Assignments. In V Ibero-American Symposium in Computer Graphics, F. Silva, D. Gutierrez, J. Rodríguez, and M. Figueiredo (Eds.). The Eurographics Association. https://doi.org/10.2312/pt.20111126
[31]
Rohaida Romli, Shahida Sulaiman, and Kamal Zuhairi Zamli. 2010. Automatic Programming Assessment and Test Data Generation A review on its approaches. In 2010 International Symposium on Information Technology, Vol. 3. 1186--1192. https://doi.org/10.1109/ITSIM.2010.5561488
[32]
Andrea Sanna, Fabrizio Lamberti, Gianluca Paravati, and Claudio Demartini. 2012. Automatic Assessment of 3D Modeling Exams. IEEE Transactions on Learning Technologies, Vol. 5, 01 (2012), 2--10. https://doi.org/10.1109/TLT.2011.4
[33]
Andrea Sanna, Federico Manuri, Francesco De Pace, Fabrizio Valpreda, and Claudio Fornaro. 2022. An Integrated Framework to Support Bachelor Design Students in 3-D Modeling. IEEE Transactions on Learning Technologies, Vol. 15, 6 (2022), 673--684. https://doi.org/10.1109/TLT.2022.3215208
[34]
Draylson M. Souza, Katia R. Felizardo, and Ellen F. Barbosa. 2016. A Systematic Literature Review of Assessment Tools for Programming Assignments. In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET). 147--156. https://doi.org/10.1109/CSEET.2016.48
[35]
Burkhard Claus Wuensche, Kai-Cheung Leung, Davis Dimalen, Wannes van der Mark, Thomas Suselo, Marylyn Alex, Alex Shaw, Andrew Luxton-Reilly, and Richard Lobb. 2022. Using an Assessment Tool to Create Sandboxes for Computer Graphics Teaching in an Online Environment. In Proceedings of the 10th Computer Science Education Research Conference (Virtual Event, Netherlands) (CSERC '21). Association for Computing Machinery, New York, NY, USA, 21--30. https://doi.org/10.1145/3507923.3507929
[36]
Burkhard C. Wünsche, Zhen Chen, Lindsay Shaw, Thomas Suselo, Kai-Cheung Leung, Davis Dimalen, Wannes van der Mark, Andrew Luxton-Reilly, and Richard Lobb. 2018. Automatic Assessment of OpenGL Computer Graphics Assignments. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018). Association for Computing Machinery, New York, NY, USA, 81--86. https://doi.org/10.1145/3197091.3197112
[37]
Burkhard C. Wünsche, Edward Huang, Lindsay Shaw, Thomas Suselo, Kai-Cheung Leung, Davis Dimalen, Wannes van der Mark, Andrew Luxton-Reilly, and Richard Lobb. 2019. CodeRunnerGL - An Interactive Web-Based Tool for Computer Graphics Teaching and Assessment. In 2019 International Conference on Electronics, Information, and Communication (ICEIC). 1--7. https://doi.org/10.23919/ELINFOCOM.2019.8706402
[38]
Francisco A. Zampirolli, João M. Borovina Josko, Mirtha L. F. Venero, Guiou Kobayashi, Francisco J. Fraga, Denise Goya, and Heitor R. Savegnago. 2021. An experience of automated assessment in a large-scale introduction programming course. Computer Applications in Engineering Education, Vol. 29, 5 (2021), 1284--1299. https://doi.org/10.1002/cae.22385 io
Index Terms
- Advancing Automated Assessment Tools - Opportunities for Innovations in Upper-level Computing Courses: A Position Paper
Recommendations
An Online Marking, Feedback and Moderation Tool for Computer Science Higher Education Courses
CEP '23: Proceedings of the 7th Conference on Computing Education PracticeWhile the value of marking, feedback and their moderation are widely acknowledged, there is great variety in their application in Higher Education Computer Science (CS) courses, due to the diversity of topics, learning outcomes, assessment types, and ...
Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
ICER '18: Proceedings of the 2018 ACM Conference on International Computing Education ResearchMost novice programmers are not explicitly aware of the problem-solving process used to approach programming problems and cannot articulate to an instructor where they are in that process. Many are now arguing that this skill, called metacognitive ...
Implementing an Automated Assessment Tool Supporting Assessment and Feedback on Assessments for Novice Programmers in the Higher-Education Setting.
ITiCSE '22: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2Many Higher-Education (HE) providers in the UK have observed and reported the number of novice programming students is rising HESA1, with no proportional increase in teaching-staff numbers HESA2. Novice programming students need frequent practice ...
Comments
Information & Contributors
Information
Published In
March 2024
1583 pages
ISBN:9798400704239
DOI:10.1145/3626252
- General Chairs:
- Ben Stephenson,
- Jeffrey A. Stone,
- Program Chairs:
- Lina Battestilli,
- Samuel A. Rebelsky,
- Libby Shoop
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 March 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SIGCSE 2024
Sponsor:
SIGCSE 2024: The 55th ACM Technical Symposium on Computer Science Education
March 20 - 23, 2024
OR, Portland, USA
Acceptance Rates
Overall Acceptance Rate 1,595 of 4,542 submissions, 35%
Upcoming Conference
SIGCSE Virtual 2024
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 157Total Downloads
- Downloads (Last 12 months)157
- Downloads (Last 6 weeks)27
Reflects downloads up to 11 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in