Cited By
View all- Soares JBrito MGonçalves C(2023)Codeflex 2.0Internet of Behaviors Implementation in Organizational Contexts10.4018/978-1-6684-9039-6.ch003(40-67)Online publication date: 1-Nov-2023
Automated Assessment: Experiences From the Trenches
Authors: 
Andrew Luxton-Reilly, 
Ewan Tempero, 
Nalin Arachchilage, Angela Chang, 
Paul Denny, Allan Fowler, Nasser Giacaman, Igor Kontorovich, 
Danielle Lottridge, Sathiamoorthy Manoharan, Shyamli Sindhwani, Paramvir Singh, Ulrich Speidel, Sudeep Stephen, Valerio Terragni, 
Jacqueline Whalley, Burkhard Wuensche, and Xinfeng YeAuthors Info & Claims
Andrew Luxton-Reilly, Ewan Tempero, Nalin Arachchilage, Angela Chang, Paul Denny, Allan Fowler, Nasser Giacaman, Igor Kontorovich, Danielle Lottridge, Sathiamoorthy Manoharan, Shyamli Sindhwani, Paramvir Singh, Ulrich Speidel, Sudeep Stephen, Valerio Terragni, Jacqueline Whalley, Burkhard Wuensche, and Xinfeng YeAuthors Info & ClaimsJanuary 2023
Pages 1 - 10
Abstract
Automated assessment is commonly used across the spectrum of computing courses offered by Tertiary institutions. Such assessment is frequently intended to address the scalability of feedback that is essential for learning, and assessment for accreditation purposes. Although many reviews of automated assessment have been reported, the voices of teachers are not present. In this paper we present a variety of cases that illustrate some of the varied motivations and experiences of teaching using automated assessment.
References
[1]
Kirsti M Ala-Mutka. 2005. A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education 15, 2 (2005), 83–102.
[2]
Elisa Baniassad, Lucas Zamprogno, Braxton Hall, and Reid Holmes. 2021. STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE ’21). ACM, New York, NY, USA, 1062–1068. https://doi.org/10.1145/3408877.3432430
[3]
Leopold Bayerlein. 2014. Students’ feedback preferences: how do students react to timely and automatically generated assessment feedback?Assessment & Evaluation in Higher Education 39, 8 (2014), 916–931.
[4]
Janet Carter, Kirsti Ala-Mutka, Ursula Fuller, Martin Dick, John English, William Fone, and Judy Sheard. 2003. How Shall We Assess This?SIGCSE Bull. 35, 4 (June 2003), 107–123. https://doi.org/10.1145/960492.960539
[5]
Sophie H. Cormack, Laurence A. Eagle, and Mark S. Davies. 2020. A large-scale test of the relationship between procrastination and performance using learning analytics. Assessment & Evaluation in Higher Education 45, 7 (2020), 1046–1059.
[6]
Galina Deeva, Daria Bogdanova, Estefanía Serral, Monique Snoeck, and Jochen De Weerdt. 2021. A review of automated feedback systems for learners: Classification framework, challenges and opportunities. Computers & Education 162 (2021), 104094. https://doi.org/10.1016/j.compedu.2020.104094
[7]
Paul Denny, Jacqueline Whalley, and Juho Leinonen. 2021. Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback. In Australasian Computing Education Conference(Virtual, SA, Australia) (ACE ’21). ACM, New York, NY, USA, 88–95. https://doi.org/10.1145/3441636.3442309
[8]
Christopher Douce, David Livingstone, and James Orwell. 2005. Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing (JERIC) 5, 3 (2005), 1–14.
[9]
Stephen H. Edwards, Joshua Martin, and Clfford A. Shaffer. 2015. Examining Classroom Interventions to Reduce Procrastination. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education(Vilnius, Lithuania) (ITiCSE ’15). ACM, New York, NY, USA, 254–259.
[10]
Emily Faulconer, John Griffith, and Amy Gruss. 2022. The impact of positive feedback on student outcomes and perceptions. Assessment & Evaluation in Higher Education 47, 2 (2022), 259–268. https://doi.org/10.1080/02602938.2021.1910140
[11]
Graham Gibbs and Claire Simpson. 2005. Conditions under which assessment supports students’ learning. Learning and teaching in higher education1 (2005), 3–31.
[12]
Yoshiko Goda, Masanori Yamada, Hiroshi Kato, Takeshi Matsuda, Yutaka Saito, and Hiroyuki Miyagawa. 2015. Procrastination and other learning behavioral types in e-learning and their relationship with learning outcomes. Learning and Individual Differences 37 (2015), 72–80. https://doi.org/10.1016/j.lindif.2014.11.001
[13]
John Hattie and Helen Timperley. 2007. The Power of Feedback. Review of Educational Research 77, 1 (2007), 81–112. https://doi.org/10.3102/003465430298487
[14]
Fiona Hyland and Ken Hyland. 2001. Sugaring the pill: Praise and criticism in written feedback. Journal of Second Language Writing 10, 3 (2001), 185–212. https://doi.org/10.1016/S1060-3743(01)00038-8
[15]
Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research(Koli, Finland) (Koli Calling ’10). ACM, New York, NY, USA, 86–93. https://doi.org/10.1145/1930464.1930480
[16]
Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE) 19, 1 (2018), 1–43.
[17]
Eun Jung Kim and Kyeong Ryong Lee. 2019. Effects of an examiner’s positive and negative feedback on self-assessment of skill performance, emotional response, and self-efficacy in Korea: a quasi-experimental study. BMC medical education 19, 1 (2019), 1–7.
[18]
George Kinnear, Ian Jones, Chris Sangwin, Maryam Alarfaj, Ben Davies, Sam Fearn, Colin Foster, André Heck, Karen Henderson, Tim Hunt, Paola Iannone, Igor’ Kontorovich, Niclas Larson, Tim Lowe, John Christopher Meyer, Ann O’Shea, Peter Rowlett, Indunil Sikurajapathi, and Thomas Wong. 2022. A collaboratively-derived research agenda for e-assessment in undergraduate mathematics. Int. J. Res. Undergrad. Math. Ed. (8 2022), 31 pages. https://doi.org/10.1007/s40753-022-00189-6
[19]
Richard Lobb and Jenny Harlow. 2016. Coderunner: A Tool for Assessing Computer Programming Skills. ACM Inroads 7, 1 (feb 2016), 47–51. https://doi.org/10.1145/2810041
[20]
Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). ACM, New York, NY, USA, 55–106. https://doi.org/10.1145/3293881.3295779
[21]
Sathiamoorthy Manoharan and Ulrich Speidel. 2020. Individualized Assessments using Dividni-Enhancing Learning via Assessments Unique to Every Student. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. ACM, New York, NY, USA, 1419–1419. https://doi.org/10.1145/3328778.3372545
[22]
Emma Mulliner and Matthew Tucker. 2017. Feedback on feedback practice: perceptions of students and academics. Assessment & Evaluation in Higher Education 42, 2 (2017), 266–288. https://doi.org/10.1080/02602938.2015.1103365
[23]
David J. Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31, 2 (2006), 199–218. https://doi.org/10.1080/03075070600572090
[24]
Claudia Ott, Anthony Robins, and Kerry Shephard. 2016. Translating Principles of Effective Feedback for Students into the CS1 Context. ACM Trans. Comput. Educ. 16, 1, Article 1 (jan 2016), 27 pages. https://doi.org/10.1145/2737596
[25]
José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Trans. Comput. Educ. 22, 3, Article 34 (jun 2022), 40 pages. https://doi.org/10.1145/3513140
[26]
Chris Sangwin. 2013. Computer aided assessment of mathematics. Oxford University Press.
[27]
Draylson M. Souza, Katia R. Felizardo, and Ellen F. Barbosa. 2016. A Systematic Literature Review of Assessment Tools for Programming Assignments. In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET). IEEE, 147–156. https://doi.org/10.1109/CSEET.2016.48
[28]
Jacqueline L. Whalley and Anne Philpott. 2011. A Unit Testing Approach to Building Novice Programmers’ Skills and Confidence. In Proceedings of the 13th Australasian Computing Education Conference (Perth, Australia) (ACE ’11). Australian Computer Society, Inc., Australia, 113–118.
[29]
Benedikt Wisniewski, Klaus Zierer, and John Hattie. 2020. The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research. Frontiers in Psychology 10 (2020). https://doi.org/10.3389/fpsyg.2019.03087
[30]
Shulin Yu, Feng Geng, Chunhong Liu, and Yao Zheng. 2021. What works may hurt: The negative side of feedback in second language writing. Journal of Second Language Writing 54 (2021), 100850. https://doi.org/10.1016/j.jslw.2021.100850
Index Terms
- Automated Assessment: Experiences From the Trenches
Recommendations
Automated Assessment in Computer Science Education: A State-of-the-Art Review
Practical programming competencies are critical to the success in computer science (CS) education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through ...
Feedback on Student Programming Assignments: Teaching Assistants vs Automated Assessment Tool
Koli Calling '23: Proceedings of the 23rd Koli Calling International Conference on Computing Education ResearchExisting research does not quantify and compare the differences between automated and manual assessment in the context of feedback on programming assignments. This makes it hard to reason about the effects of adopting automated assessment at the expense ...
Improving Student Learning with Automated Assessment
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1Students taking computing courses develop skills by applying programming to various problems. In the past decade, courses have started to move from manually graded assessments to those which can be automated. A typical motivation for the use of automated ...
Comments
Information & Contributors
Information
Published In
January 2023
139 pages
ISBN:9781450399418
DOI:10.1145/3576123
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 January 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ACE '23
ACE '23: Australasian Computing Education Conference
January 30 - February 3, 2023
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 161 of 359 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 178Total Downloads
- Downloads (Last 12 months)110
- Downloads (Last 6 weeks)16
Other Metrics
Citations
Cited By
View all- Soares JBrito MGonçalves C(2023)Codeflex 2.0Internet of Behaviors Implementation in Organizational Contexts10.4018/978-1-6684-9039-6.ch003(40-67)Online publication date: 1-Nov-2023
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format