research-article

Studying the Impact of Auto-Graders Giving Immediate Feedback in Programming Assignments

Author:

Joydeep MitraAuthors Info & Claims

SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

Pages 388 - 394

https://doi.org/10.1145/3545945.3569726

Published: 03 March 2023 Publication History

Abstract

Immediate feedback from auto-graders positively impacts students' grades and self-efficacy in introductory programming courses. However, recent research has observed that students are not likely to develop testing skills since they over-rely on the feedback from the auto-grader. Therefore, in this paper, we designed and conducted an empirical investigation to study the impact of using immediate feedback on students' ability to write correct programs and test them. The results indicate that while students use immediate feedback from an auto-grader, it does not dissuade them from attaining independent testing skills. Moreover, the feedback helps students, especially underrepresented groups (e.g., women), learn more effectively and gain confidence.

References

[1]

Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education, Vol. 15, 2 (2005), 83--102.

[2]

Elisa Baniassad, Lucas Zamprogno, Braxton Hall, and Reid Holmes. 2021. Stop the (autograder) insanity: Regression penalties to deter autograder overreliance. In Proceedings of the 52nd ACM technical symposium on computer science education. 1062--1068.

Digital Library

[3]

Luciana Benotti, Federico Aloi, Franco Bulgarelli, and Marcos J Gomez. 2018. The effect of a web-based coding tool with automatic feedback on students' performance and perceptions. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. 2--7.

Digital Library

[4]

Kevin Buffardi and Stephen H. Edwards. 2015. Reconsidering Automated Feedback: A Test-Driven Approach. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, Missouri, USA) (SIGCSE '15). Association for Computing Machinery, New York, NY, USA, 416--420. https://doi.org/10.1145/2676723.2677313

Digital Library

[5]

Yoonsik Cheon and Gary T Leavens. 2002. A simple and practical approach to unit testing: The JML and JUnit way. In European Conference on Object-Oriented Programming. Springer, 231--255.

[6]

Lucas Cordova, Jeffrey Carver, Noah Gershmel, and Gursimran Walia. 2021. A Comparison of Inquiry-Based Conceptual Feedback vs. Traditional Detailed Feedback Mechanisms in Software Testing Education: An Empirical Investigation. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE '21). Association for Computing Machinery, New York, NY, USA, 87--93. https://doi.org/10.1145/3408877.3432417

Digital Library

[7]

Christopher Douce, David Livingstone, and James Orwell. 2005. Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing (JERIC), Vol. 5, 3 (2005), 4-es.

Digital Library

[8]

Stephen H. Edwards. 2008. Web-CAT. https://web-cat.cs.vt.edu. Accessed: 17-Jun-2022.

[9]

Stephen H. Edwards and Manuel A. Perez-Quinones. 2008. Web-CAT: Automatically Grading Programming Assignments. SIGCSE Bull., Vol. 40, 3 (jun 2008), 328. https://doi.org/10.1145/1597849.1384371

Digital Library

[10]

GitHub. 2021. GitHub Classroom. https://classroom.github.com/. Accessed: 17-Jun-2022.

[11]

Spacco J. 2013. Marmoset. https://marmoset.cs.umd.edu/. Accessed: 17-Jun-2022.

[12]

David Jackson and Michelle Usher. 1997. Grading Student Programs Using ASSYST. In Proceedings of the Twenty-Eighth SIGCSE Technical Symposium on Computer Science Education (San Jose, California, USA) (SIGCSE '97). Association for Computing Machinery, New York, NY, USA, 335--339. https://doi.org/10.1145/268084.268210

Digital Library

[13]

Yue Jia and Mark Harman. 2010. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, Vol. 37, 5 (2010), 649--678.

[14]

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE), Vol. 19, 1 (2018), 1--43.

Digital Library

[15]

Zachary Kurmas. 2017. MIPSUnit: A Unit Testing Framework for MIPS Assembly. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (Seattle, Washington, USA) (SIGCSE '17). Association for Computing Machinery, New York, NY, USA, 351--355. https://doi.org/10.1145/3017680.3017747

Digital Library

[16]

Allison Scott, Alexis Martin, Frieda McAlear, and Sonia Koshy. 2017. Broadening participation in computing: examining experiences of girls of color. In Proceedings of the 2017 ACM conference on innovation and technology in computer science education. 252--256.

Digital Library

[17]

Roli Varma. 2006. Making computer science minority-friendly. Commun. ACM, Vol. 49, 2 (2006), 129--134.

Digital Library

[18]

Kenneth Vollmar and Pete Sanderson. 2006. MARS: An Education-Oriented MIPS Assembly Language Simulator. In Proceedings of the 37th SIGCSE Technical Symposium on Computer Science Education (Houston, Texas, USA) (SIGCSE '06). Association for Computing Machinery, New York, NY, USA, 239--243. https://doi.org/10.1145/1121341.1121415

Digital Library

Cited By

Cunha AMacedo NCampos JMargolis ISousa ERoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Assessing the impact of hints in learning formal specificationProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640050(151-161)Online publication date: 24-May-2024
https://doi.org/10.1145/3639474.3640050
Sherif EEverson JKivuva FKirdani-Ryan MKo A(2024)Exploring the Impact of Assessment Policies on Marginalized Students' Experiences in Post-Secondary Programming CoursesProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671100(233-245)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671100
Railing B(2023)CADSS: Computer Architecture Design Simulator for StudentsProceedings of the Workshop on Computer Architecture Education10.1145/3605507.3610626(34-40)Online publication date: 17-Jun-2023
https://dl.acm.org/doi/10.1145/3605507.3610626
Show More Cited By

Index Terms

Studying the Impact of Auto-Graders Giving Immediate Feedback in Programming Assignments
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Student assessment

Recommendations

Automatic assessment and immediate feedback in first grade mathematics
Koli Calling '14: Proceedings of the 14th Koli Calling International Conference on Computing Education Research

In this paper we present a study where automatic assessment and immediate feedback was utilized to support the learning of mathematical concepts and automatization of basic arithmetic calculations. Two first grade classes from a Finnish elementary ...
The impact of critical feedback choice on students' revision, performance, learning, and memory

This article examines empirically the impact of students' critical feedback choices on their memory for feedback. It also examines the effect of choosing versus receiving feedback on learning outcomes. First, a correlational study was designed to ...
Automated Grading and Feedback of Programming Assignments
ITiCSE '22: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2

Over the last few years, Computer Science class sizes have increased, resulting in a higher grading workload. To manage this workload, universities often use multiple graders to deliver the grades and associated feedback quickly. While using multiple ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

March 2023

1481 pages

ISBN:9781450394314

DOI:10.1145/3545945

General Chairs:
Maureen Doyle
Northern Kentucky University, USA
,
Ben Stephenson
University of Calgary, Canada
,
Program Chairs:
Brian Dorn
University of Nebraska at Omaha, USA
,
Leen-Kiat Soh
University of Nebraska-Lincoln, USA
,
Lina Battestilli
North Carolina State University, USA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 March 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCSE 2023

Sponsor:

SIGCSE

SIGCSE 2023: The 54th ACM Technical Symposium on Computer Science Education

March 15 - 18, 2023

Toronto ON, Canada

Acceptance Rates

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE Virtual 2024

Sponsor:
sigcse

1st ACM Virtual Global Computing Education Conference

December 5 - 8, 2024

Virtual Event , NC , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
297
Total Downloads

Downloads (Last 12 months)146
Downloads (Last 6 weeks)13

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cunha AMacedo NCampos JMargolis ISousa ERoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Assessing the impact of hints in learning formal specificationProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640050(151-161)Online publication date: 24-May-2024
https://doi.org/10.1145/3639474.3640050
Sherif EEverson JKivuva FKirdani-Ryan MKo A(2024)Exploring the Impact of Assessment Policies on Marginalized Students' Experiences in Post-Secondary Programming CoursesProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671100(233-245)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671100
Railing B(2023)CADSS: Computer Architecture Design Simulator for StudentsProceedings of the Workshop on Computer Architecture Education10.1145/3605507.3610626(34-40)Online publication date: 17-Jun-2023
https://dl.acm.org/doi/10.1145/3605507.3610626
Benjamin MBrown LSticklen JUreel LJarvie-Eggart‡ M(2023)Engaging Novice Programmers: A Literature Review of the Effect of Code Critiquers on Programming Self-efficacy2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10342975(1-9)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10342975

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents