research-article

Open access

Assisting Teaching Assistants with Automatic Code Corrections

Authors:

Yana Malysheva,

Caitlin KelleherAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 231, Pages 1 - 18

https://doi.org/10.1145/3491102.3501820

Published: 29 April 2022 Publication History

All formats PDF

Abstract

Undergraduate Teaching Assistants(TAs) in Computer Science courses are often the first and only point of contact when a student gets stuck on a programming problem. But these TAs are often relative beginners themselves, both in programming and in teaching. In this paper, we examine the impact of availability of corrected code on TAs’ ability to find, fix, and address bugs in student code. We found that seeing a corrected version of the student code helps TAs debug code 29% faster, and write more accurate and complete student-facing explanations of the bugs (30% more likely to correctly address a given bug). We also observed that TAs do not generally struggle with the conceptual understanding of the underlying material. Rather, their difficulties seem more related to issues with working memory, attention, and overall high cognitive load.

Supplemental Material

MP4 File

Talk Video

Transcript for: Talk Video

MP4 File

Video Preview

Transcript for: Video Preview

References

[1]

Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. 2005. An analysis of patterns of debugging among novice computer science students. In Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education(ITiCSE ’05). Association for Computing Machinery, Caparica, Portugal, 84–88. https://doi.org/10.1145/1067445.1067472

Digital Library

[2]

John R. Anderson and Robin Jeffries. 1985. Novice LISP Errors: Undetected Losses of Information from Working Memory. Human–Computer Interaction 1, 2 (June 1985), 107–131. https://doi.org/10.1207/s15327051hci0102_2 Publisher: Taylor & Francis _eprint: https://doi.org/10.1207/s15327051hci0102_2.

Digital Library

[3]

Roman Bednarik. 2012. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human-Computer Studies 70, 2 (2012), 143–155. Publisher: Elsevier.

Digital Library

[4]

Roman Bednarik and Markku Tukiainen. 2005. Effects of display blurring on the behavior of novices and experts during program debugging. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems. ACM, Portland OR USA, 1204–1207. https://doi.org/10.1145/1056808.1056877

Digital Library

[5]

Maureen Biggers, Tuba Yilmaz, and Monica Sweat. 2009. Using collaborative, modified peer led team learning to improve student success and retention in intro cs. In Proceedings of the 40th ACM technical symposium on Computer science education. 9–13.

Digital Library

[6]

Rebecca Brent, Jason Maners, Dianne Raubenheimer, and Amy Craig. 2007. Preparing undergraduates to teach computer applications to engineering freshmen. In 2007 37th Annual Frontiers In Education Conference-Global Engineering: Knowledge Without Borders, Opportunities Without Passports. IEEE, F1J–19.

[7]

Mark J. Canup and Russell L. Shackelford. 1998. Using software to solve problems in large computing courses. ACM SIGCSE Bulletin 30, 1 (1998), 135–139. Publisher: ACM New York, NY, USA.

Digital Library

[8]

Henian Chen, Patricia Cohen, and Sophie Chen. 2010. How Big is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies. Communications in Statistics - Simulation and Computation 39, 4 (March 2010), 860–864. https://doi.org/10.1080/03610911003650383

[9]

Susan Chinn. 2000. A simple method for converting an odds ratio to effect size for use in meta-analysis. Statistics in Medicine 19, 22 (2000), 3127–3131. https://doi.org/10.1002/1097-0258(20001130)19:22<3127::AID-SIM784>3.0.CO;2-M _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/1097-0258%2820001130%2919%3A22%3C3127%3A%3AAID-SIM784%3E3.0.CO%3B2-M.

[10]

Sammi Chow, Kalina Yacef, Irena Koprinska, and James Curran. 2017. Automated data-driven hints for computer programming students. In Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization. 5–10.

Digital Library

[11]

Adrienne Decker, Phil Ventura, and Christopher Egert. 2006. Through the looking glass: reflections on using undergraduate teaching assistants in CS1. In Proceedings of the 37th SIGCSE technical symposium on Computer science education. 46–50.

Digital Library

[12]

M. Ducasse and A.-M. Emde. 1988. A review of automated debugging systems: knowledge, strategies and techniques. In Proceedings. [1989] 11th International Conference on Software Engineering. 162–171. https://doi.org/10.1109/ICSE.1988.93698

[13]

Ronald Erdei, John A. Springer, and David M. Whittinghill. 2017. An impact comparison of two instructional scaffolding strategies employed in our programming laboratories: Employment of a supplemental teaching assistant versus employment of the pair programming methodology. In 2017 IEEE Frontiers in Education Conference (FIE). 1–6. https://doi.org/10.1109/FIE.2017.8190650

Digital Library

[14]

Joseph Fong, Dawn Leung, and Donny Lai. 2009. A Peer-to-Peer eLearning Supporting System for Computer Programming Debugging System. In Hybrid Learning and Education, David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri Terzopoulos, Doug Tygar, Moshe Y. Vardi, Gerhard Weikum, Fu Lee Wang, Joseph Fong, Liming Zhang, and Victor S. K. Lee (Eds.). Vol. 5685. Springer Berlin Heidelberg, Berlin, Heidelberg, 230–239. https://doi.org/10.1007/978-3-642-03697-2_22 Series Title: Lecture Notes in Computer Science.

Digital Library

[15]

Meg Fryling, MaryAnne Egan, Robin Y. Flatland, Scott Vandenberg, and Sharon Small. 2018. Catch’em Early: Internship and Assistantship CS Mentoring Programs for Underclassmen. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. 658–663.

Digital Library

[16]

Elena L. Glassman, Christopher J. Terman, and Robert C. Miller. 2015. Learner-Sourcing in an Engineering Class at Scale. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale(L@S ’15). Association for Computing Machinery, New York, NY, USA, 363–366. https://doi.org/10.1145/2724660.2728694

Digital Library

[17]

John D. Gould and Paul Drongowski. 1974. An exploratory study of computer program debugging. Human Factors 16, 3 (1974), 258–277. Publisher: SAGE Publications Sage CA: Los Angeles, CA.

[18]

Sebastian Gross, Bassam Mokbel, Benjamin Paaßen, Barbara Hammer, and Niels Pinkwart. 2014. Example-based feedback provision using structured solution spaces. International Journal of Learning Technology 10 9, 3 (2014), 248–280. Publisher: Inderscience Publishers Ltd.

Digital Library

[19]

L. Gugerty and G. Olson. 1986. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’86). Association for Computing Machinery, Boston, Massachusetts, USA, 171–174. https://doi.org/10.1145/22627.22367

Digital Library

[20]

Leo Gugerty and Gary M. Olson. 1986. Comprehension differences in debugging by skilled and novice programmers. In Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers. Ablex Publishing Corp., Washington, D.C., USA, 13–27.

[21]

Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D’Antoni, and Björn Hartmann. 2017. Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale - L@S ’17. ACM Press, Cambridge, Massachusetts, USA, 89–98. https://doi.org/10.1145/3051457.3051467

Digital Library

[22]

Prateek Hejmady and N. Hari Narayanan. 2012. Visual attention patterns during program debugging with an IDE. In Proceedings of the Symposium on Eye Tracking Research and Applications - ETRA ’12. ACM Press, Santa Barbara, California, 197. https://doi.org/10.1145/2168556.2168592

Digital Library

[23]

Wei Jin, Tiffany Barnes, John Stamper, Michael John Eagle, Matthew W. Johnson, and Lorrie Lehmann. 2012. Program representation for automatic hint generation for a data-driven novice programming tutor. In International conference on intelligent tutoring systems. Springer, 304–309.

Digital Library

[24]

Irvin R. Katz and John R. Anderson. 1987. Debugging: An analysis of bug-location strategies. Human-Computer Interaction 3, 4 (1987), 351–399. Publisher: Taylor & Francis.

Digital Library

[25]

Brittany Ann Kos. 2017. The collaborative learning framework: Scaffolding for untrained peer-to-peer collaboration. (2017).

[26]

Timotej Lazar, Aleksander Sadikov, and Ivan Bratko. 2017. Rewrite Rules for Debugging Student Programs in Programming Tutors. IEEE Transactions on Learning Technologies 11, 4 (2017), 429–440.

[27]

Yu-Tzu Lin, Cheng-Chih Wu, Ting-Yun Hou, Yu-Chih Lin, Fang-Ying Yang, and Chia-Hu Chang. 2016. Tracking Students’ Cognitive Processes During Program Debugging—An Eye-Movement Approach. IEEE Transactions on Education 59, 3 (Aug. 2016), 175–186. https://doi.org/10.1109/TE.2015.2487341 Conference Name: IEEE Transactions on Education.

Digital Library

[28]

Yi Liu, Gita PhelpsA, and Fengxia Yan. 2019. Developing a guided peer-assisted learning community for CS students. Journal of Computing Sciences in Colleges 34, 7 (2019), 72–80. Publisher: Consortium for Computing Sciences in Colleges.

Digital Library

[29]

Renee McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: a review of the literature from an educational perspective. Computer Science Education 18, 2 (2008), 67–92. Publisher: Taylor & Francis.

[30]

Ruth McKeever and Kevin McDaid. 2010. How do Range Names Hinder Novice Spreadsheet Debugging Performance?arXiv:1009.2765 [cs] (Sept. 2010). http://arxiv.org/abs/1009.2765 arXiv:1009.2765.

[31]

Mia Minnes, Christine Alvarado, and Leo Porter. 2018. Lightweight Techniques to Support Students in Large Classes. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. ACM, Baltimore Maryland USA, 122–127. https://doi.org/10.1145/3159450.3159601

Digital Library

[32]

Diba Mirza, Phillip T. Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate Teaching Assistants in Computer Science: A Systematic Literature Review. In Proceedings of the 2019 ACM Conference on International Computing Education Research(ICER ’19). Association for Computing Machinery, New York, NY, USA, 31–40. https://doi.org/10.1145/3291279.3339422

Digital Library

[33]

Murthi Nanja and Curtis R. Cook. 1987. An analysis of the on-line debugging process. In Empirical studies of programmers: Second workshop. 172–184.

[34]

Benjamin Paaßen, Barbara Hammer, Thomas William Price, Tiffany Barnes, Sebastian Gross, and Niels Pinkwart. 2018. The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces. arXiv:1708.06564 [cs] (June 2018). http://arxiv.org/abs/1708.06564 arXiv:1708.06564.

[35]

Inna Pivkina. 2016. Peer learning assistants in undergraduate computer science courses. In 2016 IEEE Frontiers in Education Conference (FIE). IEEE, 1–4.

Digital Library

[36]

Thomas Price. 2021. thomaswp/CSEDM2019-Data-Challenge. https://github.com/thomaswp/CSEDM2019-Data-Challenge original-date: 2018-12-30T21:05:59Z.

[37]

Thomas Price, Rui Zhi, and Tiffany Barnes. 2017. Evaluation of a Data-Driven Feedback Algorithm for Open-Ended Programming.International Educational Data Mining Society (2017).

[38]

Thomas W. Price, Yihuan Dong, Rui Zhi, Benjamin Paaßen, Nicholas Lytle, Veronica Cateté, and Tiffany Barnes. 2019. A Comparison of the Quality of Data-driven Programming Hint Generation Algorithms. International Journal of Artificial Intelligence in Education 29, 3(2019), 368–395. Publisher: Springer.

[39]

Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay. 2016. sk_p: a neural program corrector for MOOCs. In Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity. 39–40.

Digital Library

[40]

Kelly Rivers and Kenneth R. Koedinger. 2017. Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Journal of Artificial Intelligence in Education 27, 1 (March 2017), 37–64. https://doi.org/10.1007/s40593-015-0070-z

[41]

T. J. Robertson, Shrinu Prabhakararao, Margaret Burnett, Curtis Cook, Joseph R. Ruthruff, Laura Beckwith, and Amit Phalgune. 2004. Impact of interruption style on end-user debugging. In Proceedings of the SIGCHI conference on Human factors in computing systems. 287–294.

Digital Library

[42]

Pablo Romero, Richard Cox, Benedict du Boulay, and Rudi Lutz. 2002. Visual Attention and Representation Switching During Java Program Debugging: A Study Using the Restricted Focus Viewer. In Diagrammatic Representation and Inference(Lecture Notes in Computer Science), Mary Hegarty, Bernd Meyer, and N. Hari Narayanan (Eds.). Springer, Berlin, Heidelberg, 221–235. https://doi.org/10.1007/3-540-46037-3_23

[43]

Abdallah Tubaishat. 2001. A knowledge base for program debugging. In Proceedings ACS/IEEE International Conference on Computer Systems and Applications. IEEE, 321–327.

Digital Library

[44]

Iris Vessey. 1985. Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies 23, 5 (1985), 459–494. Publisher: Elsevier.

[45]

Arto Vihavainen, Thomas Vikberg, Matti Luukkainen, and Jaakko Kurhila. 2013. Massive increase in eager TAs: Experiences from extreme apprenticeship-based CS1. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 123–128.

Digital Library

[46]

Erin Walker, Nikol Rummel, and Kenneth R. Koedinger. 2009. Integrating collaboration and intelligent tutoring data in the evaluation of a reciprocal peer tutoring environment. Research and Practice in Technology Enhanced Learning 4, 03 (2009), 221–251. Publisher: World Scientific.

[47]

Ke Wang, Benjamin Lin, Bjorn Rettig, Paul Pardi, and Rishabh Singh. 2017. Data-driven feedback generator for online programing courses. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. 257–260.

Digital Library

[48]

E. M. Wilcox, J. W. Atwood, M. M. Burnett, J. J. Cadiz, and C. R. Cook. 1997. Does continuous visual feedback aid debugging in direct-manipulation programming systems?. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. ACM, Atlanta Georgia USA, 258–265. https://doi.org/10.1145/258549.258721

Digital Library

[49]

Kurtis Zimmerman and Chandan R. Rupakheti. 2015. An automated framework for recommending program elements to novices (n). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 283–288.

Digital Library

Cited By

Ganguli IBhowmick RBiswas SSil J(2024)Intelligent Program Correction and Evaluation SystemInternational Journal on Artificial Intelligence Tools10.1142/S021821302350071933:02Online publication date: 30-Mar-2024
https://doi.org/10.1142/S0218213023500719
Yang DHe JMao XLi TLei YYi XWu J(2024)Strider: Signal Value Transition-Guided Defect Repair for HDL Programming AssignmentsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.334175043:5(1594-1607)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1109/TCAD.2023.3341750
Zegura CShapiro BMacDonald RBorenstein JZegura E(2023)“Moment to Moment”: A Situated View of Teaching Ethics from the Perspective of Computing Ethics Teaching AssistantsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581572(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581572
Show More Cited By

Recommendations

Training computer science teaching assistants: a seminar for new TAs
SIGCSE '95: Proceedings of the twenty-sixth SIGCSE technical symposium on Computer science education

We describe a one-quarter seminar required of first-time teaching assistants in computer science. The seminar acclimates TAs to teaching undergraduates in the university, to departmental policies and to special problems and issues in teaching computer ...
Mentoring Student Teaching Assistants for Computer Science (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

Institutions small and large often use student teaching assistants (TAs) to provide office hours, tutor students, and grade student work, and the quality of a course's TAs can greatly affect the quality of the course. In this BOF we will discuss the ...
Automatic prediction of bug fixing effort measured by code churn size
SoftwareMining 2016: Proceedings of the 5th International Workshop on Software Mining

During software maintenance, developers often receive many bug reports. Project managers often need to manage limited resources to resolve the many bugs that a project receives. To help project managers perform their job, past studies have proposed ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
1,026
Total Downloads

Downloads (Last 12 months)379
Downloads (Last 6 weeks)47

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ganguli IBhowmick RBiswas SSil J(2024)Intelligent Program Correction and Evaluation SystemInternational Journal on Artificial Intelligence Tools10.1142/S021821302350071933:02Online publication date: 30-Mar-2024
https://doi.org/10.1142/S0218213023500719
Yang DHe JMao XLi TLei YYi XWu J(2024)Strider: Signal Value Transition-Guided Defect Repair for HDL Programming AssignmentsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.334175043:5(1594-1607)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1109/TCAD.2023.3341750
Zegura CShapiro BMacDonald RBorenstein JZegura E(2023)“Moment to Moment”: A Situated View of Teaching Ethics from the Perspective of Computing Ethics Teaching AssistantsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581572(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581572
Carayon AMartinie CPalanque P(2023)Task Models as a Mean to Identify and Justify Automations in Development Tasks2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)10.1109/MODELS-C59198.2023.00122(757-764)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/MODELS-C59198.2023.00122
Chatterjee PKalita PLahiri SMuduli SSingh VTakhar GRoy SBissyandé TKlein JBird CSarro F(2023)An Integrated Program Analysis Framework for Graduate Courses in Programming Languages and Software EngineeringProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00101(598-610)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00101
Kuo YChang P(2023)Flipped classroom combined with WPACQ learning mode on student learning effect - exemplified by program design coursesEducation and Information Technologies10.1007/s10639-023-12384-229:10(12621-12651)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1007/s10639-023-12384-2
Kuo JLin HWang PNie Z(2022)A Feedback System Supporting Students Approaching a High-Level Programming CourseApplied Sciences10.3390/app1214706412:14(7064)Online publication date: 13-Jul-2022
https://doi.org/10.3390/app12147064
Malysheva Y(2022)Helping TAs Help Students2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC53370.2022.9833118(1-3)Online publication date: 12-Sep-2022
https://doi.org/10.1109/VL/HCC53370.2022.9833118

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten