research-article

Open access

What Fails Once, Fails Again: Common Repeated Errors in Introductory Programming Automated Assessments

Authors:

Brett A. BeckerAuthors Info & Claims

SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1

Pages 955 - 961

https://doi.org/10.1145/3478431.3499419

Published: 22 February 2022 Publication History

Abstract

In this paper, we analyze 32,000 Java programming assessments submitted to the CodeRunner platform by introductory programming students. We identify common sequences of errors and link these patterns with observations of code that compiles but still contains logical errors. We establish reoccurring errors and common error pathways through a Markov chain analysis of attempts and use Association Rule Mining to link common error patterns with logical errors that occur later in the attempts. As a result, we provide a number of suggestions for instructors of introductory Java courses regarding how to potentially navigate these issues in their teaching practice and discuss possible pedagogical interventions on the basis of our observations.

References

[1]

A. Altadmri and N. Brown. 2015. 37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, Missouri, USA) (SIGCSE '15). ACM, NY, NY, USA, 522--527. https://doi.org/10.1145/2676723.2677258

Digital Library

[2]

P. Ardimento, M. Bernardi, and M. Cimitile. 2020. Software Analytics to Support Students in Object-Oriented Programming Tasks: An Empirical Study. IEEE Access, Vol. 8 (2020), 132171--132187.

[3]

M. Barkmin and T. Brinda. 2020. Analysis of Programming Assessments - Building an Open Repository for Measuring Competencies. In Koli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '20). ACM, NY, NY, USA, Article 31, bibinfonumpages10 pages. https://doi.org/10.1145/3428029.3428039

Digital Library

[4]

B. Becker. 2015. An Exploration of the Effects of Enhanced Compiler Error Messages for Computer Programming Novices . Master's thesis. Dublin Institute of Technology.

[5]

B. Becker, P. Denny, R. Pettit, et almbox. 2019. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (Aberdeen, Scotland Uk) (ITiCSE-WGR '19). ACM, NY, NY, USA, 177--210. https://doi.org/10.1145/3344429.3372508

Digital Library

[6]

B. Becker, C. Murray, T. Tao, et almbox. 2018. Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Baltimore, Maryland, USA) (SIGCSE '18). ACM, NY, NY, USA, 634--639. https://doi.org/10.1145/3159450.3159453

Digital Library

[7]

Brett A. Becker. 2021. What Does Saying That 'programming is Hard' Really Say, and about Whom? Commun. ACM, Vol. 64, 8 (jul 2021), 27--29. https://doi.org/10.1145/3469115

Digital Library

[8]

Brett A Becker and Catherine Mooney. 2016. Categorizing Compiler Error Messages with Principal Component Analysis. In 12th China-Europe International Symposium on Software Engineering Education (CEISEE 2016), Shenyang, China, 28--29 May 2016 .

[9]

J. Bennedsen and M. Caspersen. 2007. Failure Rates in Introductory Programming. AcM SIGcSE Bulletin, Vol. 39, 2 (2007), 32--36.

Digital Library

[10]

N. Brown and A. Altadmri. 2014. Investigating Novice Programming Mistakes: Educator Beliefs vs. Student Data. In Proceedings of the Tenth Annual Conference on International Computing Education Research (Glasgow, Scotland, United Kingdom) (ICER '14). ACM, NY, NY, USA, 43--50. https://doi.org/10.1145/2632320.2632343

Digital Library

[11]

N. Brown, M. Kölling, D. McCall, and I. Utting. 2014. Blackbox: A Large Scale Repository of Novice Programmers' Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (Atlanta, Georgia, USA) (SIGCSE '14). ACM, NY, NY, USA, 223--228. https://doi.org/10.1145/2538862.2538924

Digital Library

[12]

J. Campbell, A. Hindle, and J. Amaral. 2014. Syntax Errors Just Aren't Natural: Improving Error Reporting with Language Models. In Proceedings of the 11th Working Conference on Mining Software Repositories (Hyderabad, India) (MSR 2014). ACM, NY, NY, USA, 252--261. https://doi.org/10.1145/2597073.2597102

Digital Library

[13]

R. Crosby. 2021. Student Perceptions of Assessment and Feedback - Are They Valid? In Computing Education Practice 2021 . ACM, NY, NY, USA, 5--8. https://doi.org/10.1145/3437914.3437970

Digital Library

[14]

P. Denny, A. Luxton-Reilly, and E. Tempero. 2012. All Syntax Errors Are Not Equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (Haifa, Israel) (ITiCSE '12). ACM, NY, NY, USA, 75--80. https://doi.org/10.1145/2325296.2325318

Digital Library

[15]

P. Denny, J. Prather, B. Becker, et almbox. 2019. A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '19). ACM, NY, NY, USA, Article 11, bibinfonumpages10 pages. https://doi.org/10.1145/3364510.3366170

Digital Library

[16]

Paul Denny, James Prather, Brett A. Becker, Catherine Mooney, John Homer, Zachary C Albrecht, and Garrett B. Powell. 2021. On Designing Programming Error Messages for Novices: Readability and Its Constituent Factors. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, NY, NY, USA, Article 55, bibinfonumpages15 pages. https://doi.org/10.1145/3411764.3445696

Digital Library

[17]

T. Dy and M. Rodrigo. 2010. A Detector for Non-Literal Java Errors. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '10). ACM, NY, NY, USA, 118--122. https://doi.org/10.1145/1930464.1930485

Digital Library

[18]

F. Fonseca and L. Spence. 2014. The Karate Kid Method of Problem Based Learning .Springer International Publishing, Cham, 9--17. https://doi.org/10.1007/978--3--319-03656--4_2

[19]

S. Garner, P. Haden, and A. Robins. 2005. My Program is Correct but It Doesn't Run: A Preliminary Investigation of Novice Programmers' Problems. In Proceedings of the 7th Australasian Conference on Computing Education - Volume 42 (Newcastle, New South Wales, Australia) (ACE '05). Australian Computer Society, Inc., AUS, 173--180.

[20]

R. Gupta, S. Pal, A. Kanade, and S. Shevade. 2017. DeepFix: Fixing Common C Language Errors by Deep Learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (San Francisco, California, USA) (AAAI'17). AAAI Press, Palo Alto, California, 1345--1351.

[21]

E. Hegarty-Kelly and A. Mooney. 2021. Analysis of an Automatic Grading System within First Year Computer Science Programming Modules. In Computing Education Practice 2021 (Durham, United Kingdom) (CEP '21). ACM, NY, NY, USA, 17--20. https://doi.org/10.1145/3437914.3437973

Digital Library

[22]

Jochen Hipp, Ulrich Güntzer, and Gholamreza Nakhaeizadeh. 2000. Algorithms for Association Rule Mining -- A General Survey and Comparison. ACM sigkdd explorations newsletter, Vol. 2, 1 (2000), 58--64.

[23]

M. Hristova, A. Misra, M. Rutter, and R. Mercuri. 2003. Identifying and Correcting Java Programming Errors for Introductory Computer Science Students. ACM SIGCSE Bulletin, Vol. 35, 1 (2003), 153--156.

Digital Library

[24]

J. Jackson, M. Cobb, and C. Carver. 2005. Identifying Top Java Errors for Novice Programmers. In Proceedings Frontiers in Education 35th Annual Conference. IEEE, NY, T4C--T4C. https://doi.org/10.1109/FIE.2005.1611967

[25]

N. Kandru. 2018. Intelligent Goal-Oriented Feedback for Java Programming Assignments . Ph.D. Dissertation. Virginia Tech.

[26]

E. Lahtinen, K. Ala-Mutka, and H. J"arvinen. 2005. A Study of the Difficulties of Novice Programmers. ACM SIGCSE Bulletin, Vol. 37, 3 (2005), 14--18.

Digital Library

[27]

J. Liebenberg, M. Huisman, and E. Mentz. 2014. Knowledge and Skills Requirements for Software Developer Students. International Journal of Social, Behavioral, Educational, Economic, Business and Industrial Engineering, Vol. 8, 8 (2014), 2604--2609.

[28]

R. Lobb and J. Harlow. 2016. Coderunner: A Tool for Assessing Computer Programming Skills. ACM Inroads, Vol. 7, 1 (2016), 47--51.

Digital Library

[29]

D. Loksa, B. Xie, H. Kwik, and Amy J. Ko. 2020. Investigating Novices' In Situ Reflections on Their Programming Process. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE '20). ACM, NY, NY, USA, 149--155. https://doi.org/10.1145/3328778.3366846

Digital Library

[30]

A. Luxton-Reilly. 2016. Learning to Program is Easy. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (Arequipa, Peru) (ITiCSE '16). ACM, NY, NY, USA, 284--289. https://doi.org/10.1145/2899415.2899432

Digital Library

[31]

A. Luxton-Reilly, V. Ajanovski, E. Fouh, et almbox. 2019. Pass Rates in Introductory Programming and in Other STEM Disciplines. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education . 53--71.

[32]

Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). ACM, NY, NY, USA, 55--106. https://doi.org/10.1145/3293881.3295779

Digital Library

[33]

D. McCall and M. Kölling. 2014. Meaningful Categorisation of Novice Programmer Errors. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings. IEEE, NY, 1--8. https://doi.org/10.1109/FIE.2014.7044420

[34]

M. Pfadenhauer. 2013. Competence -- More than Just a Buzzword and a Provocative Term? SensePublishers, Rotterdam, 81--90. https://doi.org/10.1007/978--94--6091--867--4_6

[35]

D. Pritchard. 2015. Frequency Distribution of Error Messages. In Proceedings of the 6th Workshop on Evaluation and Usability of Programming Languages and Tools (Pittsburgh, PA, USA) (PLATEAU 2015). ACM, NY, NY, USA, 1--8. https://doi.org/10.1145/2846680.2846681

Digital Library

[36]

A. Robins, J. Rountree, and N. Rountree. 2003. Learning and Teaching Programming: A Review and Discussion. Computer science education, Vol. 13, 2 (2003), 137--172.

[37]

Ma. Mercedes T. Rodrigo, Ryan S. Baker, Matthew C. Jadud, et almbox. 2009. Affective and Behavioral Predictors of Novice Programmer Achievement. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (Paris, France) (ITiCSE '09). ACM, NY, NY, USA, 156--160. https://doi.org/10.1145/1562877.1562929

Digital Library

[38]

C. Rogerson and E. Scott. 2010. The fear factor: How it Affects Students Learning to Program in a tertiary Environment. Journal of Information Technology Education: Research, Vol. 9, 1 (2010), 147--171.

[39]

B. Sahil and R. Singh. 2016. Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks. CoRR, Vol. abs/1603.06129 (2016), 1--11. arxiv: 1603.06129 http://arxiv.org/abs/1603.06129

[40]

E. Santos, J. Campbell, D. Patel, et almbox. 2018. Syntax and Sensibility: Using Language Models to Detect and Correct Syntax Errors. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, NY, 311--322. https://doi.org/10.1109/SANER.2018.8330219

[41]

S. Smith, N. Zemljic, and A. Petersen. 2015. Modern Goto: Novice Programmer Usage of Non-Standard Control Flow. In Proceedings of the 15th Koli Calling Conference on Computing Education Research (Koli, Finland) (Koli Calling '15). ACM, NY, NY, USA, 171--172. https://doi.org/10.1145/2828959.2828980

Digital Library

[42]

A. Soares, N. Martin, and F. Fonseca. 2015. Teaching Introductory Programming With Game Design and Problem-based Larning. Issues in Information Systems, Vol. 16, 3 (2015), 128--137.

[43]

E. Tabanao, M. Rodrigo, and M. Jadud. 2008. Identifying at-risk novice java programmers through the analysis of online protocols. In Philippine Computing Science Congress. De La Salle University Publishing House, Manilla, 1--8.

[44]

R. Takhar and V. Aggarwal. 2019. Grading Uncompilable Programs. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33 (Jul. 2019), 9389--9396. https://doi.org/10.1609/aaai.v33i01.33019389

Digital Library

[45]

L. Winslow. 1996. Programming Pedagogy -- A Psychological Overview. ACM SIGCSE Bulletin, Vol. 28, 3 (1996), 17--22.

Digital Library

Cited By

Dagklis ESatratzemi MKoloniari GKarakasidis A(2024)Programming Errors and Academic Performance in an Introductory Data Structures Course: A Per Gender AnalysisTowards a Hybrid, Flexible and Socially Engaged Higher Education10.1007/978-3-031-53382-2_6(57-68)Online publication date: 26-Feb-2024
https://doi.org/10.1007/978-3-031-53382-2_6
Scott SHellas ALeinonen JEdwards JHaiduc SSchneider J(2023)Factors Affecting Compilable State at Each Keystroke in CS1Proceedings of the 45th International Conference on Software Engineering: Software Engineering Education and Training10.1109/ICSE-SEET58685.2023.00036(314-323)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-SEET58685.2023.00036
Becker BBradley SMaguire JBlack MCrick TSaqr MSentance SQuille K(2023)Computing Education Research in the UK & IrelandPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_19(421-479)Online publication date: 5-Jan-2023
https://doi.org/10.1007/978-3-031-25336-2_19
Show More Cited By

Index Terms

What Fails Once, Fails Again: Common Repeated Errors in Introductory Programming Automated Assessments
1. Applied computing
  1. Education
    1. Computer-assisted instruction
    2. Learning management systems
2. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Student assessment

Recommendations

Teaching Programming Error Message Understanding
CompEd 2023: Working Group Reports on 2023 ACM Conference on Global Computing Education

About a decade ago there was a sharp increase in the number of research publications focusing on programming error messages. The majority of this work focused on improving or "enhancing'' compiler error messages to make them easier for humans, especially ...
Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement
CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 1

Programming error messages (PEMs) are notoriously difficult for novice programmers to utilise. Many efforts have been made to enhance PEMs such that they are reworded to explain problems in terms that novices can understand. However, the effectiveness of ...
Using Large Language Models to Enhance Programming Error Messages
SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the messages are typically intended for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1

February 2022

1049 pages

ISBN:9781450390705

DOI:10.1145/3478431

General Chairs:
Larry Merkle
Air Force Institute of Technology, USA
,
Maureen Doyle
Northern Kentucky University, USA
,
Program Chairs:
Judithe Sheard
Monash University, Australia
,
Leen-Kiat Soh
University of Nebraska-Lincoln, USA
,
Brian Dorn
University of Nebraska at Omaha, USA

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2022

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCSE 2022

Sponsor:

SIGCSE

SIGCSE 2022: The 53rd ACM Technical Symposium on Computer Science Education

March 3 - 5, 2022

RI, Providence, USA

Acceptance Rates

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE Virtual 2024

Sponsor:
sigcse

1st ACM Virtual Global Computing Education Conference

December 5 - 8, 2024

Virtual Event , NC , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
481
Total Downloads

Downloads (Last 12 months)158
Downloads (Last 6 weeks)30

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dagklis ESatratzemi MKoloniari GKarakasidis A(2024)Programming Errors and Academic Performance in an Introductory Data Structures Course: A Per Gender AnalysisTowards a Hybrid, Flexible and Socially Engaged Higher Education10.1007/978-3-031-53382-2_6(57-68)Online publication date: 26-Feb-2024
https://doi.org/10.1007/978-3-031-53382-2_6
Scott SHellas ALeinonen JEdwards JHaiduc SSchneider J(2023)Factors Affecting Compilable State at Each Keystroke in CS1Proceedings of the 45th International Conference on Software Engineering: Software Engineering Education and Training10.1109/ICSE-SEET58685.2023.00036(314-323)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-SEET58685.2023.00036
Becker BBradley SMaguire JBlack MCrick TSaqr MSentance SQuille K(2023)Computing Education Research in the UK & IrelandPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_19(421-479)Online publication date: 5-Jan-2023
https://doi.org/10.1007/978-3-031-25336-2_19
Sim TLau S(2022)Review on Challenges and Solutions in Novice Programming Education2022 IEEE International Conference on Computing (ICOCO)10.1109/ICOCO56118.2022.10031657(55-61)Online publication date: 14-Nov-2022
https://doi.org/10.1109/ICOCO56118.2022.10031657

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents