research-article

Benchmarking Introductory Programming Exams: Some Preliminary Results

Authors:

Peter Klemperer,

Martijn Stegeman,

Daniel ZingaroAuthors Info & Claims

ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

Pages 103 - 111

https://doi.org/10.1145/2960310.2960337

Published: 25 August 2016 Publication History

Abstract

The programming education literature includes many observations that pass rates are low in introductory programming courses, but few or no comparisons of student performance across courses. This paper addresses that shortcoming. Having included a small set of identical questions in the final examinations of a number of introductory programming courses, we illustrate the use of these questions to examine the relative performance of the students both across multiple institutions and within some institutions. We also use the questions to quantify the size and overall difficulty of each exam. We find substantial differences across the courses, and venture some possible explanations of the differences. We conclude by explaining the potential benefits to instructors of using the same questions in their own exams.

References

[1]

J Bennedsen and ME Caspersen (2007). Failure rates in introductory programming. SIGCSE Bulletin, 39:2, 32--36.

Digital Library

[2]

MJ Guzdial and B Ericson (2013). Introduction to computing and programming in Python: a multimedia approach. Pearson Education.

[3]

R Lister (2011). Concrete and other neo-Piagetian forms of reasoning in the novice programmer. 13th Australasian Computing Education Conference (ACE 2011), 9--18.

Digital Library

[4]

R Lister, ES Adams S Fitzgerald, W Fone, J Hamer, M Lindholm, R McCartney, JE Moström, K Sanders, O Seppälä, B Simon, and L Thomas (2004). A multi-national study of reading and tracing skills in novice programmers. SIGCSE Bulletin, 36:4, 119--150.

Digital Library

[5]

R Lister, M Corney, J Curran, D D'Souza, C Fidge, R Gluga, M Hamilton, J Harland, J Hogan, J Kay, T Murphy, M Roggenkamp, J Sheard, Simon, and D Teague (2012). Toward a shared understanding of competency in programming: An invitation to the BABELnot project. 14th Australasian Computing Education Conference (ACE 2012), 53--60.

Digital Library

[6]

M McCracken, V Almstrum, D Diaz, M Guzdial, D Hagan, Y Ben-David Kolikant, C Laxer, L Thomas, I Utting, and T Wilusz (2001). A multi-national, multi-institutional study assessment of programming skills of first-year CS students. SIGCSE Bulletin - Working Group reports: Making inroads to improve computing education, 33:4, 125--140.

Digital Library

[7]

A Petersen, M Craig, and D Zingaro (2011). Reviewing CS1 exam question content. 42nd ACM Technical Symposium on Computer Science Education (SIGCSE'11), 631--636.

Digital Library

[8]

J Sheard, A Carbone, R Lister, B Simon, E Thompson, and JL Whalley (2008). Going SOLO to access novice programmers. 13th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2008), 209--213.

Digital Library

[9]

J Sheard, Simon, A Carbone, D Chinn, T Clear, M Corney, D D'Souza, J Fenwick, J Harland, M-J Laakso, and D Teague (2013). How difficult are exams? A framework for assessing the complexity of introductory programming exams. 15th Australasian Computing Education Conference (ACE 2013), 145--154.

Digital Library

[10]

Simon (2011). Assignment and sequence: why some students can't recognise a simple swap. 11th Koli Calling International Conference on Computing Education Research, 21--30.

Digital Library

[11]

Simon, A Carbone, M de Raadt, R Lister. M Hamilton, and J Sheard (2008). Classifying computing education papers: process and results. Fourth International Computing Education Research Workshop (ICER 2008), 161--171.

Digital Library

[12]

Simon, J Sheard, A Carbone, D D'Souza, J Harland, and M-J Laakso (2012). Can computing academics assess the difficulty of programming examination questions? 11th Koli Calling International Conference on Computing Education Research, 160--163.

Digital Library

[13]

Simon, J Sheard, D D'Souza, P Klemperer, L Porter, J Sorva, M Stegeman, and D Zingaro (2016). Benchmarking introductory programming exams: how and why. 21st ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16).

Digital Library

[14]

Simon, J Sheard, D D'Souza, M Lopez, A Luxton-Reilly, IH Putro, P Robbins, D Teague, and J Whalley (2015). How (not) to write an introductory programming exam. 17th Australasian Computing Education Conference (ACE 2015), 137--146.

[15]

Simon and S Snowdon (2014). Multiple-choice vs free-text code-explaining examination questions. 14th Koli Calling International Conference on Computing Education Research, 91--97.

Digital Library

[16]

B Simon, M Clancy, R McCartney, B Morrison, B Richards, and K Sanders (2010). Making sense of data structures exams. Sixth International Computing Education Research workshop (ICER 2010), 97--105.

Digital Library

[17]

E Soloway (1986). Learning to program = learning to construct mechanisms and explanations. Communications of the ACM, 29:9, 850--858.

Digital Library

[18]

C Watson and FW Li (2014). Failure rates in introductory programming revisited. 19th ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '14), 39--44.

Digital Library

[19]

J Whalley, R Lister, E Thompson, T Clear, P Robbins, PKA Kumar, and C Prasad (2006). An Australasian study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies. Eighth Australasian Computing Education conference (ACE 2006), 243--251.

Digital Library

Cited By

Simon Sheard JLuxton-Reilly ASzabo C(2023)Computing Education Research in AustralasiaPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_17(373-394)Online publication date: 5-Jan-2023
https://doi.org/10.1007/978-3-031-25336-2_17
Wiegand RBucci AKumar AAlbert JGaspar A(2021)Identifying Informatively Easy and Informatively Hard ConceptsACM Transactions on Computing Education10.1145/347796822:1(1-28)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3477968
Barkmin MBrinda T(2020)Analysis of Programming Assessments — Building an Open Repository for Measuring CompetenciesProceedings of the 20th Koli Calling International Conference on Computing Education Research10.1145/3428029.3428039(1-10)Online publication date: 19-Nov-2020
https://dl.acm.org/doi/10.1145/3428029.3428039
Show More Cited By

Index Terms

Benchmarking Introductory Programming Exams: Some Preliminary Results
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

Benchmarking Introductory Programming Exams: How and Why
ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

Ten selected questions have been included in 13 introductory programming exams at seven institutions in five countries. The students' results on these questions, and on the exams as a whole, lead to the development of a benchmark against which the exams ...
Introductory programming: a systematic literature review
ITiCSE 2018 Companion: Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education

As computing becomes a mainstream discipline embedded in the school curriculum and acts as an enabler for an increasing range of academic disciplines in higher education, the literature on introductory programming is growing. Although there have been ...
A Survey of Introductory Programming Courses in Ireland
ITiCSE '19: Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education

Between January and April of 2018, a comprehensive survey of introductory programming courses was undertaken across all sectors of Irish third-level institutions (universities, institutes of technology, and private colleges). The survey instrument was ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

August 2016

310 pages

ISBN:9781450344494

DOI:10.1145/2960310

General Chairs:
Judy Sheard
Monash University, Australia
,
Josh Tenenberg
University of Washington Tacoma, USA
,
Donald Chinn
University of Washington Tacoma, USA
,
Brian Dorn
University of Nebraska at Omaha, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICER '16

Sponsor:

SIGCSE

ICER '16: International Computing Education Research Conference

September 8 - 12, 2016

VIC, Melbourne, Australia

Acceptance Rates

ICER '16 Paper Acceptance Rate 26 of 102 submissions, 25%;

Overall Acceptance Rate 189 of 803 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
580
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)5

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Simon Sheard JLuxton-Reilly ASzabo C(2023)Computing Education Research in AustralasiaPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_17(373-394)Online publication date: 5-Jan-2023
https://doi.org/10.1007/978-3-031-25336-2_17
Wiegand RBucci AKumar AAlbert JGaspar A(2021)Identifying Informatively Easy and Informatively Hard ConceptsACM Transactions on Computing Education10.1145/347796822:1(1-28)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3477968
Barkmin MBrinda T(2020)Analysis of Programming Assessments — Building an Open Repository for Measuring CompetenciesProceedings of the 20th Koli Calling International Conference on Computing Education Research10.1145/3428029.3428039(1-10)Online publication date: 19-Nov-2020
https://dl.acm.org/doi/10.1145/3428029.3428039
Steinhorst PPetersen AVahrenhold JRobins AMoskal AKo AMcCauley RRobins AKo A(2020)Revisiting Self-Efficacy in Introductory ProgrammingProceedings of the 2020 ACM Conference on International Computing Education Research10.1145/3372782.3406281(158-169)Online publication date: 10-Aug-2020
https://dl.acm.org/doi/10.1145/3372782.3406281
Jackova JKanalikova A(2019)Introductory Programming Exams and Their Benchmarking in Slovakia2019 17th International Conference on Emerging eLearning Technologies and Applications (ICETA)10.1109/ICETA48886.2019.9039988(279-284)Online publication date: Nov-2019
https://doi.org/10.1109/ICETA48886.2019.9039988
Austin THorstmann CVu H(2018)Explicit short program practice in a programming languages courseJournal of Computing Sciences in Colleges10.5555/3199572.319958833:4(114-122)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.5555/3199572.3199588
Luxton-Reilly ASimon Albluwi IBecker BGiannakos MKumar AOtt LPaterson JScott MSheard JSzabo CRößling GScharlau B(2018)Introductory programming: a systematic literature reviewProceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education10.1145/3293881.3295779(55-106)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3293881.3295779
Luxton-Reilly ABecker BCao YMcDermott RMirolo CMühling APetersen ASanders KSimon Whalley JSheard JKorhonen A(2018)Developing Assessments to Determine Mastery of Programming FundamentalsProceedings of the 2017 ITiCSE Conference on Working Group Reports10.1145/3174781.3174784(47-69)Online publication date: 30-Jan-2018
https://dl.acm.org/doi/10.1145/3174781.3174784

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents