Evaluating Micro Parsons Problems as Exam Questions
Abstract
Parsons problems are a type of programming activity that present learners with blocks of existing code and requiring them to arrange those blocks to form a program rather than write the code from scratch. Micro Parsons problems extend this concept by having students assemble segments of code to form a single line of code rather than an entire program. Recent investigations into micro Parsons problems have primarily focused on supporting learners leaving open the question of micro Parsons efficacy as an exam item and how students perceive it when preparing for exams.
To fill this gap, we included a variety of micro Parsons problems on four exams in an introductory programming course taught in Python. We use Item Response Theory to investigate the difficulty of the micro Parsons problems as well as the ability of the questions to differentiate between high and low ability students. We then compare these results to results for related questions where students are asked to write a single line of code from scratch. Finally, we conduct a thematic analysis of the survey responses to investigate how students' perceptions of micro Parsons both when practicing for exams and as they appear on exams.
References
[1]
Frank B Baker. 2001. The basics of item response theory. ERIC.
[2]
Mark A Barton and Frederic M Lord. 1981. An upper asymptote for the three-parameter logistic item-response model. ETS Research Report Series, Vol. 1981, 1 (1981), i--8.
[3]
Jeff Bender, Bingpu Zhao, Alex Dziena, and Gail Kaiser. 2023. Integrating Parsons puzzles within Scratch enables efficient computational thinking learning. Research and Practice in Technology Enhanced Learning, Vol. 18 (2023), 022--022.
[4]
Allan Birnbaum. 1968. Some latent trait models and their use in inferring an examinee's ability. Statistical theories of mental test scores (1968).
[5]
André F De Champlain. 2010. A primer on classical test theory and item response theory for assessments in medical education. Medical education, Vol. 44, 1 (2010), 109--117.
[6]
Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the Fourth International Workshop on Computing Education Research (Sydney, Australia) (ICER '08). Association for Computing Machinery, New York, NY, USA, 113--124. https://doi.org/10.1145/1404520.1404532
[7]
Robert L Ebel and David A Frisbie. 1972. Essentials of educational measurement.
[8]
Barbara J. Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S. Miller, Briana B. Morrison, Janice L. Pearce, and Susan H. Rodger. 2022. Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs. In Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR '22). Association for Computing Machinery, New York, NY, USA, 191--234. https://doi.org/10.1145/3571785.3574127
[9]
Barbara J. Ericson, James D. Foley, and Jochen Rick. 2018. Evaluating the Efficiency and Effectiveness of Adaptive Parsons Problems. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER '18). Association for Computing Machinery, New York, NY, USA, 60--68. https://doi.org/10.1145/3230977.3231000
[10]
Barbara J. Ericson, Mark J. Guzdial, and Briana B. Morrison. 2015. Analysis of Interactive Features Designed to Enhance Learning in an Ebook. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (Omaha, Nebraska, USA) (ICER '15). Association for Computing Machinery, New York, NY, USA, 169--178. https://doi.org/10.1145/2787622.2787731
[11]
Barbara J. Ericson, Janice L. Pearce, Susan H. Rodger, Andrew Csizmadia, Rita Garcia, Francisco J. Gutierrez, Konstantinos Liaskos, Aadarsh Padiyath, Michael James Scott, David H. Smith, Jayakrishnan M. Warriem, and Angela Zavaleta Bernuy. 2023. Multi-Institutional Multi-National Studies of Parsons Problems. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education. Association for Computing Machinery, New York, NY, USA, 57--107. https://doi.org/10.1145/3623762.3633498
[12]
Geela Venise Firmalo Fabic, Antonija Mitrovic, and Kourosh Neshatian. 2019. Evaluation of Parsons problems with menu-based self-explanation prompts in a mobile python tutor. International Journal of Artificial Intelligence in Education, Vol. 29 (2019), 507--535.
[13]
Carl C. Haynes and Barbara J. Ericson. 2021. Problem-Solving Efficiency and Cognitive Load for Adaptive Parsons Problems vs. Writing the Equivalent Code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 60, bibinfonumpages15 pages. https://doi.org/10.1145/3411764.3445292
[14]
Slava Kalyuga. 2009. The expertise reversal effect. In Managing cognitive load in adaptive multimedia learning. IGI Global, 58--80.
[15]
Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.
[16]
Amruth N. Kumar. 2017. The Effect of Providing Motivational Support in Parsons Puzzle Tutors. In Artificial Intelligence in Education, Elisabeth André, Ryan Baker, Xiangen Hu, Ma. Mercedes T. Rodrigo, and Benedict du Boulay (Eds.). Springer International Publishing, Cham, 528--531.
[17]
Brooke Morin and Krista M Kecskemety. 2021. Collaborative Parsons Problems in a Remote-learning First-year Engineering Classroom. In 2021 ASEE Virtual Annual Conference Content Access.
[18]
Brooke C Morin, Krista M Kecskemety, Kathleen A Harper, and Paul Alan Clingan. 2020. Work in Progress: Parsons Problems as a Tool in the First-Year Engineering Classroom. In 2020 ASEE Virtual Annual Conference Content Access.
[19]
Briana B. Morrison, Lauren E. Margulieux, Barbara Ericson, and Mark Guzdial. 2016. Subgoals Help Students Solve Parsons Problems. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (Memphis, Tennessee, USA) (SIGCSE '16). Association for Computing Machinery, New York, NY, USA, 42--47. https://doi.org/10.1145/2839509.2844617
[20]
Solomon Sunday Oyelere, Friday Joseph Agbo, Ismaila Temitayo Sanusi, Abdullahi Abubakar Yunusa, and Kissinger Sunday. 2019. Impact of puzzle-based learning technique for programming education in Nigeria context. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Vol. 2161. IEEE, 239--241.
[21]
Dale Parsons and Patricia Haden. 2006. Parson's programming puzzles: a fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 157--163.
[22]
Seth Poulsen, Hongxuan Chen, Yael Gertner, Benjamin Cosman, Matthew West, and Geoffrey L Herman. 2023. Measuring the Impact of Distractors on Student Learning Gains while Using Proof Blocks. arXiv preprint arXiv:2311.00792 (2023).
[23]
Seth Poulsen, Mahesh Viswanathan, Geoffrey L Herman, and Matthew West. 2022. Evaluating proof blocks problems as exam questions. ACM Inroads, Vol. 13, 1 (2022), 41--51.
[24]
David H. Smith, Max Fowler, and Craig Zilles. 2023 a. Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 417--423. https://doi.org/10.1145/3587102.3588819
[25]
David H. Smith and Craig Zilles. 2023. Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 924--930. https://doi.org/10.1145/3545945.3569801
[26]
David H. Smith, IV, Seth Poulsen, Max Fowler, and Craig Zilles. 2023 b. Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments. In Proceedings of the ACM Conference on Global Computing Education Vol 1 (CompEd 2023). Association for Computing Machinery, New York, NY, USA, 154--160. https://doi.org/10.1145/3576882.3617927
[27]
David H. Smith IV. 2023. Useful Distractions? Investigating the Utility of Distractors in Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER '23). Association for Computing Machinery, New York, NY, USA, 62--63. https://doi.org/10.1145/3568812.3603463
[28]
Matthew West, Geoffrey L Herman, and Craig Zilles. 2015. Prairielearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition. 26--1238.
[29]
Zihan Wu. 2023. Investigating the Effectiveness of Variations of Micro Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER '23). Association for Computing Machinery, New York, NY, USA, 120--122. https://doi.org/10.1145/3568812.3603447
[30]
Zihan Wu, Barbara Ericson, and Christopher Brooks. 2021. Regex Parsons: Using Horizontal Parsons Problems to Scaffold Learning Regex. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research (Joensuu, Finland) (Koli Calling '21). Association for Computing Machinery, New York, NY, USA, Article 31, bibinfonumpages3 pages. https://doi.org/10.1145/3488042.3489968
[31]
Zihan Wu, Barbara J. Ericson, and Christopher Brooks. 2023. Using Micro Parsons Problems to Scaffold the Learning of Regular Expressions. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 457--463. https://doi.org/10.1145/3587102.3588853
Index Terms
- Evaluating Micro Parsons Problems as Exam Questions
Recommendations
Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs
ITiCSE-WGR '22: Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science EducationProgramming is a complex task that requires the development of many skills including knowledge of syntax, problem decomposition, algorithm development, and debugging. Code-writing activities are commonly used to help students develop these skills, but ...
Investigating the Effectiveness of Variations of Micro Parsons Problems
ICER '23: Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2Parsons problems have been used to provide scaffolding for introductory learners. Instead of asking learners to write from scratch, Parsons problems provide blocks of mixed-up code, and ask learners to rearrange them into the correct order. In ...
Using Micro Parsons Problems to Scaffold the Learning of Regular Expressions
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1Regular expressions (regex) are a text processing method widely used in data analysis, web scraping, and input validation. However, students find regular expressions difficult to create since they use a terse language of characters. Parsons problems can ...
Comments
Information & Contributors
Information
Published In
July 2024
776 pages
ISBN:9798400706004
DOI:10.1145/3649217
- General Chairs:
- Mattia Monga,
- Violetta Lonati,
- Erik Barendsen,
- Program Chairs:
- Judithe Sheard,
- James Paterson
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ITiCSE 2024
Sponsor:
Acceptance Rates
Overall Acceptance Rate 552 of 1,613 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 59Total Downloads
- Downloads (Last 12 months)59
- Downloads (Last 6 weeks)8
Reflects downloads up to 25 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in