research-article

Open access

Prompt Problems: A New Programming Exercise for the Generative AI Era

Authors:

Andrew Luxton-Reilly,

Thezyrie Amarouche,

Brett A. Becker,

Brent N. ReevesAuthors Info & Claims

SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1

Pages 296 - 302

https://doi.org/10.1145/3626252.3630909

Published: 07 March 2024 Publication History

Abstract

Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.

References

[1]

Joe Michael Allen, Kelly Downey, Kris Miller, Alex Daniel Edgcomb, and Frank Vahid. 2019. Many Small Programs in CS1: Usage Analysis from Multiple Universities. In 2019 ASEE Annual Conference & Exposition. ASEE Conferences, Tampa, Florida, 1--13. https://peer.asee.org/33084.

[2]

Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson. 2023. StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code. arxiv: 2306.04556 [cs.LG]

[3]

Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, NY, USA, 500--506. https://doi.org/10.1145/3545945.3569759

Digital Library

[4]

Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice L. Pearce, and James Prather. 2019. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (Aberdeen, Scotland Uk) (ITiCSE-WGR '19). ACM, NY, NY, USA, 177--210. https://doi.org/10.1145/3344429.3372508

Digital Library

[5]

Bruno Pereira Cipriano and Pedro Alves. 2023. GPT-3 vs Object Oriented Programming Assignments: An Experience Report. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, NY, USA, 61--67. https://doi.org/10.1145/3587102.3588814

Digital Library

[6]

Paul Denny, Brett A. Becker, Juho Leinonen, and James Prather. 2023 a. Chat Overflow: Artificially Intelligent Models for Computing Education - RenAIssance or ApocAIypse?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, NY, USA, 3--4. https://doi.org/10.1145/3587102.3588773

Digital Library

[7]

Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023 b. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, NY, USA, 1136--1142. https://doi.org/10.1145/3545945.3569823

Digital Library

[8]

Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. CodeWrite: Supporting Student-Driven Practice of Java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (Dallas, TX, USA) (SIGCSE '11). Association for Computing Machinery, NY, USA, 471--476. https://doi.org/10.1145/1953163.1953299

Digital Library

[9]

Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2023 c. Computing Education in the Era of Generative AI. arxiv: 2306.02608 [cs.CY]

[10]

Yuemeng Du, Andrew Luxton-Reilly, and Paul Denny. 2020. A Review of Research on Parsons Problems. In Proceedings of the Twenty-Second Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE'20). Association for Computing Machinery, NY, USA, 195--202. https://doi.org/10.1145/3373165.3373187

Digital Library

[11]

Barbara J. Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S. Miller, Briana B. Morrison, Janice L. Pearce, and Susan H. Rodger. 2022. Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs. In Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (Dublin, Ireland) (ITiCSE-WGR '22). Association for Computing Machinery, NY, USA, 191--234. https://doi.org/10.1145/3571785.3574127

Digital Library

[12]

Tony Haoran Feng, Paul Denny, Burkhard C. Wünsche, Andrew Luxton-Reilly, and Steffan Hooper. 2024. More Than Meets the AI: Evaluating the Performance of GPT-4 on Computer Graphics Assessment Questions. In Proceedings of the 26th Australasian Computing Education Conference (Sydney, NSW, Australia) (ACE '24). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3636243.3636263

Digital Library

[13]

James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22). ACM, NY, USA, 10--19. https://doi.org/10.1145/3511861.3511863

Digital Library

[14]

James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI Wants to Know If This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE '23). Association for Computing Machinery, NY, USA, 97--104. https://doi.org/10.1145/3576123.3576134

Digital Library

[15]

Breanna Jury, Angela Lorusso, Juho Leinonen, Paul Denny, and Andrew Luxton-Reilly. 2024. Evaluating LLM-generated Worked Examples in an Introductory Programming Course. In Proceedings of the 26th Australasian Computing Education Conference (Sydney, NSW, Australia) (ACE '24). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3636243.3636252

Digital Library

[16]

Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, NY, USA, Article 455, bibinfonumpages23 pages. https://doi.org/10.1145/3544548.3580919

Digital Library

[17]

Sam Lau and Philip Guo. 2023. From "Ban It Till We Understand It" to "Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools Such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER '23). Association for Computing Machinery, New York, NY, USA, 106--121. https://doi.org/10.1145/3568813.3600138

Digital Library

[18]

Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023 a. Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 124--130. https://doi.org/10.1145/3587102.3588785

Digital Library

[19]

Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023 b. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, NY, USA, 563--569. https://doi.org/10.1145/3545945.3569770

Digital Library

[20]

Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. arxiv: 2308.06921 [cs.CY]

[21]

Stephen MacNeil, Joanne Kim, Juho Leinonen, Paul Denny, Seth Bernstein, Brett A. Becker, Michel Wermelinger, Arto Hellas, Andrew Tran, Sami Sarsa, James Prather, and Viraj Kumar. 2023 a. The Implications of Large Language Models for CS Teachers and Students. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, NY, USA, 1255. https://doi.org/10.1145/3545947.3573358

Digital Library

[22]

Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023 b. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, NY, USA, 931--937. https://doi.org/10.1145/3545945.3569785

Digital Library

[23]

Kamil Malinka, Martin Peres'ini, Anton Firc, Ondrej Hujnák, and Filip Janus. 2023. On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). ACM, NY, USA, 47--53. https://doi.org/10.1145/3587102.3588827

Digital Library

[24]

Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming: With Github Copilot and ChatGPT. Manning, Shelter Island, NY.

[25]

James Prather, Paul Denny, Juho Leinonen, Brett Becker, et al. 2023 a. The Robots are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (Turku, Finland) (ITiCSE-WGR '23). ACM, NY, USA.

Digital Library

[26]

James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023 b. “It's Weird That It Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. ACM Trans. Comput.-Hum. Interact., Vol. 31, 1, Article 4 (Nov 2023), bibinfonumpages31 pages. https://doi.org/10.1145/3617367

Digital Library

[27]

Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). ACM, NY, USA, 299--305. https://doi.org/10.1145/3587102.3588805

Digital Library

[28]

Advait Sarkar. 2023. Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?. In Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Cascais, Portugal) (Onward! 2023). Association for Computing Machinery, New York, NY, USA, 153--167. https://doi.org/10.1145/3622758.3622882

Digital Library

[29]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22). Association for Computing Machinery, NY, USA, 27--43. https://doi.org/10.1145/3501385.3543957

Digital Library

[30]

Leonard Tang, Elizabeth Ke, Nikhil Singh, Bo Feng, Derek Austin, Nakul Verma, and Iddo Drori. 2022. Solving Probability And Statistics Problems By Probabilistic Program Synthesis At Human Level And Predicting Solvability. In Artificial Intelligence in Education: 23rd International Conference, AIED 2022, Durham, UK, July 27--31, 2022, Proceedings, Part II (Durham, United Kingdom). Springer-Verlag, Berlin, Heidelberg, 612--615. https://doi.org/10.1007/978--3-031--11647--6_127

[31]

Matti Tedre and Henriikka Vartiainen. 2023. K-12 Computing Education for the AI Era: From Data Literacy to Data Agency. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, NY, USA, 1--2. https://doi.org/10.1145/3587102.3593796 io

Digital Library

Cited By

Vahid F(2024)AI in CS Education: Opportunities, Challenges, and Pitfalls to AvoidACM Inroads10.1145/367920515:3(52-57)Online publication date: 21-Aug-2024
https://dl.acm.org/doi/10.1145/3679205
Smith DDenny PFowler MJoyner DKim MWang XXia M(2024)Prompting for Comprehension: Exploring the Intersection of Explain in Plain English Questions and Prompt WritingProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662039(39-50)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662039
Cipriano BMonga MLonati VBarendsen ESheard JPaterson J(2024)Towards the Integration of Large Language Models in an Object-Oriented Programming CourseProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 210.1145/3649405.3659473(832-833)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3649405.3659473
Show More Cited By

Index Terms

Prompt Problems: A New Programming Exercise for the Generative AI Era
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
        CS1

Recommendations

Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1

Much of the recent work investigating large language models and AI Code Generation tools in computing education has focused on assessing their capabilities for solving typical programming problems and for generating resources such as code explanations ...
Self-Regulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming Problems
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

We explored how undergraduate introductory programming students naturalistically used generative AI to solve programming problems. We focused on the relationship between their use of AI to their self-regulation strategies, self-efficacy, and fear of ...
AI-Driven Content Creation: Revolutionizing Educational Materials
L@S '24: Proceedings of the Eleventh ACM Conference on Learning @ Scale

The integration of Artificial Intelligence (AI) into the field of education is an unprecedented trend with the potential to revolutionize teaching approaches and significantly improve the overall learning experience. This workshop offers an opportunity ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1

March 2024

1583 pages

ISBN:9798400704239

DOI:10.1145/3626252

General Chairs:
Ben Stephenson
University of Calgary, Canada
,
Jeffrey A. Stone
Penn State University
,
Program Chairs:
Lina Battestilli
North Carolina State University, USA
,
Samuel A. Rebelsky
Grinnell College
,
Libby Shoop
Macalester College

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 March 2024

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ulla Tuominen Foundation

Conference

SIGCSE 2024

Sponsor:

SIGCSE

SIGCSE 2024: The 55th ACM Technical Symposium on Computer Science Education

March 20 - 23, 2024

OR, Portland, USA

Acceptance Rates

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE Virtual 2024

Sponsor:
sigcse

1st ACM Virtual Global Computing Education Conference

December 5 - 8, 2024

Virtual Event , NC , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
2,625
Total Downloads

Downloads (Last 12 months)2,625
Downloads (Last 6 weeks)368

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vahid F(2024)AI in CS Education: Opportunities, Challenges, and Pitfalls to AvoidACM Inroads10.1145/367920515:3(52-57)Online publication date: 21-Aug-2024
https://dl.acm.org/doi/10.1145/3679205
Smith DDenny PFowler MJoyner DKim MWang XXia M(2024)Prompting for Comprehension: Exploring the Intersection of Explain in Plain English Questions and Prompt WritingProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662039(39-50)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662039
Cipriano BMonga MLonati VBarendsen ESheard JPaterson J(2024)Towards the Integration of Large Language Models in an Object-Oriented Programming CourseProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 210.1145/3649405.3659473(832-833)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3649405.3659473
Margulieux LPrather JReeves BBecker BCetin Uzun GLoksa DLeinonen JDenny PMonga MLonati VBarendsen ESheard JPaterson J(2024)Self-Regulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming ProblemsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653621(276-282)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653621
Koutcheme CDainese NSarsa SHellas ALeinonen JDenny PMonga MLonati VBarendsen ESheard JPaterson J(2024)Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-JudgeProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653612(52-58)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653612
Denny PSmith DFowler MPrather JBecker BLeinonen JMonga MLonati VBarendsen ESheard JPaterson J(2024)Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting SkillsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653587(283-289)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653587
del Carpio Gutierrez ADenny PLuxton-Reilly AMonga MLonati VBarendsen ESheard JPaterson J(2024)Automating Personalized Parsons Problems with Customized Contexts and ConceptsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653568(688-694)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653568
Potriasaeva ADzialets KGolubev YBirillo A(2024)Using a Low-Code Environment to Teach Programming in the Era of LLMsProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 210.1145/3632621.3671429(542-543)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632621.3671429
Prather JReeves BLeinonen JMacNeil SRandrianasolo ABecker BKimmel BWright JBriggs B(2024)The Widening Gap: The Benefits and Harms of Generative AI for Novice ProgrammersProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671116(469-486)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671116
Skripchuk JBacher JPrice T(2024)An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAIProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671112(487-501)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671112
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents