research-article

Open access

Using GitHub Copilot to Solve Simple Programming Problems

Author:

Michel WermelingerAuthors Info & Claims

SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

Pages 172 - 178

https://doi.org/10.1145/3545945.3569830

Published: 03 March 2023 Publication History

PDF eReader

Abstract

The teaching and assessment of introductory programming involves writing code that solves a problem described by text. Previous research found that OpenAI's Codex, a natural language machine learning model trained on billions of lines of code, performs well on many programming problems, often generating correct and readable Python code. GitHub's version of Codex, Copilot, is freely available to students. This raises pedagogic and academic integrity concerns. Educators need to know what Copilot is capable of, in order to adapt their teaching to AI-powered programming assistants. Previous research evaluated the most performant Codex model quantitatively, e.g. how many problems have at least one correct suggestion that passes all tests. Here I evaluate Copilot instead, to see if and how it differs from Codex, and look qualitatively at the generated suggestions, to understand the limitations of Copilot. I also report on the experience of using Copilot for other activities asked of students in programming courses: explaining code, generating tests and fixing bugs. The paper concludes with a discussion of the implications of the observed capabilities for the teaching of programming.

References

[1]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. https://doi.org/10.48550/ARXIV.2107.03374

Crossref

Google Scholar

[2]

Thomas Dohmke. 2022. GitHub Copilot is generally available to all developers. GitHub blog. https://github.blog/2022-06-21-github-copilot-is-generally-available-to-all-developers

Google Scholar

[3]

Alireza Ebrahimi. 1994. Novice Programmer Errors: Language Constructs and Plan Composition. Int. J. Hum.-Comput. Stud., Vol. 41, 4 (Oct. 1994), 457--480. https://doi.org/10.1006/ijhc.1994.1069

Digital Library

Google Scholar

[4]

James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22). Association for Computing Machinery, New York, NY, USA, 10--19. https://doi.org/10.1145/3511861.3511863

Digital Library

Google Scholar

[5]

Kathi Fisler. 2014. The Recurring Rainfall Problem. In Proceedings of the Tenth Annual Conference on International Computing Education Research (Glasgow, Scotland, United Kingdom) (ICER '14). Association for Computing Machinery, New York, NY, USA, 35--42. https://doi.org/10.1145/2632320.2632346

Digital Library

Google Scholar

[6]

Mark Guzdial, Rachel Fithian, Andrea Forte, and Lauren Rich. 2003. Report on Pilot Offering of CS1315 Introduction to Media Computation With Comparison to CS1321 and COE1361. Technical Report. Georgia Tech.

Google Scholar

[7]

Antti-Jussi Lakanen, Vesa Lappalainen, and Ville Isomöttönen. 2015. Revisiting Rainfall to Explore Exam Questions and Performance on CS1. In Proceedings of the 15th Koli Calling Conference on Computing Education Research (Koli, Finland) (Koli Calling '15). Association for Computing Machinery, New York, NY, USA, 40--49. https://doi.org/10.1145/2828959.2828970

Digital Library

Google Scholar

[8]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. https://doi.org/10.48550/ARXIV.2203.07814

Crossref

Google Scholar

[9]

Richard Lobb and Jenny Harlow. 2016. Coderunner: A Tool for Assessing Computer Programming Skills. ACM Inroads, Vol. 7, 1 (Feb. 2016), 47--51. https://doi.org/10.1145/2810041

Digital Library

Google Scholar

[10]

Orna Muller, David Ginat, and Bruria Haberman. 2007. Pattern-Oriented Instruction and Its Influence on Problem Decomposition and Solution Construction. In Proceedings of the 12th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (Dundee, Scotland) (ITiCSE '07). Association for Computing Machinery, New York, NY, USA, 151--155. https://doi.org/10.1145/1268784.1268830

Digital Library

Google Scholar

[11]

Paul Piwek, Michel Wermelinger, Robin Laney, and Richard Walker. 2019. Learning to Program: From Problems to Code. In Proceedings of the 3rd Conference on Computing Education Practice (Durham, United Kingdom) (CEP '19). Association for Computing Machinery, New York, NY, USA, Article 14, 4 pages. https://doi.org/10.1145/3294016.3294024

Digital Library

Google Scholar

[12]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22). Association for Computing Machinery, New York, NY, USA, 27--43. https://doi.org/10.1145/3501385.3543957

Digital Library

Google Scholar

[13]

Simon. 2013. Soloway's Rainfall Problem Has Become Harder. In Learning and Teaching in Computing and Engineering. IEEE, 130--135. https://doi.org/10.1109/LaTiCE.2013.44

Digital Library

Google Scholar

[14]

Elliot Soloway. 1986. Learning to Program = Learning to Construct Mechanisms and Explanations. Commun. ACM, Vol. 29, 9 (Sept. 1986), 850--858. https://doi.org/10.1145/6592.6594iog

Crossref

Google Scholar

Cited By

View all

Lira WSantos Neto POsorio L(2024)Uma análise do uso de ferramentas de geração de código por alunos de computaçãoAnais do IV Simpósio Brasileiro de Educação em Computação (EDUCOMP 2024)10.5753/educomp.2024.237427(63-71)Online publication date: 22-Apr-2024
https://doi.org/10.5753/educomp.2024.237427
Poitras ECrane BDempsey DBragg TSiegel ALin M(2024)Cognitive Apprenticeship and Artificial Intelligence Coding AssistantsNavigating Computer Science Education in the 21st Century10.4018/979-8-3693-1066-3.ch013(261-281)Online publication date: 26-Feb-2024
https://doi.org/10.4018/979-8-3693-1066-3.ch013
Jošt GTaneski VKarakatič S(2024)The Impact of Large Language Models on Programming Education and Student Learning OutcomesApplied Sciences10.3390/app1410411514:10(4115)Online publication date: 13-May-2024
https://doi.org/10.3390/app14104115
Show More Cited By

Index Terms

Using GitHub Copilot to Solve Simple Programming Problems

Recommendations

Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code ...
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
ACE '23: Proceedings of the 25th Australasian Computing Education Conference

The introduction of OpenAI Codex sparked a surge of interest in the impact of generative AI models on computing education practices. Codex is also the underlying model for GitHub Copilot, a plugin which makes AI-generated code accessible to students ...
Checking Conformance to a Subset of the Python Language
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2

Introductory courses usually only teach a small subset of a programming language and its library, in order to focus on the general concepts rather than overwhelm students with the syntactic, semantic and API minutiae of a particular language.

This paper ...

Comments

Information & Contributors

Information

Published In

SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

March 2023

1481 pages

ISBN:9781450394314

DOI:10.1145/3545945

General Chairs:
Maureen Doyle
Northern Kentucky University, USA
,
Ben Stephenson
University of Calgary, Canada
,
Program Chairs:
Brian Dorn
University of Nebraska at Omaha, USA
,
Leen-Kiat Soh
University of Nebraska-Lincoln, USA
,
Lina Battestilli
North Carolina State University, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 March 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCSE 2023

Sponsor:

SIGCSE

SIGCSE 2023: The 54th ACM Technical Symposium on Computer Science Education

March 15 - 18, 2023

Toronto ON, Canada

Acceptance Rates

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE Virtual 2024

Sponsor:
sigcse

1st ACM Virtual Global Computing Education Conference

December 5 - 8, 2024

Virtual Event , NC , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
3,030
Total Downloads

Downloads (Last 12 months)2,384
Downloads (Last 6 weeks)213

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Lira WSantos Neto POsorio L(2024)Uma análise do uso de ferramentas de geração de código por alunos de computaçãoAnais do IV Simpósio Brasileiro de Educação em Computação (EDUCOMP 2024)10.5753/educomp.2024.237427(63-71)Online publication date: 22-Apr-2024
https://doi.org/10.5753/educomp.2024.237427
Poitras ECrane BDempsey DBragg TSiegel ALin M(2024)Cognitive Apprenticeship and Artificial Intelligence Coding AssistantsNavigating Computer Science Education in the 21st Century10.4018/979-8-3693-1066-3.ch013(261-281)Online publication date: 26-Feb-2024
https://doi.org/10.4018/979-8-3693-1066-3.ch013
Jošt GTaneski VKarakatič S(2024)The Impact of Large Language Models on Programming Education and Student Learning OutcomesApplied Sciences10.3390/app1410411514:10(4115)Online publication date: 13-May-2024
https://doi.org/10.3390/app14104115
Sena JBarreto ABarbosa JAlves K(2024)POTENCIALIDADES E DESAFIOS DO GITHUB COPILOT COMO FERRAMENTA DA INTELIGÊNCIA ARTIFICIALPOTENTIALS AND CHALLENGES OF GITHUB COPILOT AS AN ARTIFICIAL INTELLIGENCE TOOLPOTENCIALES Y DESAFÍOS DE GITHUB COPILOT COMO HERRAMIENTA DE INTELIGENCIA ARTIFICIALP2P E INOVAÇÃO10.21728/p2p.2024v10n2e-703110:2Online publication date: 22-Jun-2024
https://doi.org/10.21728/p2p.2024v10n2e-7031
Russo D(2024)Navigating the Complexity of Generative AI Adoption in Software EngineeringACM Transactions on Software Engineering and Methodology10.1145/365215433:5(1-50)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3652154
Mahon JMac Namee BBecker BMonga MLonati VBarendsen ESheard JPaterson J(2024)Guidelines for the Evolving Role of Generative AI in Introductory Programming Based on Emerging PracticeProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653602(10-16)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653602
Smith DZilles CMonga MLonati VBarendsen ESheard JPaterson J(2024)Code Generation Based Grading: Evaluating an Auto-grading Mechanism for "Explain-in-Plain-English" QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653582(171-177)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653582
Sergeyuk ATitov SIzadi MDig DBryksin TGolubev YBezzubov A(2024)In-IDE Human-AI Experience in the Era of Large Language Models; A Literature ReviewProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments10.1145/3643796.3648463(95-100)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643796.3648463
Davila NWiese ISteinmacher ILucio da Silva LKawamoto AFavaro GNunes IRoychoudhury APaiva AAbreu RStorey MAniche MNagappan N(2024)An Industry Case Study on Adoption of AI-based Programming AssistantsProceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice10.1145/3639477.3643648(92-102)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639477.3643648
Lehtinen TKoutcheme CHellas ARoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension QuestionsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640058(221-232)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640058
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises

Checking Conformance to a Subset of the Python Language