research-article

How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot

Author:

Naser Al MadiAuthors Info & Claims

ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

Article No.: 205, Pages 1 - 5

https://doi.org/10.1145/3551349.3560438

Published: 05 January 2023 Publication History

Abstract

Background: Recent advancements in large language models have motivated the practical use of such models in code generation and program synthesis. However, little is known about the effects of such tools on code readability and visual attention in practice. Objective: In this paper, we focus on GitHub Copilot to address the issues of readability and visual inspection of model generated code. Readability and low complexity are vital aspects of good source code, and visual inspection of generated code is important in light of automation bias. Method: Through a human experiment (n=21) we compare model generated code to code written completely by human programmers. We use a combination of static code analysis and human annotators to assess code readability, and we use eye tracking to assess the visual inspection of code. Results: Our results suggest that model generated code is comparable in complexity and readability to code written by human pair programmers. At the same time, eye tracking data suggests, to a statistically significant level, that programmers direct less visual attention to model generated code. Conclusion: Our findings highlight that reading code is more important than ever, and programmers should beware of complacency and automation bias with model generated code.

References

[1]

Ritchie Schacher Adam Archer and Scott Will. [n.d.]. Program in Pairs. Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/

[2]

Naser Al Madi, Drew Guarnera, Bonita Sharif, and Jonathan Maletic. 2021. EMIP Toolkit: A Python Library for Customized Post-processing of the Eye Movements in Programming Dataset. In ACM Symposium on Eye Tracking Research and Applications. 1–6.

[3]

Naser Al Madi, Cole S Peterson, Bonita Sharif, and Jonathan I Maletic. 2021. From Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 172–183.

[4]

Marco Barenkamp, Jonas Rebstadt, and Oliver Thomas. 2020. Applications of AI in classical software engineering. AI Perspectives 2, 1 (2020), 1–15.

[5]

Boehm Barry 1981. Software engineering economics. New York 197(1981).

[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.

[7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021).

[8]

Martha E Crosby, Jean Scholtz, and Susan Wiedenbeck. 2002. The Roles Beacons Play in Comprehension for Novice and Expert Programmers. In PPIG. 5.

[9]

Martha E Crosby and Jan Stelovsky. 1990. How do we read algorithms? A case study. Computer 23, 1 (1990), 25–35.

Digital Library

[10]

Cem Kaner 2004. Software engineering metrics: What do they measure and how do we know?. In In METRICS 2004. IEEE CS. Citeseer.

[11]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. (2022).

[12]

Zohar Manna and Richard J Waldinger. 1971. Toward automatic program synthesis. Commun. ACM 14, 3 (1971), 151–165.

Digital Library

[13]

Alberto S Nuñez-Varela, Héctor G Pérez-Gonzalez, Francisco E Martínez-Perez, and Carlos Soubervielle-Montalvo. 2017. Source code metrics: A systematic mapping study. Journal of Systems and Software 128 (2017), 164–197.

Digital Library

[14]

Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs?arXiv preprint arXiv:2112.02125(2021).

[15]

Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund. 2021. Program comprehension and code complexity metrics: An fmri study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 524–536.

Digital Library

[16]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.

[17]

Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research.Psychological bulletin 124, 3 (1998), 372.

[18]

Erik D Reichle, Keith Rayner, and Alexander Pollatsek. 2003. The EZ Reader model of eye-movement control in reading: Comparisons to other models. Behavioral and brain sciences 26, 4 (2003), 445–476.

[19]

Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2017. Automatically assessing code understandability: How far are we?. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 417–427.

[20]

Zohreh Sharafi, Bonita Sharif, Yann-Gaël Guéhéneuc, Andrew Begel, Roman Bednarik, and Martha Crosby. 2020. A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering 25, 5 (2020), 3128–3174.

Digital Library

[21]

Bonita Sharif and Jonathan I Maletic. 2016. iTrace: Overcoming the Limitations of Short Code Examples in Eye Tracking Experiments. In ICSME. 647.

[22]

Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875(2021).

[23]

Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2006. Analyzing individual performance of source code review using reviewers’ eye movement. In Proceedings of the 2006 symposium on Eye tracking research & applications. 133–140.

Digital Library

[24]

Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI ’22 Extended Abstracts). ACM.

Digital Library

[25]

Guido Van Rossum, Barry Warsaw, and Nick Coghlan. 2001. PEP 8: style guide for Python code. Python. org 1565(2001).

[26]

Laurie Williams. 2001. Integrating pair programming into a software development process. In Proceedings 14th Conference on Software Engineering Education and Training.’In search of a software engineering profession’(Cat. No. PR01059). IEEE, 27–36.

[27]

Frank F Xu, Bogdan Vasilescu, and Graham Neubig. 2022. In-IDE Code Generation from Natural Language: Promise and Challenges. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 2(2022), 1–47.

Digital Library

[28]

Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. 21–29.

Digital Library

Cited By

Martinović BRozić R(2025)Perceived Impact of AI-Based Tooling on Software Development Code QualitySN Computer Science10.1007/s42979-024-03608-46:1Online publication date: 6-Jan-2025
https://doi.org/10.1007/s42979-024-03608-4
Stone I(2024)Exploring Human-Centered Approaches in Generative AI and Introductory Programming Research: A Scoping ReviewProceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689553(1-7)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3689535.3689553
Yen RZhu JSuh SXia HZhao J(2024)CoLadder: Manipulating Code Generation via Multi-Level BlocksProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676357(1-20)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676357
Show More Cited By

Index Terms

How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools
2. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments

Recommendations

Copilot-in-the-Loop: Fixing Code Smells in Copilot-Generated Python Code using Copilot
ASE '24: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

As one of the most popular dynamic languages, Python experiences a decrease in readability and maintainability when code smells are present. Recent advancements in Large Language Models have sparked growing interest in AI-enabled tools for both code ...
Assessing the Readability of ChatGPT Code Snippet Recommendations: A Comparative Study
SBES '23: Proceedings of the XXXVII Brazilian Symposium on Software Engineering

Developers often rely on code search engines to find high-quality and reusable code snippets online, such as those available on Stack Overflow. Recently, ChatGPT, a language model trained for dialog tasks, has been gaining attention as a promising ...
Demystifying code snippets in code reviews: a study of the OpenStack and Qt communities and a practitioner survey
Abstract
Code review is widely known as one of the best practices for software quality assurance in software development. In a typical code review process, reviewers check the code committed by developers to ensure the quality of the code, during which ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

October 2022

2006 pages

ISBN:9781450394758

DOI:10.1145/3551349

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 January 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ASE '22

ASE '22: 37th IEEE/ACM International Conference on Automated Software Engineering

October 10 - 14, 2022

MI, Rochester, USA

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
822
Total Downloads

Downloads (Last 12 months)327
Downloads (Last 6 weeks)15

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Martinović BRozić R(2025)Perceived Impact of AI-Based Tooling on Software Development Code QualitySN Computer Science10.1007/s42979-024-03608-46:1Online publication date: 6-Jan-2025
https://doi.org/10.1007/s42979-024-03608-4
Stone I(2024)Exploring Human-Centered Approaches in Generative AI and Introductory Programming Research: A Scoping ReviewProceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689553(1-7)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3689535.3689553
Yen RZhu JSuh SXia HZhao J(2024)CoLadder: Manipulating Code Generation via Multi-Level BlocksProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676357(1-20)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676357
Cheng RWang RZimmermann TFord D(2024)“It would work for me too”: How Online Communities Shape Software Developers’ Trust in AI-Powered Code Generation ToolsACM Transactions on Interactive Intelligent Systems10.1145/365199014:2(1-39)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3651990
Gardella NPettit RRiggs SMonga MLonati VBarendsen ESheard JPaterson J(2024)Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code GenerationProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653615(290-296)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653615
Sikand SPhokela KSharma VSingi KKaulgud VTung TSharma PBurden A(2024)How much SPACE do metrics have in GenAI assisted software development?Proceedings of the 17th Innovations in Software Engineering Conference10.1145/3641399.3641419(1-5)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1145/3641399.3641419
Kim SLiao QVorvoreanu MBallard SVaughan J(2024)"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and TrustProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658941(822-835)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658941
Tang NChen MNing ZBansal AHuang YMcMillan CLi T(2024)Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00015(40-46)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00015
Cheng RBarik TLeung AHohman FNichols J(2024)BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00012(13-23)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00012
Hamer Sd’Amorim MWilliams L(2024)Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00014(87-94)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00014
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents