How Do Software Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests
Authors: 
Moataz Chouchen, Narjes Bessghaier, Mahi Begoug, Ali Ouni, Eman Alomar, Mohamed Wiem MkaouerAuthors Info & Claims
Moataz Chouchen, Narjes Bessghaier, Mahi Begoug, Ali Ouni, Eman Alomar, Mohamed Wiem MkaouerAuthors Info & ClaimsPages 212 - 216
Abstract
Nowadays, Large Language Models (LLMs) play a pivotal role in software engineering. Developers can use LLMs to address software development-related tasks such as documentation, code refactoring, debugging, and testing. ChatGPT, released by OpenAI, has become the most prominent LLM. In particular, ChatGPT is a cutting-edge tool for providing recommendations and solutions for developers in their pull requests (PRs). However, little is known about the characteristics of PRs that incorporate ChatGPT compared to those without it and what developers usually use it for. To this end, we quantitatively analyzed 243 PRs that listed at least one ChatGPT prompt against a representative sample of 384 PRs without any ChatGPT prompts. Our findings show that developers use ChatGPT in larger, time-consuming pull requests that are five times slower to be closed than PRs that do not use ChatGPT. Furthermore, we perform a qualitative analysis to build a taxonomy of the topics developers primarily address in their prompts. Our analysis results in a taxonomy comprising 8 topics and 32 sub-topics. Our findings highlight that ChatGPT is often used in review-intensive pull requests. Moreover, our taxonomy enriches our understanding of the developer's current applications of ChatGPT.
References
[1]
_____. 2023. The reolication package for our study: https://figshare.com/s/2bc2c11e694997262a87.
[2]
2023. ChatGPT Shared Links FAQ. https://help.openai.com/en/articles/7925741-chatgpt-shared-links-faq. Accessed on: Dec 01, 2023.
[3]
2023. Pull Request #1. https://github.com/Wissididom/Bonbon-Reminder-StreamElements/pull/1 Accessed on: Dec 01, 2023.
[4]
2023. Pull Request #1775. https://github.com/ggerganov/llama.cpp/pull/1775 Accessed on: Dec 01, 2023.
[5]
2023. Pull Request #2. https://github.com/DogeisCut/extensions/pull/2 Accessed on: Dec 01, 2023.
[6]
2023. Pull Request #209. https://github.com/sCrypt-Inc/scryptTS-docs/pull/209 Accessed on: Dec 01, 2023.
[7]
2023. Pull Request #3937. https://github.com/snapshot-labs/snapshot/pull/3937 Accessed on: Dec 01, 2023.
[8]
2023. Pull Request #5602. https://github.com/rancher-sandbox/rancher-desktop/pull/5602 Accessed on: Dec 01, 2023.
[9]
2023. Pull Request #838. https://github.com/CodeIntelligenceTesting/cifuzz/pull/838 Accessed on: Dec 01, 2023.
[10]
2023. Pull Request #897. https://github.com/reworkd/AgentGPT/pull/897 Accessed on: Dec 01, 2023.
[11]
2023. Pull Request#1638. https://github.com/TeamCodeStream/codestream/pull/1638 Accessed on: Dec 01, 2023.
[12]
2023. Pull Request#219. https://github.com/pwncollege/dojo/pull/219 Accessed on: Dec 01, 2023.
[13]
2023. Pull Request#2230. https://github.com/faker-js/faker/pull/2230 Accessed on: Dec 01, 2023.
[14]
2023. Pull Request#2316. https://github.com/validatorjs/validator.js/pull/2316 Accessed on: Dec 01, 2023.
[15]
2023. Pull Request#301. https://github.com/OSRSB/OsrsBot/pull/301 Accessed on: Dec 01, 2023.
[16]
2023. Pull Request#40. https://github.com/rom-py/rompy/pull/40 Accessed on: Dec 01, 2023.
[17]
2023. Pull Request#8902. https://github.com/NixOS/nix/pull/8902 Accessed on: Dec 01, 2023.
[18]
2023. snapshot-labs/snapshot. https://github.com/snapshot-labs/snapshot Accessed on: Dec 01, 2023.
[19]
2023. TeamCodeStream. https://github.com/TeamCodeStream/codestream Accessed on: Dec 01, 2023.
[20]
Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, Mst Shamima Aktar, and Tommi Mikkonen. 2023. Towards human-bot collaborative software architecting with chatgpt. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. 279--285.
[21]
Eman Abdullah AlOmar, Moataz Chouchen, Mohamed Wiem Mkaouer, and Ali Ouni. 2022. Code review practices for refactoring changes: An empirical study on openstack. In Proceedings of the 19th International Conference on Mining Software Repositories. 689--701.
[22]
Narjes Bessghaier, Mohammed Sayagh, Ali Ouni, and Mohamed Wiem Mkaouer. 2023. What Constitutes the Deployment and Run-time Configuration System? An Empirical Study on OpenStack Projects. ACM Transactions on Software Engineering and Methodology (2023).
[23]
Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.
[24]
Tzeng-Ji Chen. 2023. ChatGPT and other artificial intelligence applications speed up scientific writing. Journal of the Chinese Medical Association 86, 4 (2023), 351--353.
[25]
Moataz Chouchen, Ali Ouni, Raula Gaikovina Kula, Dong Wang, Patanamon Thongtanunam, Mohamed Wiem Mkaouer, and Kenichi Matsumoto. 2021. Anti-patterns in modern code review: Symptoms and prevalence. In 2021 IEEE international conference on software analysis, evolution and reengineering (SANER). IEEE, 531--535.
[26]
Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114, 3 (1993), 494.
[27]
Flávia Coelho, Nikolaos Tsantalis, Tiago Massoni, and Everton LG Alves. 2021. An empirical study on refactoring-inducing pull requests. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1--12.
[28]
William Jay Conover. 1999. Practical nonparametric statistics. Vol. 350. john wiley & sons.
[29]
Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. IEEE, 275--284.
[30]
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration Code Generation via ChatGPT. arXiv preprint arXiv:2304.07590 (2023).
[31]
Hongyang Du, Zonghang Li, Dusit Niyato, Jiawen Kang, Zehui Xiong, Dong In Kim, et al. 2023. Enabling AI-generated content (AIGC) services in wireless edge networks. arXiv preprint arXiv:2301.03220 (2023).
[32]
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533 (2023).
[33]
Abid Haleem, Mohd Javaid, and Ravi Pratap Singh. 2022. An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil transactions on benchmarks, standards and evaluations 2, 4 (2022), 100089.
[34]
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745 (2023).
[35]
OpenAI. 2023. Pull Request example:. https://chat.openai.com/share/cb1b4c4e-b2bc-4b1d-9df2-700be0cab72d Accessed on: Dec 01, 2023.
[36]
Partha Pratim Ray. 2023. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems (2023).
[37]
Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen'sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research, Vol. 177. 34.
[38]
Danilo Silva, Nikolaos Tsantalis, and Marco Tulio Valente. 2016. Why we refactor? confessions of github contributors. In Proceedings of the 2016 24th acm sigsoft international symposium on foundations of software engineering. 858--870.
[39]
Dong Wang, Tao Xiao, Patanamon Thongtanunam, Raula Gaikovina Kula, and Kenichi Matsumoto. 2021. Understanding shared links and their intentions to meet information needs in modern code review: A case study of the OpenStack and Qt projects. Empirical Software Engineering 26 (2021), 1--32.
[40]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023).
[41]
Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2303.07839 (2023).
[42]
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica 10, 5 (2023), 1122--1136.
[43]
Tao Xiao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. DevGPT: Studying Developer-ChatGPT Conversations. In Proceedings of the International Conference on Mining Software Repositories (MSR 2024).
Index Terms
- How Do Software Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests
Recommendations
Chatting with AI: Deciphering Developer Conversations with ChatGPT
MSR '24: Proceedings of the 21st International Conference on Mining Software RepositoriesLarge Language Models (LLMs) have been widely adopted and are becoming ubiquitous and integral to software development. However, we have little knowledge as to how these tools are being used by software developers beyond anecdotal evidence and word-of-...
An Empirical Study on GitHub Pull Requests’ Reactions
The pull request mechanism is commonly used to propose source code modifications and get feedback from the community before merging them into a software repository. On GitHub, practitioners can provide feedback on a pull request by either commenting on ...
Developers assignment for analyzing pull requests
SAC '15: Proceedings of the 30th Annual ACM Symposium on Applied ComputingA new collaboration approach is becoming increasingly common in open-source projects: the pull request model. In this kind of collaboration, developers that do not belong to the core team of a project can submit contributions to the core team. In ...
Comments
Information & Contributors
Information
Published In
April 2024
788 pages
ISBN:9798400705878
DOI:10.1145/3643991
- Chair:
- Diomidis Spinellis,
- Program Chair:
- Alberto Bacchelli,
- Program Co-chair:
- Eleni Constantinou
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MSR '24
Sponsor:
MSR '24: 21st International Conference on Mining Software Repositories
April 15 - 16, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 39Total Downloads
- Downloads (Last 12 months)39
- Downloads (Last 6 weeks)16
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in