Measuring GitHub Copilot's Impact on Productivity
Abstract
Case study asks Copilot users about its impact on their productivity, and seeks to find their perceptions mirrored in user data.
Supplementary Material
Supplemental material.
- Download
- 342.52 KB
References
[1]
Amann, S., Proksch, S., Nadi, S., and Mezini, M. A study of visual studio usage in practice. In IEEE 23rd Intern. Conf. on Software Analysis, Evolution, and Reengineering 1. IEEE Computer Society, (March 2016), 124--134
[2]
Austin, J. et al. Program synthesis with large language models. CoRR abs/2108.07732 (2021); https://arxiv.org/abs/2108.07732
[3]
Ari Aye, G., Kim, S., and Li, H. Learning autocompletion from real-world datasets. In Proceedings of the 43rd IEEE/ACM Intern. Conf. on Software Engineering: Software Engineering in Practice, (May 2021), 131--139
[4]
Beller, M., Orgovan, V., Buja, S., and Zimmermann, T. Mind the gap: On the relationship between automatically measured and self-reported productivity. IEEE Software 38, 5 (2020), 24--31.
[5]
Chen, M. et al. Evaluating large language models trained on code. CoRR abs/2107.03374 (2021); https://arxiv.org/abs/2107.03374
[6]
Forsgren, N. et al. The SPACE of developer productivity: There's more to it than you think. Queue 19, 1 (2021), 20--48.
[7]
Hellendoorn, V.J., Proksch, S., Gall, H.C., and Bacchelli, A. When code completion fails: A case study on real-world completions. In Proceedings of the 41st Intern. Conf. on Software Engineering, J.M. Atlee, T. Bultan, and J. Whittle (eds). IEEE/ACM, (May 2019), 960--970
[8]
Hendrycks, D. et al. Measuring coding challenge competence with APPS. CoRR abs/2105.09938, (2021); https://arxiv.org/abs/2105.09938
[9]
Hindle, A. et al. On the naturalness of software. In 34th Intern. Conf. on Software Engineering, M. Glinz, G.C. Murphy, and M. Pezzè (eds). IEEE Computer Society, June 2012, 837--847
[10]
Jaspan, C. and Sadowski, C. No single metric captures productivity. Rethinking Productivity in Software Engineering, (2019), 13--20.
[11]
Kulal, S. et al. Spoc: Search-based pseudocode to code. In Proceedings of Advances in Neural Information Processing Systems 32, H.M. Wallach et al (eds), Dec. 2019, 11883--11894; https://bit.ly/3H7YLtF
[12]
Meyer, A.N., Barr, E.T., Bird, C., and Zimmermann, T. Today was a good day: The daily life of software developers. IEEE Transactions on Software Engineering 47, 5 (2019), 863--880.
[13]
Meyer, A.N. et al. The work life of developers: Activities, switches and perceived productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178--1193.
[14]
Meyer, A.N., Fritz, T., Murphy, G.C., and Zimmermann, T. Software developers' perceptions of productivity. In Proceedings of the 22nd ACM SIGSOFT Intern. Symp. on Foundations of Software Engineering (2014), 19--29.
[15]
Murphy-Hill, E. et al. What predicts software developers' productivity? IEEE Transactions on Software Engineering 47, 3 (2019), 582--594.
[16]
Peng, S., Kalliamvakou, E., Cihon, P., and Demirer, M. The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv:2302.06590 [cs.SE] (2014)
[17]
Ramírez, Y.W. and Nembhard, D.A. Measuring knowledge worker productivity: A taxonomy. J. of Intellectual Capital 5, 4 (2004), 602--628.
[18]
See, A., Roller, S., Kiela, D., and Weston, J. What makes a good conversation? How controllable attributes affect human judgments. In Proceedings of the 2019 Conf. of the North American Chapter of the Assoc. for Computational Linguistics: Human Language Technologies 1, J. Burstein, C. Doran, and T. Solorio (eds). Assoc. for Computational Linguistics, (June 2019), 1702--1723
[19]
Storey, M. et al. Towards a theory of software developer job satisfaction and perceived productivity. In Proceedings of the IEEE Trans. on Software Engineering 47, 10 (2019), 2125--2142.
[20]
Svyatkovskiy, A., Deng, S.K., Fu, S., and Sundaresan, N. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint European Software Eng. Conf. and Symp. on the Foundations of Software Eng., P. Devanbu, M.B. Cohen, and T. Zimmermann (eds). ACM, (Nov. 2020), 1433--1443
[21]
Svyatkovskiy, A. et al. Fast and memory-efficient neural code completion. In Proceedings of the 18th IEEE/ACM Intern. Conf. on Mining Software Repositories, (May 2021, 329--340
[22]
Vaithilingam, P., Zhang, T., and Glassman, E. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 Conf. on Human Factors in Computing Systems.
[23]
Vaithilingam, P., Zhang, T., and Glassman, E.L. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the CHI Conf. on Human Factors in Computing Systems, Association for Computing Machinery, Article 332 (2022), 7
[24]
Wagner, S. and Ruhe, M. A systematic review of productivity factors in software development. arXiv preprint arXiv:1801.06475 (2018).
[25]
Wang, D. et al. From human-human collaboration to human-AI collaboration: Designing AI systems that can work together with people. In Proceedings of the 2020 CHI Conf. on Human Factors in Computing Systems (2020), 1--6.
[26]
Weisz, J.D. et al. Perfection not required? Human-AI partnerships in code translation. In Proceedings of the 26th Intern. Conf. on Intelligent User Interfaces, T. Hammond et al (eds). ACM, (April 2021), 402--412
[27]
Winters, T., Manshreck, T., and Wright, H. Software Engineering at Google: Lessons Learned from Programming Over Time. O'Reilly Media (2020).
[28]
Wold, S., Sjöström, M., and Eriksson, L. PLS-regression: A basic tool of chemometrics. Chemometrics and Intelligent Laboratory Systems 58, 2 (2001), 109--130
[29]
Zhou, W., Kim, S., Murali, V., and Ari Aye, G. Improving code autocompletion with transfer learning. CoRR abs/2105.05991 (2021); https://arxiv.org/abs/2105.05991
Index Terms
- Measuring GitHub Copilot's Impact on Productivity
Recommendations
Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study
The "citation score" remains the most commonly-used measure of academic impact, but is also viewed as practically and conceptually limited. The aim of this case study was to test the feasibility of creating a "citation profile" for a single, frequently-...
Analyzing information systems researchers' productivity and impacts: A perspective on the H index
Quantitative assessments of researchers' productivity and impacts are crucial for the information systems (IS) discipline. Motivated by its growing popularity and expanding use, we offer a perspective on the h index, which refers to the number of papers ...
Academic inbreeding and research productivity and impact in Australian law schools
This study compares the research productivity and impact of inbred and non-inbred faculty employed at Australian law schools. The sample consists of 429 academics, employed at 21 law schools. To measure research productivity and impact we use articles ...
Comments
Information & Contributors
Information
Published In
March 2024
98 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 February 2024
Published in CACM Volume 67, Issue 3
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 35,247Total Downloads
- Downloads (Last 12 months)35,247
- Downloads (Last 6 weeks)3,781
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderDigital Edition
View this article in digital edition.
Digital EditionMagazine Site
View this article on the magazine site (external)
Magazine SiteGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in