AI-Assisted Assessment of Coding Practices in Modern Code Review
Authors: 
Manushree Vijayvergiya, Małgorzata Salawa, Ivan Budiselić, Dan Zheng, Pascal Lamblin, Marko Ivanković, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran Petrović, Daniel Tarlow, Petros Maniatis, René JustAuthors Info & Claims
Manushree Vijayvergiya, Małgorzata Salawa, Ivan Budiselić, Dan Zheng, Pascal Lamblin, Marko Ivanković, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran Petrović, Daniel Tarlow, Petros Maniatis, René JustAuthors Info & ClaimsPages 85 - 93
Abstract
Modern code review is a process in which an incremental code contribution made by a code author is reviewed by one or more peers before it is committed to the version control system. An important element of modern code review is verifying that code contributions adhere to best practices. While some of these best practices can be automatically verified, verifying others is commonly left to human reviewers. This paper reports on the development, deployment, and evaluation of AutoCommenter, a system backed by a large language model that automatically learns and enforces coding best practices. We implemented AutoCommenter for four programming languages (C++, Java, Python, and Go) and evaluated its performance and adoption in a large industrial setting. Our evaluation shows that an end-to-end system for learning and enforcing coding best practices is feasible and has a positive impact on the developer workflow. Additionally, this paper reports on the challenges associated with deploying such a system to tens of thousands of developers and the corresponding lessons learned.
References
[1]
2024. Google Style Guides. https://google.github.io/styleguide/ Accessed: 2024-03-15
[2]
2024. Linux kernel coding style. https://www.kernel.org/doc/html/v4.10/process/coding-style.html Accessed: 2024-03-15
[3]
2024. PEP 8 – Style Guide for Python Code. https://peps.python.org/pep-0008/ Accessed: 2024-03-15
[4]
2024. Rust Style Guide. https://doc.rust-lang.org/nightly/style-guide/ Accessed: 2024-03-15
[5]
Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE). 712–721. https://doi.org/10.1109/ICSE.2013.6606617
[6]
Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). 1, 470–481.
[7]
Zimin Chen, Mał gorzata Salawa, Manushree Vijayvergiya, Goran Petrović, Marko Ivanković, and René Just. 2023. MuRS: Mutant Ranking and Suppression using Identifier Templates. In Proceedings of the Symposium on the Foundations of Software Engineering (FSE). 1798–1808.
[8]
M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Systems Journal, 15, 3 (1976), 182–211. https://doi.org/10.1147/sj.153.0182
[9]
Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. 2024. Resolving Code Review Comments with Machine Learning. In International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).
[10]
Sarah Heckman and Laurie Williams. 2011. A systematic literature review of actionable alert identification techniques for automated static code analysis. Information and Software Technology, 53, 4 (2011), 363–387. issn:0950-5849 https://doi.org/10.1016/j.infsof.2010.12.007 Special section: Software Engineering track of the 24th Annual Symposium on Applied Computing
[11]
Yang Hong, Chakkrit Tantithamthavorn, Patanamon Thongtanunam, and Aldeida Aleti. 2022. Commentfinder: a simpler, faster, more accurate code review comments recommendation. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE). 507–519.
[12]
Marko Ivanković, Goran Petrović, René Just, and Gordon Fraser. 2019. Code Coverage at Google. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE). 955–963.
[13]
Marko Ivanković, Goran Petrović, Yana Kulizhskaya, Mateusz Lewko, Luka Kalinovčić, René Just, and Gordon Fraser. 2024. Productive Coverage: Improving the Actionability of Code Coverage. In International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).
[14]
Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs? In 2013 35th International Conference on Software Engineering (ICSE). 672–681.
[15]
Stephen C Johnson. 1977. Lint, a C program checker. Bell Telephone Laboratories Murray Hill.
[16]
Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. 2022. Auger: Automatically generating review comments with pre-training models. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE). 1009–1021.
[17]
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 1035–1047. isbn:9781450394130 https://doi.org/10.1145/3540250.3549081
[18]
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2023. Please fix this mutant: How do developers resolve mutants surfaced during code review? In International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 150–161.
[19]
Rachel Potvin and Josh Levenberg. 2016. Why Google Stores Billions of Lines of Code in a Single Repository. Communications of the ACM (CACM), 59 (2016), 78–87. http://dl.acm.org/citation.cfm?id=2854146
[20]
Peter Rigby, Brendan Cleary, Frederic Painchaud, Margaret-Anne Storey, and Daniel German. 2012. Contemporary Peer Review in Action: Lessons from Open Source Development. IEEE Software, 29, 6 (2012), 56–61. https://doi.org/10.1109/MS.2012.24
[21]
Peter C. Rigby and Christian Bird. 2013. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA. 202–212. isbn:9781450322379 https://doi.org/10.1145/2491411.2491444
[22]
Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, and Afroz Mohiuddin. 2023. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24, 377 (2023), 1–8.
[23]
Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern Code Review: A Case Study at Google. In International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 181–190.
[24]
Patanamon Thongtanunam, Chanathip Pornprasit, and Chakkrit Tantithamthavorn. 2022. Autotransform: Automated code transformation to support modern code review process. In Proceedings of the International Conference on Software Engineering (ICSE). 237–248.
[25]
Rosalia Tufano, Ozren Dabić, Antonio Mastropaolo, Matteo Ciniselli, and Gabriele Bavota. 2024. Code Review Automation: Strengths and Weaknesses of the State of the Art. IEEE Transactions on Software Engineering (TSE).
[26]
Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022. Using pre-trained models to boost code review automation. In Proceedings of the International Conference on Software Engineering (ICSE). 2291–2302.
[27]
Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Harald C Gall, and Andy Zaidman. 2020. How developers engage with static analysis tools in different contexts. Empirical Software Engineering, 25 (2020), 1419–1457.
[28]
T. Winters, T. Manshreck, and H. Wright. 2020. Software Engineering at Google: Lessons Learned from Programming Over Time. O’Reilly Media. isbn:9781492082798 lccn:2021443590 https://books.google.ch/books?id=TyIrywEACAAJ
Index Terms
- AI-Assisted Assessment of Coding Practices in Modern Code Review
Recommendations
Code review practices for refactoring changes: an empirical study on OpenStack
MSR '22: Proceedings of the 19th International Conference on Mining Software RepositoriesModern code review is a widely used technique employed in both industrial and open-source projects to improve software quality, share knowledge, and ensure adherence to coding standards and guidelines. During code review, developers may discuss ...
Understanding code snippets in code reviews: a preliminary study of the OpenStack community
ICPC '22: Proceedings of the 30th IEEE/ACM International Conference on Program ComprehensionCode review is a mature practice for software quality assurance in software development with which reviewers check the code that has been committed by developers, and verify the quality of code. During the code review discussions, reviewers and ...
Refactoring practices in the context of modern code review: an industrial case study at Xerox
ICSE-SEIP '21: Proceedings of the 43rd International Conference on Software Engineering: Software Engineering in PracticeModern code review is a common and essential practice employed in both industrial and open-source projects to improve software quality, share knowledge, and ensure conformance with coding standards. During code review, developers may inspect and discuss ...
Comments
Information & Contributors
Information
Published In
July 2024
182 pages
ISBN:9798400706851
DOI:10.1145/3664646
- General Chairs:
- Bram Adams,
- Thomas Zimmermann,
- Ipek Ozkaya,
- Program Chairs:
- Dayi Lin,
- Jie M. Zhang
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
AIware '24
Sponsor:
AIware '24: 1st ACM International Conference on AI-Powered Software
July 15 - 16, 2024
Porto de Galinhas, Brazil
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 144Total Downloads
- Downloads (Last 12 months)144
- Downloads (Last 6 weeks)108
Reflects downloads up to 01 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in