Computer Science > Software Engineering
[Submitted on 19 May 2022]
Title:Which bugs are missed in code reviews: An empirical study on SmartSHARK dataset
View PDFAbstract:In pull-based development systems, code reviews and pull request comments play important roles in improving code quality. In such systems, reviewers attempt to carefully check a piece of code by different unit tests. Unfortunately, sometimes they miss bugs in their review of pull requests, which lead to quality degradations of the systems. In other words, disastrous consequences occur when bugs are observed after merging the pull requests. The lack of a concrete understanding of these bugs led us to investigate and categorize them. In this research, we try to identify missed bugs in pull requests of SmartSHARK dataset projects. Our contribution is twofold. First, we hypothesized merged pull requests that have code reviews, code review comments, or pull request comments after merging, may have missed bugs after the code review. We considered these merged pull requests as candidate pull requests having missed bugs. Based on our assumption, we obtained 3,261 candidate pull requests from 77 open-source GitHub projects. After two rounds of restrictive manual analysis, we found 187 bugs missed in 173 pull requests. In the first step, we found 224 buggy pull requests containing missed bugs after merging the pull requests. Secondly, we defined and finalized a taxonomy that is appropriate for the bugs that we found and then found the distribution of bug categories after analysing those pull requests all over again. The categories of missed bugs in pull requests and their distributions are: semantic (51.34%), build (15.5%), analysis checks (9.09%), compatibility (7.49%), concurrency (4.28%), configuration (4.28%), GUI (2.14%), API (2.14%), security (2.14%), and memory (1.6%).
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.