Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks
Pages 12 - 22
Abstract
Many modern code coverage tools track and report code coverage data generated from running tests during continuous integration. They report code coverage data through a variety of channels, including email, Slack, Mattermost, or through the web interface of social coding platforms such as GitHub. In fact, this ensemble of tools can be configured in such a way that the software engineer gets a failing status check when code coverage drops below a certain threshold. In this study, we broadly investigate the opinions and experience with code coverage tools through a survey among 279 software engineers whose projects use the Codecov coverage tool and bot. In particular, we are investigating why software engineers would ignore a failing status check caused by drop in code coverage. We observe that >80% of software engineers --- at least sometimes --- ignore these failing status checks, and we get insights into the main reasons why software engineers ignore these checks.
References
[1]
Maurício Aniche. 2022. Effective Software Testing: A Developer's Guide. Manning Publications.
[2]
Maurício Aniche, Felienne Hermans, and Arie van Deursen. 2019. Pragmatic software testing education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 414--420.
[3]
Maurício Aniche, Christoph Treude, and Andy Zaidman. 2022. How Developers Engineer Test Cases: An Observational Study. IEEE Trans. on Software Eng. 48, 12 (2022), 4925--4946.
[4]
Baris Ardiç and Andy Zaidman. 2023. Hey Teachers, Teach Those Kids Some Software Testing. In 5th IEEE/ACM International Workshop on Software Engineering Education for the Next Generation (SEENG@ICSE). IEEE, 9--16.
[5]
Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, and Andy Zaidman. 2019. Developer Testing in the IDE: Patterns, Beliefs, and Behavior. IEEE Trans. Software Eng. 45, 3 (2019).
[6]
Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman. 2015. When, how, and why developers (do not) test in their IDEs. In Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, 179--190.
[7]
Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Oops, my tests broke the build: an explorative analysis of Travis CI with GitHub. In Proceedings of the International Conference on Mining Software Repositories (MSR). IEEE, 356--367.
[8]
Casper Boone, Carolin Brandt, and Andy Zaidman. 2022. Fixing Continuous Integration Tests From Within the IDE With Contextual Information. In 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC). ACM, 287--297.
[9]
Cristian-Alexandru Botocan, Piyush Deshmukh, Pavlos Makridis, Jorge Romeu Huidobro, Mathanrajan Sundarrajan, Maurício Aniche, and Andy Zaidman. 2022. TestKnight: An Interactive Assistant to Stimulate Test Engineering. In IEEE/ACM International Conference on Software Engineering: Companion Proceedings (ICSE). ACM/IEEE, 222--226.
[10]
Carolin Brandt, Marco Castelluccio, Christian Holler, Jason Kratzer, Andy Zaidman, and Alberto Bacchelli. 2024. Mind the Gap: What Working With Developers on Fuzz Tests Taught Us About Coverage Gaps. In Proceedings of the International Conference on Software Engineering - Software Engineering In Practice (ICSE-SEIP). ACM.
[11]
Magiel Bruntink and Arie van Deursen. 2006. An empirical study into class testability. J. Syst. Softw. 79, 9 (2006), 1219--1232.
[12]
Codecov. 2022. The leading code coverage solution. https://about.codecov.io/
[13]
S. Elbaum, D. Gable, and G. Rothermel. 2001. The impact of software evolution on code coverage information. In Proceedings IEEE International Conference on Software Maintenance. ICSM 2001. IEEE, 170--179.
[14]
Michael Feathers. 2004. Working Effectively with Legacy Code. Pearson.
[15]
Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge.
[16]
Georgios Gousios, Margaret-Anne D. Storey, and Alberto Bacchelli. 2016. Work practices and challenges in pull-based development: the contributor's perspective. In Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, 285--296.
[17]
Georgios Gousios, Andy Zaidman, Margaret-Anne D. Storey, and Arie van Deursen. 2015. Work Practices and Challenges in Pull-Based Development: The Integrator's Perspective. In 37th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 358--368.
[18]
Z. Guo, T. Tan, S. Liu, X. Liu, W. Lai, Y. Yang, Y. Li, L. Chen, W. Dong, and Y. Zhou. 2023. Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities. IEEE Transactions on Software Engineering 49, 12 (2023), 5154--5188.
[19]
Hadi Hemmati. 2015. How Effective Are Code Coverage Criteria?. In 2015 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 151--156.
[20]
Michael Hilton, Jonathan Bell, and Darko Marinov. 2018. A Large-Scale Study of Test Coverage Evolution. In Proceedings of the International Conference on Automated Software Engineering (ASE). ACM, 53--63.
[21]
M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. 1994. Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In Proceedings of 16th International Conference on Software Engineering (ICSE). IEEE, 191--200.
[22]
Marko Ivanković, Goran Petrović, René Just, and Gordon Fraser. 2019. Code Coverage at Google. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 955--963.
[23]
Kush Jain, Goutamkumar Tulajappa Kalburgi, Claire Le Goues, and Alex Groce. 2023. Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts. CoRR abs/2309.02395 (2023). arXiv:2309.02395
[24]
Mehdi Jazayeri. 2004. The Education of a Software Engineer. In Proc. International Conference on Automated Software Engineering (ASE). IEEE, xviii--xxvii.
[25]
Ali Khatami and Andy Zaidman. 2023. Quality Assurance Awareness in Open Source Software Projects on GitHub. In Proceedings of the International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 174--185.
[26]
Amy J. Ko, Bryan Dosono, and Neeraja Duriseti. 2014. Thirty years of software problems in the news. In Int'l Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). ACM, 32--39.
[27]
Pavneet Singh Kochhar, David Lo, Julia Lawall, and Nachiappan Nagappan. 2017. Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects. IEEE Transactions on Reliability 66, 4 (2017), 1213--1228.
[28]
Chandra Maddila, Chetan Bansal, and Nachiappan Nagappan. 2019. Predicting Pull Request Completion Time: A Case Study on Large Scale Cloud Services. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 874--882.
[29]
Everton da S. Maldonado and Emad Shihab. 2015. Detecting and quantifying different types of self-admitted technical Debt. In 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD). 9--15.
[30]
Cosmin Marsavina, Daniele Romano, and Andy Zaidman. 2014. Studying FineGrained Co-evolution Patterns of Production and Test Code. In 14th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 195--204.
[31]
Scott Matteson. 2018. Report: Software failure caused 1.7 trillion in financial losses in 2017. https://www.techrepublic.com/article/report-software-failure-caused-1-7-trillion-in-inancial-losses-in-2017/
[32]
Harry McCracken. 2017. The Year That Software Bugs Ate The World. https://web.archive.org/web/20230307155438/https://www.fastcompany.com/40505226/the-year-that-software-bugs-ate-the-world
[33]
Greg Miller. 2006. A Scientist's Nightmare: Software Problem Leads to Five Retractions. Science 314, 5807 (2006), 1856--1857.
[34]
Leon Moonen, Arie van Deursen, Andy Zaidman, and Magiel Bruntink. 2008. On the Interplay Between Software Testing and Evolution and its Effect on Program Comprehension. In Software Evolution, Tom Mens and Serge Demeyer (Eds.). Springer, 173--202.
[35]
Akbar Siami Namin and James H. Andrews. 2009. The Influence of Size and Coverage on Test Suite Effectiveness. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis (ISSTA). ACM, 57--68.
[36]
Leandro Sales Pinto, Saurabh Sinha, and Alessandro Orso. 2012. Understanding Myths and Realities of Test-Suite Evolution. In Proceedings of the ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE). ACM, Article 33, 11 pages.
[37]
Igor Steinmacher, Gustavo Pinto, Igor Scaliante Wiese, and Marco A. Gerosa. 2018. Almost There: A Study on Quasi-Contributors in Open Source Software Projects. In Proceedings of the International Conference on Software Engineering (ICSE). ACM, 256--266.
[38]
Alexander Sterk. 2023. Exploring Code Coverage in Open-Source Development. Master's thesis. Delft University of Technology.
[39]
Alexander Sterk, Mairieli Wessel, Ali Hooten, and Andy Zaidman. 2023. Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks --- Appendix.
[40]
Anselm L Strauss and JM Corbin. 1998. Basics of qualitative research: Techniques and procedures for developing grounded theory.
[41]
Mark Swillus and Andy Zaidman. 2023. Sentiment overflow in the testing stack: Analyzing software testing posts on Stack Overflow. J. Syst. Softw. 205 (2023), 111804.
[42]
Shivashree Vysali. 2020. Enriching Code Coverage with Test Characteristics. Master's thesis. McGill University, 3480 Rue University, Montréal, QC, Canada.
[43]
Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A. Gerosa. 2020. What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. In Proceedings of the Brazilian Symposium on Software Engineering (SBES). ACM, 457--462.
[44]
Mairieli Wessel and Igor Steinmacher. 2020. The Inconvenient Side of Software Bots on Pull Requests. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSE Workshops). ACM, 51--55.
[45]
Andy Zaidman, Bart Van Rompaey, Serge Demeyer, and Arie van Deursen. 2008. Mining Software Repositories to Study Co-Evolution of Production & Test Code. In Proceedings of the International Conference on Software Testing, Verification, and Validation (ICST). IEEE, 220--229.
[46]
Andy Zaidman, Bart Van Rompaey, Arie van Deursen, and Serge Demeyer. 2011. Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empir. Softw. Eng. 16, 3 (2011), 325--364.
[47]
Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy. ACM Comput. Surv. 29, 4 (Dec. 1997), 366--427.
Recommendations
An Empirical Study on the Adequacy of Testing in Open Source Projects
APSEC '14: Proceedings of the 2014 21st Asia-Pacific Software Engineering Conference - Volume 01During software maintenance, testing is a crucial activity to ensure the quality of code as it evolves over time. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is an ...
An Empirical Study on Effects of Code Visibility on Code Coverage of Software Testing
AST '15: Proceedings of the 2015 IEEE/ACM 10th International Workshop on Automation of Software TestSoftware testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical ...
The effect of code coverage on fault detection under different testing profiles
A-MOST '05: Proceedings of the 1st international workshop on Advances in model-based testingSoftware testing is a key procedure to ensure high quality and reliability of software programs. The key issue in software testing is the selection and evaluation of different test cases. Code coverage has been proposed to be an estimator for testing ...
Comments
Information & Contributors
Information
Published In
April 2024
235 pages
ISBN:9798400705885
DOI:10.1145/3644032
- Chair:
- Francesca Lonetti,
- Proceedings Chair:
- Antonio Guerriero,
- Program Chair:
- Mehrdad Saadatmand,
- Program Co-chairs:
- Christof J. Budnik,
- Jenny Li
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- NWO
Conference
AST '24
Sponsor:
AST '24: 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)
April 15 - 16, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 86Total Downloads
- Downloads (Last 12 months)86
- Downloads (Last 6 weeks)17
Reflects downloads up to 23 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in