An Empirical Study on Code Coverage of Performance Testing
Authors: 
Muhammad Imran, Vittorio Cortellessa, 
Davide Di Ruscio, Riccardo Rubei, 
Luca TrainiAuthors Info & Claims
Muhammad Imran, Vittorio Cortellessa, Davide Di Ruscio, Riccardo Rubei, Luca TrainiAuthors Info & ClaimsPages 48 - 57
Abstract
Performance testing aims to ensure the operational efficiency of software systems. However, many factors influencing the efficacy and adoption of performance tests in practice are not yet fully understood. For instance, while code coverage is widely regarded as a key quality metric for evaluating the efficacy of functional testing suites, there is limited knowledge about the types and levels of coverage that performance tests specifically achieve. Another important factor, often perceived as a barrier to the broader adoption of performance tests yet remaining relatively unexplored, is their extended execution time. In this paper, we analyze the performance testing suites of 28 open-source systems to study (i) the magnitude of their code coverage, and (ii) their execution time. Our analysis shows that performance tests achieve significantly lower code coverage than functional tests, as expected, and it highlights a significant trade-off between coverage and execution time. Our results also suggest, in perspective, that automated test generation methods might not ensure affordable performance testing due to the associated time cost. This finding poses new challenges in the field of performance test generation.
References
[1]
Hammam M. Alghmadi, Mark D. Syer, Weiyi Shang, and Ahmed E. Hassan. 2016. An Automated Approach for Recommending When to Stop Performance Tests. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). 279–289. https://doi.org/10.1109/ICSME.2016.46
[2]
Wasim Alsaqaf, Maya Daneva, and Roel Wieringa. 2019. Quality requirements challenges in the context of large-scale distributed agile: An empirical study. Information and Software Technology 110 (2019), 39 – 55.
[3]
Tiago L Alves and Joost Visser. 2009. Static estimation of test coverage. In 2009 Ninth IEEE International Working Conference on Source Code Analysis and Manipulation. IEEE, 55–64.
[4]
Thomas Bach, Artur Andrzejak, Ralf Pannemans, and David Lo. 2017. The impact of coverage on bug density in a large industrial software project. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 307–313.
[5]
Woubshet Behutiye, Pertti Karhapää, Lidia López, Xavier Burgués, Silverio Martínez-Fernández, Anna Maria Vollmer, Pilar Rodríguez, Xavier Franch, and Markku Oivo. 2020. Management of quality requirements in agile and rapid software development: A systematic mapping study. Information and Software Technology 123 (2020), 106225.
[6]
Jake Brutlag. 2009. Google AI Blog: Speed matters. https://ai.googleblog.com/2009/06/speed-matters.html
[7]
Jinfu Chen, Zishuo Ding, Yiming Tang, Mohammed Sayagh, Heng Li, Bram Adams, and Weiyi Shang. 2023. IoPV: On Inconsistent Option Performance Variations. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(ESEC/FSE 2023). ACM, 845–857. https://doi.org/10.1145/3611643.3616319
[8]
Jinfu Chen, Weiyi Shang, and Emad Shihab. 2022. PerfJIT: Test-Level Just-in-Time Prediction for Performance Regression Introducing Commits. IEEE Transactions on Software Engineering 48, 5 (2022), 1529–1544.
[9]
Michael L. Collard, Michael J. Decker, and Jonathan I. Maletic. 2011. Lightweight Transformation and Fact Extraction with the srcML Toolkit. In 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation. 173–184. https://doi.org/10.1109/SCAM.2011.19
[10]
Zishuo Ding, Jinfu Chen, and Weiyi Shang. 2020. Towards the use of the readily available tests from the release pipeline as performance tests: are we there yet?. In ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 1435–1446. https://doi.org/10.1145/3377811.3380351
[11]
Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code Coverage for Suite Evaluation by Developers. In Proceedings of the 36th International Conference on Software Engineering(ICSE 2014). ACM, 72–82.
[12]
Sen He, Glenna Manns, John Saunders, Wei Wang, Lori Pollock, and Mary Lou Soffa. 2019. A Statistics-Based Performance Testing Methodology for Cloud Applications. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(ESEC/FSE 2019). ACM, 188–199.
[13]
Muhammad Imran, Vittorio Cortellessa, Davide Di Ruscio, Riccardo Rubei, and Luca Traini. 2023. An Empirical Study on Performance Test Coverage-Replication Package. https://github.com/SpencerLabAQ/replication-package_performance-test-coverage
[14]
Laura Inozemtseva and Reid Holmes. 2014. Coverage is Not Strongly Correlated with Test Suite Effectiveness. In Proceedings of the 36th International Conference on Software Engineering(ICSE 2014). ACM, 435–445.
[15]
Marko Ivanković, Goran Petrović, René Just, and Gordon Fraser. 2019. Code Coverage at Google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(ESEC/FSE 2019). ACM, 955–963.
[16]
Mostafa Jangali, Yiming Tang, Niclas Alexandersson, Philipp Leitner, Jinqiu Yang, and Weiyi Shang. 2023. Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit Tests. IEEE Transactions on Software Engineering 49, 4 (2023), 1704–1725. https://doi.org/10.1109/TSE.2022.3188005
[17]
Zhen Ming Jiang and Ahmed E. Hassan. 2015. A Survey on Load Testing of Large-Scale Software Systems. IEEE Transactions on Software Engineering 41, 11 (2015), 1091–1118. https://doi.org/10.1109/TSE.2015.2445340
[18]
Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. 2012. Understanding and Detecting Real-World Performance Bugs. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation(PLDI ’12). ACM, 77–88. https://doi.org/10.1145/2254064.2254075
[19]
Tomas Kalibera and Richard Jones. 2013. Rigorous Benchmarking in Reasonable Time. In Proceedings of the 2013 International Symposium on Memory Management(ISMM ’13). ACM, 63–74. https://doi.org/10.1145/2491894.2464160
[20]
Christoph Laaber, Harald C. Gall, and Philipp Leitner. 2021. Applying test case prioritization to software microbenchmarks. Empirical Software Engineering 26, 6 (2021), 133. https://doi.org/10.1007/s10664-021-10037-x
[21]
Christoph Laaber and Philipp Leitner. 2018. An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment. ACM.
[22]
Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner. 2020. Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(ESEC/FSE 2020). ACM, 989–1001.
[23]
Philipp Leitner and Cor-Paul Bezemer. 2017. An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering(ICPE ’17). ACM, 373–384. https://doi.org/10.1145/3030207.3030213
[24]
Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, and Robert Ricci. 2018. Taming Performance Variability. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 409–425.
[25]
Alejandro Mazuera-Rozo, Catia Trubiani, Mario Linares-Vásquez, and Gabriele Bavota. 2020. Investigating types and survivability of performance bugs in mobile apps. Empirical Software Engineering 25 (2020), 1644–1686.
[26]
Steve Olenski. 2016. Why Brands Are Fighting Over Milliseconds. https://www.forbes.com/sites/steveolenski/2016/11/10/why-brands-are-fighting-over-milliseconds/
[27]
Paul Piwowarski, Mitsuru Ohba, and Joe Caruso. 1993. Coverage Measurement Experience during Function Test. In Proceedings of the 15th International Conference on Software Engineering(ICSE ’93). IEEE Computer Society Press, 287–301.
[28]
Hazem Samoaa and Philipp Leitner. 2021. An Exploratory Study of the Impact of Parameterization on JMH Measurement Results in Open-Source Projects. In Proceedings of the ACM/SPEC International Conference on Performance Engineering(ICPE ’21). ACM, 213–224. https://doi.org/10.1145/3427921.3450243
[29]
Marija Selakovic and Michael Pradel. 2016. Performance issues and optimizations in javascript: an empirical study. In Proceedings of the 38th International Conference on Software Engineering. 61–72.
[30]
Luca Traini. 2022. Exploring Performance Assurance Practices and Challenges in Agile Software Development: An Ethnographic Study. Empirical Software Engineering 27, 3 (2022), 74. https://doi.org/10.1007/s10664-021-10069-3
[31]
Luca Traini, Vittorio Cortellessa, Daniele Di Pompeo, and Michele Tucci. 2022. Towards effective assessment of steady state performance in Java software: are we there yet?Empirical Software Engineering 28, 1 (2022), 13. https://doi.org/10.1007/s10664-022-10247-x
[32]
Luca Traini, Daniele Di Pompeo, Michele Tucci, Bin Lin, Simone Scalabrino, Gabriele Bavota, Michele Lanza, Rocco Oliveto, and Vittorio Cortellessa. 2021. How Software Refactoring Impacts Execution Time. ACM Trans. Softw. Eng. Methodol. 31, 2, Article 25 (dec 2021), 23 pages. https://doi.org/10.1145/3485136
[33]
Andrew Walker, Michael Coffey, Pavel Tisnovsky, and Tomas Cerny. 2020. On Limitations of Modern Static Analysis Tools. In Information Science and Applications, Kuinam J. Kim and Hye-Young Kim (Eds.). Springer Singapore, 577–586.
[34]
Elaine J. Weyuker. 2000. Experience with performance testing of software systems: issues, an approach, and case study. IEEE Transactions on Software Engineering 26, 12 (2000), 1147 – 1156. https://doi.org/10.1109/32.888628
[35]
S. Yoo and M. Harman. 2012. Regression Testing Minimization, Selection and Prioritization: A Survey. Softw. Test. Verif. Reliab. 22, 2 (mar 2012), 67–120.
[36]
Shahed Zaman, Bram Adams, and Ahmed E. Hassan. 2012. A Qualitative Study on Performance Bugs. In Proceedings of the 9th IEEE Working Conference on Mining Software Repositories(MSR ’12). IEEE Press, 199–208.
[37]
Hongyu Zhai, Casey Casalnuovo, and Prem Devanbu. 2019. Test coverage in python programs. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 116–120.
[38]
Yutong Zhao, Lu Xiao, Andre B. Bondi, Bihuan Chen, and Yang Liu. 2023. A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source Projects. IEEE Transactions on Software Engineering 49, 2 (2023), 924–946.
Index Terms
- An Empirical Study on Code Coverage of Performance Testing
Recommendations
An Empirical Study on Effects of Code Visibility on Code Coverage of Software Testing
AST '15: Proceedings of the 2015 IEEE/ACM 10th International Workshop on Automation of Software TestSoftware testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical ...
An empirical study on effects of code visibility on code coverage of software testing
AST '15: Proceedings of the 10th International Workshop on Automation of Software TestSoftware testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical ...
The effect of code coverage on fault detection under different testing profiles
A-MOST '05: Proceedings of the 1st international workshop on Advances in model-based testingSoftware testing is a key procedure to ensure high quality and reliability of software programs. The key issue in software testing is the selection and evaluation of different test cases. Code coverage has been proposed to be an estimator for testing ...
Comments
Information & Contributors
Information
Published In
June 2024
728 pages
ISBN:9798400717017
DOI:10.1145/3661167
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Ministero dell'Università e della Ricerca
Conference
EASE 2024
EASE 2024: 28th International Conference on Evaluation and Assessment in Software Engineering
June 18 - 21, 2024
Salerno, Italy
Acceptance Rates
Overall Acceptance Rate 71 of 232 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 143Total Downloads
- Downloads (Last 12 months)143
- Downloads (Last 6 weeks)43
Reflects downloads up to 17 Oct 2024
Other Metrics
Citations
Cited By
View allView Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in