Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support
Authors: 
Yibiao Yang, Maolin Sun, Yang Wang, Qingyang Li, 
Ming Wen, Yuming ZhouAuthors Info & Claims
Yibiao Yang, Maolin Sun, Yang Wang, Qingyang Li, Ming Wen, Yuming ZhouAuthors Info & ClaimsPages 670 - 681
Abstract
Ensuring the correctness of code coverage profilers is crucial, given the widespread adoption of code coverage for various software engineering tasks. Existing validation techniques, such as differential testing and metamorphic testing, have shown effectiveness in uncovering bugs in coverage profilers. However, these techniques have limitations as they primarily rely on homogeneous sources, i.e., different coverage profilers or the profilers themselves, for validation. In this paper, we propose Decov, a novel heterogeneous testing technique, to validate coverage profilers using the information provided by debuggers as a heterogeneous source. Coverage profilers record execution counts for each source line in the program, while debuggers monitor hit counts for each source line when running the program in debug mode. Our key insight is that the execution counts obtained from coverage profilers should align with the hit counts monitored by debuggers, without conflicts. Decov constructs multiple heterogeneous relations and utilizes them to uncover bugs in coverage profilers. Through experiments on Gcov and LLVM-cov, two widely used code coverage profilers, we discovered 21 new bug reports, with 19 of them directly confirmed by developers. Notably, developers have resolved 5 bugs in the latest trunk version. Decov serves as a simple yet effective coverage profiler validator and offers a complementary approach to existing techniques.
Supplementary Material
"Ensuring the correctness of code coverage profilers is of critical importance as code coverage has been widely adopted to facilitate substantial software engineering tasks. Recent studies have proposed to leverage differential testing and metamorphic testing to validate code coverage profilers, which have been shown to be effective in uncovering bugs for coverage profilers. However, existing techniques still have a bunch of drawbacks as they only use textit{homogeneous source}, i.e., different coverage profilers or the profilers themselves, to validate coverage profilers. In this paper, we propose Decov, a textit{heterogeneous testing} technique, to validate coverage profilers with the support of information provided by a textit{heterogeneous source}, which are the debuggers. The coverage profilers record the textit{execution count} for each line of a program during execution, while the debuggers monitor the textit{hit count} for each line of a program when running the program in debuggers. Our key insight is that the execution count obtained from the coverage profilers for any statement should not conflict with the hit count monitored by the debuggers. As such, Decov constructs a number of textit{heterogeneous relations} and utilizes them to uncover bugs in coverage profilers. The experiments on Gcov and LLVM-cov, two popular code coverage profilers, have led to 21 new bug reports and 18 of them are directly confirmed by developers. Of which, 5 bugs are fixed/resolved by developers in the latest trunk version. Decov is a simple yet effective coverage profiler validator that can be viewed as a complementary approach to existing ones."
- Download
- 50.74 MB
References
[1]
Sanjeev Kumar Aggarwal and M. Sarath Kumar. 2002. Debuggers for Programming Languages. The Compiler Design Handbook, 295–328.
[2]
Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage-based Greybox Fuzzing as Markov Chain. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi (Eds.). ACM, 1032–1043. https://doi.org/10.1145/2976749.2978428
[3]
Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage-based Greybox Fuzzing As Markov Chain. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16). ACM, New York, NY, USA. 1032–1043. isbn:978-1-4503-4139-4 https://doi.org/10.1145/2976749.2978428
[4]
Tsong Yueh Chen, Shing C Cheung, and Siu Ming Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong.
[5]
Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Trans. Softw. Eng. Methodol., 24, 2 (2014), Article 8, Dec., 42 pages. issn:1049-331X https://doi.org/10.1145/2685612
[6]
Markus Gaasedelen. 2017. Lighthouse - A Code Coverage Explorer for Reverse Engineers. https://github.com/gaasedelen/lighthouse
[7]
Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path Sensitive Fuzzing. In 2018 IEEE Symposium on Security and Privacy (SP). 679–696. https://doi.org/10.1109/SP.2018.00040
[8]
Milos Gligoric, Alex Groce, Chaoqiang Zhang, Rohan Sharma, Mohammad Amin Alipour, and Darko Marinov. 2015. Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites. ACM Trans. Softw. Eng. Methodol., 24, 4 (2015), Article 22, Sept., 33 pages. issn:1049-331X https://doi.org/10.1145/2660767
[9]
Patrice Godefroid, Michael Y. Levin, and David A. Molnar. 2008. Automated Whitebox Fuzz Testing. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2008, San Diego, California, USA, 10th February - 13th February 2008. The Internet Society. https://www.ndss-symposium.org/ndss2008/automated-whitebox-fuzz-testing/
[10]
Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019, Dongmei Zhang and Anders Møller (Eds.). ACM, 239–250. https://doi.org/10.1145/3293882.3330553
[11]
Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, Edinburgh, United Kingdom - June 09 - 11, 2014, Michael F. P. O’Boyle and Keshav Pingali (Eds.). ACM, 216–226. https://doi.org/10.1145/2594291.2594334
[12]
Vu Le, Chengnian Sun, and Zhendong Su. 2015. Randomized Stress-Testing of Link-Time Optimizers. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015). Association for Computing Machinery, New York, NY, USA. 327–337. isbn:9781450336208 https://doi.org/10.1145/2771783.2771785
[13]
Daniel Lehmann and Michael Pradel. 2018. Feedback-directed differential testing of interactive debuggers. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 610–620. https://doi.org/10.1145/3236024.3236037
[14]
David Lo, Siau-Cheng Khoo, Jiawei Han, and Chao Liu. 2011. Mining Software Specifications: Methodologies and Applications (1st ed.). CRC Press, Inc., Boca Raton, FL, USA. isbn:1439806268, 9781439806265
[15]
Stefan Nagy and Matthew Hicks. 2019. Full-Speed Fuzzing: Reducing Fuzzing Overhead through Coverage-Guided Tracing. In 2019 IEEE Symposium on Security and Privacy (SP). 787–802. https://doi.org/10.1109/SP.2019.00069
[16]
Sangmin Park, Richard W. Vuduc, and Mary Jean Harrold. 2010. Falcon: Fault Localization in Concurrent Programs. In Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering - Volume 1 (ICSE ’10). ACM, New York, NY, USA. 245–254. isbn:978-1-60558-719-6 https://doi.org/10.1145/1806799.1806838
[17]
Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding and Analyzing Compiler Warning Defects. In Proceedings of the 38th International Conference on Software Engineering (ICSE ’16). ACM, New York, NY, USA. 203–213. isbn:978-1-4503-3900-1 https://doi.org/10.1145/2884781.2884879
[18]
Maolin Sun, Yibiao Yang, Yang Wang, Ming Wen, Haoxiang Jia, and Yuming Zhou. 2023. SMT Solver Validation Empowered by Large Pre-trained Language Models. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Kirchberg, Luxembourg, September 11-15, 2023. IEEE.
[19]
Maolin Sun, Yibiao Yang, Ming Wen, Yongcong Wang, Yuming Zhou, and Hai Jin. 2023. Validating SMT Solvers via Skeleton Enumeration Empowered by Historical Bug-Triggering Inputs. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 69–81. https://doi.org/10.1109/ICSE48619.2023.00018
[20]
Sandro Tolksdorf, Daniel Lehmann, and Michael Pradel. 2019. Interactive Metamorphic Testing of Debuggers. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019). Association for Computing Machinery, New York, NY, USA. 273–283. isbn:9781450362245 https://doi.org/10.1145/3293882.3330567
[21]
Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2017. Skyfire: Data-Driven Seed Generation for Fuzzing. In 2017 IEEE Symposium on Security and Privacy (SP). 579–594. https://doi.org/10.1109/SP.2017.23
[22]
Ming Wen, Junjie Chen, Yongqiang Tian, Rongxin Wu, Dan Hao, Shi Han, and Shing-Chi Cheung. 2021. Historical Spectrum Based Fault Localization. IEEE Trans. Software Eng., 47, 11 (2021), 2348–2368. https://doi.org/10.1109/TSE.2019.2948158
[23]
Ming Wen, Junjie Chen, Rongxin Wu, Dan Hao, and Shing-Chi Cheung. 2018. Context-aware patch generation for better automated program repair. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 1–11. https://doi.org/10.1145/3180155.3180233
[24]
Mingyuan Wu, Ling Jiang, Jiahong Xiang, Yanwei Huang, Heming Cui, Lingming Zhang, and Yuqun Zhang. 2022. One Fuzzing Strategy to Rule Them All. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. ACM, 1634–1645. https://doi.org/10.1145/3510003.3510174
[25]
Xiaoyuan Xie, Joshua Wing Kei Ho, Christian Murphy, Gail E. Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. J. Syst. Softw., 84, 4 (2011), 544–558. https://doi.org/10.1016/j.jss.2010.11.920
[26]
Yibiao Yang, Yanyan Jiang, Zhiqiang Zuo, Yang Wang, Hao Sun, Hongmin Lu, Yuming Zhou, and Baowen Xu. 2019. Automatic Self-Validation for Code Coverage Profilers. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE ’19). IEEE Press, 79–90. isbn:9781728125084 https://doi.org/10.1109/ASE.2019.00018
[27]
Yibiao Yang, Yuming Zhou, Hao Sun, Zhendong Su, Zhiqiang Zuo, Lei Xu, and Baowen Xu. 2019. Hunting for bugs in code coverage tools via randomized differential testing. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.). IEEE / ACM, 488–498. https://doi.org/10.1109/ICSE.2019.00061
[28]
Yang Yibiao, Sun Maolin, Wang Yang, Li Qingyang, Wen Ming, and Zhou Yuming. 2023. ESEC/FSE 2023 Artifact for "Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support". https://doi.org/10.5281/zenodo.8275866
[29]
S. Yoo and M. Harman. 2012. Regression Testing Minimization, Selection and Prioritization: A Survey. Softw. Test. Verif. Reliab., 22, 2 (2012), March, 67–120. issn:0960-0833 https://doi.org/10.1002/stv.430
[30]
Zhi Quan Zhou and Liqun Sun. 2019. Metamorphic Testing of Driverless Cars. Commun. ACM, 62, 3 (2019), Feb., 61–67. issn:0001-0782 https://doi.org/10.1145/3241979
[31]
Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022. Fuzzing: A Survey for Roadmap. ACM Comput. Surv., 54, 11s (2022), Article 230, sep, 36 pages. issn:0360-0300 https://doi.org/10.1145/3512345
[32]
Zhiqiang Zuo and Siau-Cheng Khoo. 2013. Mining Dataflow Sensitive Specifications. In Formal Methods and Software Engineering - 15th International Conference on Formal Engineering Methods, ICFEM 2013, Queenstown, New Zealand, October 29 - November 1, 2013, Proceedings, Lindsay Groves and Jing Sun (Eds.) (Lecture Notes in Computer Science, Vol. 8144). Springer, 36–52. https://doi.org/10.1007/978-3-642-41202-8_4
Index Terms
- Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support
Recommendations
Automatic self-validation for code coverage profilers
ASE '19: Proceedings of the 34th IEEE/ACM International Conference on Automated Software EngineeringCode coverage as the primitive dynamic program behavior information, is widely adopted to facilitate a rich spectrum of software engineering tasks, such as testing, fuzzing, debugging, fault detection, reverse engineering, and program understanding. ...
Hunting for bugs in code coverage tools via randomized differential testing
ICSE '19: Proceedings of the 41st International Conference on Software EngineeringReliable code coverage tools are critically important as it is heavily used to facilitate many quality assurance activities, such as software testing, fuzzing, and debugging. However, little attention has been devoted to assessing the reliability of code ...
The effect of code coverage on fault detection under different testing profiles
A-MOST '05: Proceedings of the 1st international workshop on Advances in model-based testingSoftware testing is a key procedure to ensure high quality and reliability of software programs. The key issue in software testing is the selection and evaluation of different test cases. Code coverage has been proposed to be an estimator for testing ...
Comments
Information & Contributors
Information
Published In
November 2023
2215 pages
ISBN:9798400703270
DOI:10.1145/3611643
- General Chair:
- Satish Chandra,
- Program Chairs:
- Kelly Blincoe,
- Paolo Tonella
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 November 2023
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province
- CCF-Huawei Populus euphratica Fund
Conference
ESEC/FSE '23
Sponsor:
ESEC/FSE '23: 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
December 3 - 9, 2023
CA, San Francisco, USA
Acceptance Rates
Overall Acceptance Rate 112 of 543 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 87Total Downloads
- Downloads (Last 12 months)62
- Downloads (Last 6 weeks)6
Reflects downloads up to 31 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in