research-article

FSX: fine-grained incremental unit test generation for C/C++ programs

Authors:

Hiroaki Yoshida,

Susumu Tokumoto,

Mukul R. Prasad,

Indradeep Ghosh,

Tadahiro UeharaAuthors Info & Claims

ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Pages 106 - 117

https://doi.org/10.1145/2931037.2931055

Published: 18 July 2016 Publication History

Abstract

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, we propose a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. The technique uses iterative, incremental refinement of test-drivers and symbolic execution, guided by a diagnostics engine. The diagnostics engine works off a novel precise and efficient byte-level dynamic dependence analysis built using Reduced Ordered Binary Decision Diagrams (ROBDDs). We present a tool FSX implementing this technique and evaluate it under two practical use-cases of incremental unit test generation, on five revisions of the open-source software iPerf, as well as on 3 large subjects, comprising more than 60 thousand lines of code, from in-house commercial network products. The evaluation shows that FSX can generate high-quality unit tests on large industrial software while minimizing the maintenance cost of the overall test-suite.

References

[1]

CLANG, http://clang.llvm.org/.

[2]

CUDD: CU Decision Diagram Package, http://vlsi.colorado.edu/~fabio/CUDD/.

[3]

Free Software Foundation, Inc. How to Minimize Test Cases for Bugs, https://gcc.gnu.org/bugs/minimize.html.

[4]

googletest - Google C++ Testing Framework, http://code.google.com/p/googletest/.

[5]

How SQLite Is Tested, http://www.sqlite.org/testing.html.

[6]

How to submit an LLVM bug report, http://llvm.org/docs/HowToSubmitABug.html.

[7]

iPerf - The network bandwidth measurement tool, https://iperf.fr/.

[8]

LLVM, http://www.llvm.org/.

[9]

Mozilla Developer Network. Reducing testcases, https://developer.mozilla.org/en-US/docs/Mozilla/QA/ Reducing_testcases.

[10]

PIT Mutation Testing, http://pitest.org/.

[11]

SQLite, http://www.sqlite.org/.

[12]

Webkit. Test Case Reduction, https://webkit.org/test-case-reduction/.

[13]

B. Beizer. Software Testing Techniques. John Wiley & Sons, Inc., New York, NY, USA, 1990.

Digital Library

[14]

R. E. Bryant. Symbolic boolean manipulation with ordered binary-decision diagrams. ACM Computing Surveys, 24(3):293–318, Sept. 1992.

Digital Library

[15]

C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 209–224, 2008.

Digital Library

[16]

C. Cadar, P. Godefroid, S. Khurshid, C. S. Păsăreanu, K. Sen, N. Tillmann, and W. Visser. Symbolic execution for software testing in practice: Preliminary assessment. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pages 1066–1071, New York, NY, USA, 2011. ACM.

Digital Library

[17]

J. Campos, A. Arcuri, G. Fraser, and R. Abreu. Continuous test generation: Enhancing continuous integration with automated test generation. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, pages 55–66, New York, NY, USA, 2014. ACM.

Digital Library

[18]

J. Chang and D. Richardson. Structural specification-based testing: Automated support and experimental evaluation. In O. Nierstrasz and M. Lemoine, editors, Software Engineering ESEC/FSE ’99, volume 1687 of Lecture Notes in Computer Science, pages 285–302. Springer Berlin Heidelberg, 1999.

Digital Library

[19]

T. Y. Chen and R. Merkel. Quasi-random testing. Reliability, IEEE Transactions on, 56(3):562–568, Sept 2007.

[20]

L. De Moura and N. Bjørner. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS/ETAPS’08, pages 337–340, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[21]

R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE Trans. Softw. Eng., 17(9):900–910, Sept. 1991.

Digital Library

[22]

G. Fraser and A. Arcuri. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, 2011.

Digital Library

[23]

G. Fraser and A. Zeller. Mutation-driven Generation of Oracles and Unit Tests. IEEE Trans. on Software Engineering, 2012.

Digital Library

[24]

V. Ganesh and D. L. Dill. A decision procedure for bit-vectors and arrays. In Proceedings of the 19th International Conference on Computer Aided Verification, CAV’07, pages 519–531, Berlin, Heidelberg, 2007. Springer-Verlag.

Digital Library

[25]

I. Ghosh, N. Shafiei, G. Li, and W.-F. Chiang. JST: an automatic test generation tool for industrial Java applications with strings. In 35th International Conference on Software Engineering (ICSE), pages 992–1001, 2013.

Digital Library

[26]

P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In PLDI, 2005.

Digital Library

[27]

P. Godefroid, M. Y. Levin, and D. A. Molnar. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2008, San Diego, California, USA, 10th February - 13th February 2008, 2008.

[28]

J. B. Goodenough and S. L. Gerhart. Toward a Theory of Test Data Selection. IEEE Tran. on Software Engineering, 1975.

Digital Library

[29]

Y. Kim, Y. Kim, T. Kim, G. Lee, Y. Jang, and M. Kim. Automated unit testing of large industrial embedded software using concolic testing. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 519–528, Nov 2013.

Digital Library

[30]

Y. Kim, Z. Zu, M. Kim, M. Cohen, and G. Rothermel. Hybrid directed test suite augmentation: An interleaving framework. In Software Testing, Verification and Validation (ICST), 2014 IEEE Seventh International Conference on, pages 263–272, March 2014.

Digital Library

[31]

C. Lattner and V. Adve. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Int’l Symposium on Code Generation and Optimization (CGO), 2004.

Digital Library

[32]

G. Li, E. Andreasen, and I. Ghosh. Symjs: Automatic symbolic testing of javascript web applications. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pages 449–459, New York, NY, USA, 2014. ACM.

Digital Library

[33]

G. Li, I. Ghosh, and S. P. Rajan. KLOVER : A symbolic execution and automatic test generation tool for C++ programs. In 23rd International Conference on Computer Aided Verification (CAV), pages 609–615, 2011.

Digital Library

[34]

R. Majumdar and R.-G. Xu. Reducing test inputs using information partitions. In Proceedings of the 21st International Conference on Computer Aided Verification, CAV ’09, pages 555–569, Berlin, Heidelberg, 2009. Springer-Verlag.

Digital Library

[35]

P. McMinn. Search-based software test data generation: A survey: Research articles. Softw. Test. Verif. Reliab., 14(2):105–156, June 2004.

[36]

C. Nguyen, H. Yoshida, M. R. Prasad, I. Ghosh, and K. Sen. Generating succinct test cases using don’t care analysis. In Proceedings of the 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST’15, April 2015.

[37]

C. Pacheco and M. D. Ernst. Randoop: Feedback-directed random testing for java. In Companion to the 22Nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion, OOPSLA ’07, pages 815–816, New York, NY, USA, 2007. ACM.

Digital Library

[38]

S. Person, G. Yang, N. Rungta, and S. Khurshid. Directed incremental symbolic execution. In Proceedings of the 32Nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’11, pages 504–515, New York, NY, USA, 2011. ACM.

Digital Library

[39]

C. S. Păsăreanu and W. Visser. A survey of new trends in symbolic execution for software testing and analysis. Int. J. Softw. Tools Technol. Transf., 11(4):339–353, Oct. 2009.

[40]

C. S. Pˇasˇareanu and N. Rungta. Symbolic PathFinder: symbolic execution of Java bytecode. In 25th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 179–180, 2010.

Digital Library

[41]

Research Triangle Institute. The Economic Impacts of Inadequate Infrastructure for Software Testing. NIST Planning Report 02-3, National Institute of Standards and Technology, May 2002.

[42]

R. Santelices, P. K. Chittimalli, T. Apiwattanapong, A. Orso, and M. J. Harrold. Test-suite augmentation for evolving software. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, ASE ’08, pages 218–227, Washington, DC, USA, 2008. IEEE Computer Society.

Digital Library

[43]

K. Sen and G. Agha. CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-checking Tools. In Proceedings of the 18th International Conference on Computer Aided Verification, CAV’06, pages 419–423, Berlin, Heidelberg, 2006. Springer-Verlag.

Digital Library

[44]

K. Sen, D. Marinov, and G. Agha. CUTE: A Concolic Unit Testing Engine for C. In ESEC-FSE, 2005.

Digital Library

[45]

D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, J. Newsome, P. Poosankam, and P. Saxena. Bitblaze: A new approach to computer security via binary analysis. In Proceedings of the 4th International Conference on Information Systems Security, ICISS ’08, pages 1–25, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[46]

K. Taneja, T. Xie, N. Tillmann, and J. de Halleux. eXpress: Guided path exploration for efficient regression test generation. In Proceedings of the 2011 International Symposium on Software Testing and Analysis, ISSTA ’11, pages 1–11, New York, NY, USA, 2011. ACM.

Digital Library

[47]

N. Tillmann and J. de Halleux. PEX – White Box Test Generation for .NET. In Tests and Proofs, 2008.

Digital Library

[48]

N. Tillmann and W. Schulte. Parameterized Unit Tests. In ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 253–262, 2005.

Digital Library

[49]

Z. Xu, M. B. Cohen, and G. Rothermel. Factors affecting the use of genetic algorithms in test suite augmentation. In Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, GECCO ’10, pages 1365–1372, New York, NY, USA, 2010. ACM.

Digital Library

[50]

Z. Xu, Y. Kim, M. Kim, G. Rothermel, and M. B. Cohen. Directed test suite augmentation: Techniques and tradeoffs. In Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE ’10, pages 257–266, New York, NY, USA, 2010. ACM.

Digital Library

[51]

Z. Xu and G. Rothermel. Directed test suite augmentation. In 16th Asia-Pacific Software Engineering Conference, APSEC 2009, 1-3 December 2009, Batu Ferringhi, Penang, Malaysia, pages 406–413, 2009.

Digital Library

[52]

G. Yang, C. S. Păsăreanu, and S. Khurshid. Memoized symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA 2012, pages 144–154, New York, NY, USA, 2012. ACM.

Digital Library

Cited By

Wanjiru Bvan Bommel PHiemstra D(2024)Sensitivity of Automated SQL Grading in Computer Science CoursesProceedings of the Third International Conference on Innovations in Computing Research (ICR’24)10.1007/978-3-031-65522-7_26(283-299)Online publication date: 1-Aug-2024
https://doi.org/10.1007/978-3-031-65522-7_26
Aşik SYayan U(2023)Generating Python Mutants From Bug Fixes Using Neural Machine TranslationIEEE Access10.1109/ACCESS.2023.330269511(85678-85693)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3302695
Herlim RKim YKim M(2022)CITRUS: Automated Unit Testing Tool for Real-world C++ Programs2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00046(400-410)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00046
Show More Cited By

Index Terms

FSX: fine-grained incremental unit test generation for C/C++ programs
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

FSX: a tool for fine-grained incremental unit test generation for C/C++ programs
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. ...
Fine-grained test minimization
ICSE '18: Proceedings of the 40th International Conference on Software Engineering

As a software system evolves, its test suite can accumulate redundancies over time. Test minimization aims at removing redundant test cases. However, current techniques remove whole test cases from the test suite using test adequacy criteria, such as ...
Understandable Test Generation Through Capture/Replay and LLMs
ICSE-Companion '24: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

Automatic unit test generators, particularly search-based software testing (SBST) tools such as EvoSuite, efficiently generate unit test suites with acceptable coverage. Although this removes the burden of writing unit tests from developers, these ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

July 2016

452 pages

ISBN:9781450343909

DOI:10.1145/2931037

General Chair:
Andreas Zeller
Saarland University, Germany
,
Program Chair:
Abhik Roychoudhury
National University of Singapore, Singapore

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '16

Sponsor:

SIGSOFT

ISSTA '16: International Symposium on Software Testing and Analysis

July 18 - 20, 2016

Saarbrücken, Germany

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
454
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)2

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wanjiru Bvan Bommel PHiemstra D(2024)Sensitivity of Automated SQL Grading in Computer Science CoursesProceedings of the Third International Conference on Innovations in Computing Research (ICR’24)10.1007/978-3-031-65522-7_26(283-299)Online publication date: 1-Aug-2024
https://doi.org/10.1007/978-3-031-65522-7_26
Aşik SYayan U(2023)Generating Python Mutants From Bug Fixes Using Neural Machine TranslationIEEE Access10.1109/ACCESS.2023.330269511(85678-85693)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3302695
Herlim RKim YKim M(2022)CITRUS: Automated Unit Testing Tool for Real-world C++ Programs2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00046(400-410)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00046
Abdi MRocha HDemeyer SBergel A(2022)Small-Amp: Test amplification in a dynamically typed languageEmpirical Software Engineering10.1007/s10664-022-10169-827:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10169-8
Tran HHung P(2021)An Efficient Method for Automated Regression Test Data Generation for C/C++ Projects2021 8th NAFOSTED Conference on Information and Computer Science (NICS)10.1109/NICS54270.2021.9701454(199-204)Online publication date: 21-Dec-2021
https://doi.org/10.1109/NICS54270.2021.9701454
Rutledge ROrso ARothermel GBae D(2020)PG-KLEEProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings10.1145/3377812.3382148(65-68)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377812.3382148
Bach TPannemans RAndrzejak A(2020)Determining Method-Call Sequences for Object Creation in C++2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST46399.2020.00021(108-119)Online publication date: Oct-2020
https://doi.org/10.1109/ICST46399.2020.00021
Rutledge RPark SKhan HOrso APrvulovic MZajic AAtlee JBultan TWhittle J(2019)Zero-overhead path prediction with progressive symbolic executionProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00039(234-245)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00039
Zhang MGong YWang YJin D(2019)Unit Test Data Generation for C Using Rule-Directed Symbolic ExecutionJournal of Computer Science and Technology10.1007/s11390-019-1935-734:3(670-689)Online publication date: 10-May-2019
https://doi.org/10.1007/s11390-019-1935-7
Peleg HRasin DYahav E(2017)Generating Tests by ExampleVerification, Model Checking, and Abstract Interpretation10.1007/978-3-319-73721-8_19(406-429)Online publication date: 29-Dec-2017
https://doi.org/10.1007/978-3-319-73721-8_19

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents