research-article

FSX: a tool for fine-grained incremental unit test generation for C/C++ programs

Authors:

Hiroaki Yoshida,

Susumu Tokumoto,

Mukul R. Prasad,

Indradeep Ghosh,

Tadahiro UeharaAuthors Info & Claims

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 1052 - 1056

https://doi.org/10.1145/2950290.2983937

Published: 01 November 2016 Publication History

Abstract

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, in previous work, we proposed a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. In this paper we describe a tool FSX, implementing this technique. We describe the architecture, user-interface, and salient features of FSX, and specific practical use-cases of its technology. We also report on a real, large-scale deployment of FSX, as a practical validation of the underlying research contribution and of automated test generation research in general.

References

[1]

CLANG, http://clang.llvm.org/.

[2]

CUDD: CU Decision Diagram Package, http://vlsi.colorado.edu/~fabio/CUDD/.

[3]

Free Software Foundation, Inc. How to Minimize Test Cases for Bugs, https://gcc.gnu.org/bugs/minimize.html.

[4]

googletest - Google C++ Testing Framework, http://code.google.com/p/googletest/.

[5]

How to submit an LLVM bug report, http://llvm.org/docs/HowToSubmitABug.html.

[6]

iPerf - The network bandwidth measurement tool, https://iperf.fr/.

[7]

LLVM, http://www.llvm.org/.

[8]

Mozilla Developer Network. Reducing testcases, https://developer.mozilla.org/en-US/docs/Mozilla/QA/ Reducing_testcases.

[9]

PIT Mutation Testing, http://pitest.org/.

[10]

Sinatra, http://www.sinatrarb.com/.

[11]

Webkit. Test Case Reduction, https://webkit.org/test-case-reduction/.

[12]

B. Beizer. Software Testing Techniques. John Wiley & Sons, Inc., New York, NY, USA, 1990.

Digital Library

[13]

R. E. Bryant. Symbolic boolean manipulation with ordered binary-decision diagrams. ACM Computing Surveys, 24(3):293–318, 1992.

Digital Library

[14]

C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2008.

Digital Library

[15]

C. Cadar, P. Godefroid, S. Khurshid, C. S. Păsăreanu, K. Sen, N. Tillmann, and W. Visser. Symbolic execution for software testing in practice: Preliminary assessment. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11. ACM, 2011.

Digital Library

[16]

L. De Moura and N. Bjørner. Z3: An Efficient SMT Solver. In 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS/ETAPS’08. Springer-Verlag, 2008.

Digital Library

[17]

G. Fraser and A. Arcuri. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, 2011.

Digital Library

[18]

V. Ganesh and D. L. Dill. A decision procedure for bit-vectors and arrays. In Proceedings of the 19th International Conference on Computer Aided Verification, CAV’07. Springer-Verlag, 2007.

Digital Library

[19]

P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In PLDI, 2005.

Digital Library

[20]

P. Godefroid, M. Y. Levin, and D. A. Molnar. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium, NDSS, 2008.

[21]

J. B. Goodenough and S. L. Gerhart. Toward a Theory of Test Data Selection. IEEE Tran. on Software Engineering, 1975.

Digital Library

[22]

Y. Kim, Y. Kim, T. Kim, G. Lee, Y. Jang, and M. Kim. Automated unit testing of large industrial embedded software using concolic testing. In 28th International Conference on Automated Software Engineering (ASE), 2013.

Digital Library

[23]

Y. Kim, Z. Zu, M. Kim, M. Cohen, and G. Rothermel. Hybrid directed test suite augmentation: An interleaving framework. In 2014 IEEE International Conference on Software Testing, Verification and Validation (ICST), 2014.

Digital Library

[24]

G. Li, E. Andreasen, and I. Ghosh. SymJS: Automatic symbolic testing of javascript web applications. In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014. ACM, 2014.

Digital Library

[25]

G. Li, I. Ghosh, and S. P. Rajan. KLOVER : A symbolic execution and automatic test generation tool for C++ programs. In 23rd International Conference on Computer Aided Verification (CAV), pages 609–615, 2011.

Digital Library

[26]

P. McMinn. Search-based software test data generation: A survey: Research articles. Softw. Test. Verif. Reliab., 14(2), June 2004.

Digital Library

[27]

C. Nguyen, H. Yoshida, M. R. Prasad, I. Ghosh, and K. Sen. Generating succinct test cases using don’t care analysis. In Proc. of the 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST’15, 2015.

[28]

S. Person, G. Yang, N. Rungta, and S. Khurshid. Directed incremental symbolic execution. In Proceedings of the 32Nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’11. ACM, 2011.

Digital Library

[29]

C. S. Păsăreanu and W. Visser. A survey of new trends in symbolic execution for software testing and analysis. Int. J. Softw. Tools Technol. Transf., 11(4):339–353, Oct. 2009.

[30]

C. S. Pˇasˇareanu and N. Rungta. Symbolic PathFinder: symbolic execution of Java bytecode. In 25th Intl. Conf. on Automated Software Engineering (ASE), 2010.

Digital Library

[31]

R. Santelices, P. K. Chittimalli, T. Apiwattanapong, A. Orso, and M. J. Harrold. Test-suite augmentation for evolving software. In 23rd International Conference on Automated Software Engineering, ASE. IEEE, 2008.

Digital Library

[32]

K. Sen and G. Agha. CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-checking Tools. In Proc. of the 18th International Conference on Computer Aided Verification, CAV’06. Springer-Verlag, 2006.

Digital Library

[33]

K. Sen, D. Marinov, and G. Agha. CUTE: A Concolic Unit Testing Engine for C. In ESEC-FSE, 2005.

Digital Library

[34]

D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, J. Newsome, P. Poosankam, and P. Saxena. Bitblaze: A new approach to computer security via binary analysis. In 4th International Conference on Information Systems Security, ICISS ’08. Springer-Verlag, 2008.

Digital Library

[35]

K. Taneja, T. Xie, N. Tillmann, and J. de Halleux. eXpress: Guided path exploration for efficient regression test generation. In 2011 International Symposium on Software Testing and Analysis, ISSTA ’11. ACM, 2011.

Digital Library

[36]

N. Tillmann and J. de Halleux. PEX – White Box Test Generation for .NET. In Tests and Proofs, 2008.

Digital Library

[37]

N. Tillmann and W. Schulte. Parameterized Unit Tests. In ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 253–262, 2005.

Digital Library

[38]

Z. Xu, Y. Kim, M. Kim, G. Rothermel, and M. B. Cohen. Directed test suite augmentation: Techniques and tradeoffs. In 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE ’10. ACM, 2010.

Digital Library

[39]

Z. Xu and G. Rothermel. Directed test suite augmentation. In 16th Asia-Pacific Software Engineering Conference, APSEC 2009, 2009.

Digital Library

[40]

G. Yang, C. S. Păsăreanu, and S. Khurshid. Memoized symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA 2012. ACM, 2012.

Digital Library

[41]

H. Yoshida, S. Tokumoto, M. R. Prasad, I. Ghosh, and T. Uehara. FSX: Fine-grained incremental unit test generation for C/C++ programs. In 2016 International Symposium on Software Testing and Analysis, ISSTA 2016, 2016.

Digital Library

Cited By

Greca RMiranda BBertolino A(2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3579851
Bach TPannemans RAndrzejak A(2020)Determining Method-Call Sequences for Object Creation in C++2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST46399.2020.00021(108-119)Online publication date: Oct-2020
https://doi.org/10.1109/ICST46399.2020.00021

Index Terms

FSX: a tool for fine-grained incremental unit test generation for C/C++ programs
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

FSX: fine-grained incremental unit test generation for C/C++ programs
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. ...
Fine-grained test minimization
ICSE '18: Proceedings of the 40th International Conference on Software Engineering

As a software system evolves, its test suite can accumulate redundancies over time. Test minimization aims at removing redundant test cases. However, current techniques remove whole test cases from the test suite using test adequacy criteria, such as ...
Understandable Test Generation Through Capture/Replay and LLMs
ICSE-Companion '24: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

Automatic unit test generators, particularly search-based software testing (SBST) tools such as EvoSuite, efficiently generate unit test suites with acceptable coverage. Although this removes the burden of writing unit tests from developers, these ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2016

1156 pages

ISBN:9781450342186

DOI:10.1145/2950290

General Chair:
Thomas Zimmermann
Microsoft Research, USA
,
Program Chairs:
Jane Cleland-Huang
University of Notre Dame, USA
,
Zhendong Su
University of California at Davis, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13 - 18, 2016

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
267
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Greca RMiranda BBertolino A(2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3579851
Bach TPannemans RAndrzejak A(2020)Determining Method-Call Sequences for Object Creation in C++2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST46399.2020.00021(108-119)Online publication date: Oct-2020
https://doi.org/10.1109/ICST46399.2020.00021

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents