research-article

Enhancing Search-based Testing with Testability Transformations for Existing APIs

Authors: Andrea Arcuri, Juan P. GaleottiAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 31, Issue 1

Article No.: 1, Pages 1 - 34

https://doi.org/10.1145/3477271

Published: 28 September 2021 Publication History

Abstract

Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag problem, where the fitness landscape presents a plateau that provides no guidance to the search. In this article, we provide a series of novel testability transformations aimed at providing guidance in the context of commonly used API calls (e.g., strings that need to be converted into valid date/time objects). We also provide specific transformations aimed at helping the testing of REST Web Services. We implemented our novel techniques as an extension to EvoMaster, an SBST tool that generates system-level test cases. Experiments on nine open-source REST web services, as well as an industrial web service, show that our novel techniques improve performance significantly.

References

[1]

[n.d.].OpenAPI/Swagger. Retrieved from https://swagger.io/.

[2]

[n.d.]. RestAssured. Retrieved from https://github.com/rest-assured/rest-assured.

[3]

[n.d.]. Spring Framework. Retrieved from https://spring.io.

[4]

S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Trans. Softw. Eng. 36, 6 (2010), 742–762.

Digital Library

[5]

Nadia Alshahwan, Xinbo Gao, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, Taijin Tei, and Ilya Zorin. 2018. Deploying search based software engineering with Sapienz at Facebook. In International Symposium on Search Based Software Engineering (SSBSE'18). Springer, 3–45.

[6]

Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test., Verif. Reliab. 16, 3 (2006), 175–203.

Digital Library

[7]

Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test., Verif. Reliab. 16, 3 (2006), 175–203.

Digital Library

[8]

Andrea Arcuri. 2018. EvoMaster: Evolutionary multi-context automated system test generation. In IEEE International Conference on Software Testing, Verification and Validation (ICST'18). IEEE.

[9]

Andrea Arcuri. 2018. An experience report on applying software testing academic results in industry: We need usable automated test generation. Empir. Softw. Eng. 23, 4 (2018), 1959–1981.

Digital Library

[10]

A. Arcuri. 2018. Test suite generation with the many independent objective (MIO) algorithm. Inf. Softw. Technol. 104 (2018), 195–206.

[11]

Andrea Arcuri. 2019. RESTful API automated test case generation with evomaster. ACM Trans. Softw. Eng. Methodol. 28, 1 (2019), 3.

Digital Library

[12]

Andrea Arcuri. 2020. Automated blackbox and whitebox testing of RESTful APIs with evomaster. IEEE Softw. 38, 3 (2020), 72–78.

[13]

A. Arcuri and L. Briand. 2014. A hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test., Verif. Reliab. 24, 3 (2014), 219–250.

Digital Library

[14]

Andrea Arcuri and Juan P Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Trans. Softw. Eng. Methodol. 29, 4 (2020), 1–31.

Digital Library

[15]

Andrea Arcuri and Juan P. Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Trans. Softw. Eng. Methodol. 29, 4 (2020), 1–31.

Digital Library

[16]

Andrea Arcuri and Juan P. Galeotti. 2020. Testability transformations for existing APIs. In IEEE 13th International Conference on Software Testing, Validation and Verification (ICST'20). IEEE, 153–163.

[17]

Andrea Arcuri, Juan Pablo Galeotti, Bogdan Marculescu, and Man Zhang. 2021. EvoMaster: A search-based system test generation tool. J. Open Source Softw. 6, 57 (2021), 2153.

[18]

Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API fuzzing. In 41st International Conference on Software Engineering (ICSE'19). IEEE Press, 748–758.

Digital Library

[19]

Roberto Baldoni, Emilio Coppa, Daniele Cono D'Elia, Camil Demetrescu, and Irene Finocchi. 2018. A survey of symbolic execution techniques. ACM Comput. Surv. 51, 3 (2018), 1–39. https://dl.acm.org/doi/pdf/10.1145/3182657? casa_token=b1T2K3NHAsIAAAAA:IIJUpB57vPm9Ld2_gSX9qTFbeR2ti87KuvCpp2fLVIA7x6LoQzcvqMBILwvwVRJF 4dyib3KCOmkd4A.

Digital Library

[20]

A. Baresel, D. Binkley, M. Harman, and B. Korel. 2004. Evolutionary testing in the presence of loop-assigned flags: A testability transformation approach. In ACM International Symposium on Software Testing and Analysis (ISSTA'04). 108–118.

Digital Library

[21]

A. Baresel and H. Sthamer. 2003. Evolutionary testing of flag conditions. In Genetic and Evolutionary Computation Conference (GECCO'03). 2442–2454.

Digital Library

[22]

D. W. Binkley, M. Harman, and K. Lakhotia. 2011. FlagRemover: A testability transformation for transforming loop-assigned flags. ACM Trans. Softw. Eng. Methodol. 20, 3 (2011), 12:1–12:33.

Digital Library

[23]

Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: three decades later. Commun. ACM 56, 2 (2013), 82–90.

Digital Library

[24]

James Clause, Wanchun Li, and Alessandro Orso. 2007. Dytan: A generic dynamic taint analysis framework. In International Symposium on Software Testing and Analysis. 196–206.

Digital Library

[25]

H. Converse, O. Olivo, and S. Khurshid. 2017. Non-semantics-preserving transformations for higher-coverage test generation using symbolic execution. In IEEE International Conference on Software Testing, Verification and Validation (ICST'17). 241–252.

[26]

Hamza Ed-douibi, Javier Luis Cànovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A Specification-based approach. In IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC'18). 181–190.

[27]

Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. Dissertation. University of California, Irvine.

Digital Library

[28]

Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE'11). 416–419.

Digital Library

[29]

Gordon Fraser and Andrea Arcuri. 2013. Whole test suite generation. IEEE Trans. Softw. Eng. 39, 2 (2013), 276–291.

Digital Library

[30]

Juan Pablo Galeotti, Gordon Fraser, and Andrea Arcuri. 2014. Extending a search-based test generator with adaptive dynamic symbolic execution. In ACM International Symposium on Software Testing and Analysis (ISSTA'14). ACM, 421–424.

Digital Library

[31]

Matthew J. Gallagher and V. Lakshmi Narasimhan. 1997. Adtest: A test data generation suite for ADA software systems. IEEE Trans. Softw. Eng. 23, 8 (1997), 473–484.

Digital Library

[32]

Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'20). Association for Computing Machinery, New York, NY, 725–736.

Digital Library

[33]

D. Gong and X. Yao. 2012. Testability transformation based on equivalence of target statements. Neural Comput. Applic. 21, 8 (2012), 1871–1882.

[34]

Mark Harman. 2018. We need a testability transformation semantics. In International Conference on Software Engineering and Formal Methods. Springer, 3–17.

[35]

M. Harman, A. Baresel, D. W. Binkley, R. M. Hierons, L. Hu, B. Korel, P. McMinn, and M. Roper. 2008. Testability transformation–-program transformation to improve testability. In Formal Methods and Testing, An Outcome of the FORTEST Network, Revised Selected Papers. 320–344.

Digital Library

[36]

Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, André Baresel, and Marc Roper. 2004. Testability transformation. IEEE Trans. Softw. Eng. 30, 1 (2004), 3–16.

Digital Library

[37]

Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1 (2012), 11.

Digital Library

[38]

Wei Huang, Yao Dong, Ana Milanova, and Julian Dolby. 2015. Scalable and precise taint analysis for Android. In International Symposium on Software Testing and Analysis. 106–117.

Digital Library

[39]

Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of openAPI described RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST'20). IEEE.

[40]

J. C. King. 1976. Symbolic execution and program testing. Commun. ACM 19, 7 (1976), 385–394.

Digital Library

[41]

B. Korel. 1990. Automated software test data generation. IEEE Trans. Softw. Eng. 16, 8 (1990), 870–879.

Digital Library

[42]

Y. Li and G. Fraser. 2011. Bytecode testability transformation. In 3rd International Symposium on Search Based Software Engineering (SSBSE'11). 237–251.

Digital Library

[43]

Yun Lin, Jun Sun, Gordon Fraser, Ziheng Xiu, Ting Liu, and Jin Song Dong. 2020. Recovering fitness gradients for interprocedural Boolean flags in search-based testing. In 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 440–451.

Digital Library

[44]

Linghui Luo, Eric Bodden, and Johannes Späth. 2019. A qualitative analysis of Android taint-analysis results. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE'19). IEEE, 102–114.

Digital Library

[45]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-box constraint-based testing of RESTful web APIs. In International Conference on Service-oriented Computing.

Digital Library

[46]

Phil McMinn. 2009. Search-based failure discovery using testability transformations to generate pseudo-oracles. In Genetic and Evolutionary Computation Conference (GECCO'09). 1689–1696.

Digital Library

[47]

P. McMinn, D. Binkley, and M. Harman. 2009. Empirical evaluation of a nesting testability transformation for evolutionary testing. ACM Trans. Softw. Eng. Methodol. 18, 3 (2009), 11:1–11:27.

Digital Library

[48]

Sam Newman. 2015. Building Microservices. O'Reilly Media, Inc.

Digital Library

[49]

Felix Pauck, Eric Bodden, and Heike Wehrheim. 2018. Do Android taint analysis tools keep their promises? In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 331–341.

Digital Library

[50]

R. V. Rajesh. 2016. Spring Microservices. Packt Publishing Ltd.

Digital Library

[51]

José Miguel Rojas, José Campos, Mattia Vivanti, Gordon Fraser, and Andrea Arcuri. 2015. Combining multiple coverage criteria in search-based unit test generation. In International Symposium on Search Based Software Engineering. Springer, 93–108.

[52]

José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2016. Seeding strategies in search-based unit test generation. Softw. Test., Verif. Reliab. 26, 5 (2016), 366–401.

Digital Library

[53]

Edward J. Schwartz, Thanassis Avgerinos, and David Brumley. 2010. All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In IEEE Symposium on Security and Privacy. IEEE, 317–331.

Digital Library

[54]

Omer Tripp, Marco Pistoia, Stephen J. Fink, Manu Sridharan, and Omri Weisman. 2009. TAJ: Effective taint analysis of web applications. ACM SIGPLAN Not. 44, 6 (2009), 87–97.

Digital Library

[55]

Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated black-box testing of RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST'20). IEEE.

[56]

Stefan Wappler, Joachim Wegener, and André Baresel. 2009. Evolutionary testing of software with function-assigned flags. J. Syst. Softw. 82, 11 (2009), 1767–1779.

Digital Library

[57]

Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2019. Resource-based test case generation for RESTful web services. In Genetic and Evolutionary Computation Conference. 1426–1434.

Digital Library

Cited By

Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652157
Belhadi AZhang MArcuri A(2024)Random Testing and Evolutionary Testing for Fuzzing GraphQL APIsACM Transactions on the Web10.1145/360942718:1(1-41)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3609427
Tang YLiu ZZhou ZLuo X(2024)ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.338236550:6(1340-1359)Online publication date: Jun-2024
https://doi.org/10.1109/TSE.2024.3382365
Show More Cited By

Index Terms

Enhancing Search-based Testing with Testability Transformations for Existing APIs
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
    2. Software verification and validation

Recommendations

SQL data generation to enhance search-based system testing
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference

Automated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links), or direct HTTP calls when dealing with web services (e.g., REST and SOAP). However, web/enterprise systems do ...
Search-based failure discovery using testability transformations to generate pseudo-oracles
GECCO '09: Proceedings of the 11th Annual conference on Genetic and evolutionary computation

Testability transformations are source-to-source program transformations that are designed to improve the testability of a program. This paper introduces a novel approach in which transformations are used to improve testability of a program by ...
Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIs
Due to its importance and widespread use in industry, automated testing of REST APIs has attracted major interest from the research community in the last few years. However, most of the work in the literature has been focused on black-box fuzzing. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 31, Issue 1

January 2022

665 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3481711

Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology

Issue’s Table of Contents

Copyright © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 September 2021

Accepted: 01 May 2021

Revised: 01 April 2021

Received: 01 February 2021

Published in TOSEM Volume 31, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

European Research Council (ERC)
UBACYT-2018

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
463
Total Downloads

Downloads (Last 12 months)86
Downloads (Last 6 weeks)5

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652157
Belhadi AZhang MArcuri A(2024)Random Testing and Evolutionary Testing for Fuzzing GraphQL APIsACM Transactions on the Web10.1145/360942718:1(1-41)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3609427
Tang YLiu ZZhou ZLuo X(2024)ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.338236550:6(1340-1359)Online publication date: Jun-2024
https://doi.org/10.1109/TSE.2024.3382365
Zhou ZZhou YFang CChen ZLuo XHe JTang Y(2024)Coverage Goal Selector for Combining Multiple Criteria in Search-Based Unit Test GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.336661350:4(854-883)Online publication date: Apr-2024
https://doi.org/10.1109/TSE.2024.3366613
Da Silva LBorba PMaciel TMahmood WBerger TMoisakis JGomes ALeite V(2024)Detecting semantic conflicts with unit testsJournal of Systems and Software10.1016/j.jss.2024.112070214(112070)Online publication date: Aug-2024
https://doi.org/10.1016/j.jss.2024.112070
Ahsan FAnwer F(2024)A systematic literature review on software security testing using metaheuristicsAutomated Software Engineering10.1007/s10515-024-00433-031:2Online publication date: 23-May-2024
https://dl.acm.org/doi/10.1007/s10515-024-00433-0
Golmohammadi AZhang MArcuri A(2023)Testing RESTful APIs: A SurveyACM Transactions on Software Engineering and Methodology10.1145/361717533:1(1-41)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3617175
Zhang MArcuri A(2023)Open Problems in Fuzzing RESTful APIs: A Comparison of ToolsACM Transactions on Software Engineering and Methodology10.1145/359720532:6(1-45)Online publication date: 30-Sep-2023
https://dl.acm.org/doi/10.1145/3597205
Zhang MBelhadi AArcuri A(2023)JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIsACM Transactions on Software Engineering and Methodology10.1145/359380132:6(1-29)Online publication date: 28-Sep-2023
https://dl.acm.org/doi/10.1145/3593801
Zhang MArcuri ALi YLiu YXue K(2023)White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case StudyACM Transactions on Software Engineering and Methodology10.1145/358500932:5(1-38)Online publication date: 21-Jul-2023
https://dl.acm.org/doi/10.1145/3585009
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents