research-article

Handling SQL Databases in Automated System Test Generation

Authors:

Andrea Arcuri and

Juan P. GaleottiAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 29, Issue 4

Article No.: 22, Pages 1 - 31

https://doi.org/10.1145/3391533

Published: 06 July 2020 Publication History

Abstract

Automated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links and form buttons) or direct HTTP calls when dealing with web services (e.g., REST and SOAP). When doing white-box testing of such systems, their code can be analyzed, and the same type of heuristics (e.g., the branch distance) used in search-based unit testing can be employed to improve performance. However, web/enterprise systems do often interact with a database. To obtain higher coverage and find new faults, the state of the databases needs to be taken into account when generating white-box tests. In this work, we present a novel heuristic to enhance search-based software testing of web/enterprise systems, which takes into account the state of the accessed databases. Furthermore, we enable the generation of SQL data directly from the test cases. This is useful when it is too difficult or time consuming to generate the right sequence of events to put the database in the right state. Also, it is useful when dealing with databases that are “read-only” for the system under test, and the actual data are generated by other services. We implemented our technique as an extension of EVOMASTER, where system tests are generated in the JUnit format. Experiments on six RESTful APIs (five open-source and one industrial) show that our novel techniques improve coverage significantly (up to +16.5%), finding seven new faults in those systems.

References

[1]

[n.d.]. Antlr. Retrieved from https://www.antlr.org/.

[2]

[n.d.]. EclipseLink. Retrieved from http://www.eclipse.org/eclipselink/.

[3]

[n.d.]. Hibernate. Retrieved from http://hibernate.org.

[4]

[n.d.]. JDBI. Retrieved from http://jdbi.org/.

[5]

[n.d.]. JOOQ. Retrieved from https://www.jooq.org/.

[6]

[n.d.]. MongoDB. Retrieved from https://www.mongodb.com/.

[7]

[n.d.]. P6Spy. Retrieved from https://github.com/p6spy/p6spy.

[8]

[n.d.]. RestAssured. Retrieved from https://github.com/rest-assured/rest-assured.

[9]

[n.d.]. Spring Framework. Retrieved from https://spring.io.

[10]

[n.d.]. SQL. Retrieved from https://www.iso.org/standard/63555.html.

[11]

S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Trans. Softw. Eng. 36, 6 (2010), 742--762.

Digital Library

[12]

S. Ali, M. Z. Iqbal, A. Arcuri, and L. C. Briand. 2013. Generating test data from OCL constraints with search techniques. IEEE Trans. Softw. Eng. 39, 10 (2013), 1376--1402.

Digital Library

[13]

Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test. Verif. Reliabil. 16, 3 (2006), 175--203.

Digital Library

[14]

Andrea Arcuri. 2011. A theoretical and empirical analysis of the role of test sequence length in software testing for structural coverage. IEEE Trans. Softw. Eng. 38, 3 (2011), 497--519.

Digital Library

[15]

A. Arcuri. 2018. EvoMaster: Evolutionary multi-context automated system test generation. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST’18). IEEE, 394--397.

[16]

A. Arcuri. 2018. Test suite generation with the many independent objective (MIO) algorithm. Information and Software Technology (IST) 104 (2018), 195--206.

[17]

Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Trans. Softw. Eng. Methodol. 28, 1 (2019), 3.

Digital Library

[18]

A. Arcuri and L. Briand. 2014. A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verif. Reliabil. 24, 3 (2014), 219--250.

Digital Library

[19]

A. Arcuri and G. Fraser. 2016. Java enterprise edition support in search-based junit test generation. In Proceedings of the International Symposium on Search Based Software Engineering (SSBSE’16). Springer, 3--17.

[20]

Andrea Arcuri and Juan P. Galeotti. 2019. SQL data generation to enhance search-based system testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’19). ACM, 1390--1398.

[21]

A. Baresel and H. Sthamer. 2003. Evolutionary testing of flag conditions. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’03). 2442--2454.

[22]

C. Binnig, D. Kossmann, E. Lo, and M. T. Özsu. 2007. QAGen: Generating query-aware test databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data. ACM, 341--352.

Digital Library

[23]

C. Cadar and K. Sen. 2013. Symbolic execution for software testing: Three decades later. Commun. ACM 56, 2 (2013), 82--90.

Digital Library

[24]

J. Castelein, M. Aniche, M. Soltani, A. Panichella, and A. van Deursen. 2018. Search-based test data generation for SQL queries. In Proceedings of the ACM/IEEE International Conference on Software Engineering (ICSE’18). ACM, 1230--1230.

Digital Library

[25]

M. Emmi, R. Majumdar, and K. Sen. 2007. Dynamic test input generation for database applications. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA’07). 151--162.

[26]

G. Fraser and A. Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the ACM Symposium on the Foundations of Software Engineering (FSE’11). 416--419.

[27]

Gordon Fraser and Andrea Arcuri. 2013. Handling test length bloat. Softw. Test. Verif. Reliabil. 23, 7 (2013), 553--582.

[28]

G. Fraser and A. Arcuri. 2013. Whole test suite generation. IEEE Trans. Softw. Eng. 39, 2 (2013), 276--291.

Digital Library

[29]

A. Fuchs and H. Kuchen. 2017. Unit testing of database-driven Java enterprise edition applications. In Proceedings of the International Conference on Tests and Proofs. Springer, 59--76.

[30]

Maryam Abdul Ghafoor, Muhammad Suleman Mahmood, and Junaid Haroon Siddiqui. 2020. Extending symbolic execution for automated testing of stored procedures. Softw. Qual. J. 28 (2020), 853--887. https://link.springer.com/article/10.1007/s11219-019-09453-6.

[31]

James Gosling, Bill Joy, Guy L. Steele, Gilad Bracha, and Alex Buckley. 2014. The Java Language Specification, Java SE 8 Edition (1st ed.). Addison-Wesley Professional.

Digital Library

[32]

Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, André Baresel, and Marc Roper. 2004. Testability transformation. IEEE Trans. Softw. Eng. 30, 1 (2004), 3--16.

Digital Library

[33]

M. Harman, S. A. Mansouri, and Y. Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1 (2012), 11.

Digital Library

[34]

S. Khalek, B. Elkarablieh, Y. Laleye, and S. Khurshid. 2008. Query-aware test generation using a relational constraint solver. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE’08). IEEE, 238--247.

[35]

A. Kieyzun, P. J. Guo, K. Jayaraman, and M. D. Ernst. 2009. Automatic creation of SQL injection and cross-site scripting attacks. In Proceedings of the ACM/IEEE International Conference on Software Engineering (ICSE’09). IEEE Computer Society, 199--209.

[36]

B. Korel. 1990. Automated software test data generation. IEEE Trans. Softw. Eng. 16, 8 (1990), 870--879.

Digital Library

[37]

William B. Langdon and Riccardo Poli. 1998. Fitness causes bloat. In Soft Computing in Engineering Design and Manufacturing. Springer, 13--22.

[38]

Y. Li and G. Fraser. 2011. Bytecode testability transformation. In Proceedings of the 3rd International Symposium on Search Based Software Engineering (SSBSE’11), Lecture Notes in Computer Science, Myra B. Cohen and Mel Ó. Cinnéide (Eds.), Vol. 6956. Springer, 237--251.

[39]

K. Mao, M. Harman, and Y. Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA’16). ACM, 94--105.

[40]

Phil McMinn. 2004. Search-based software test data generation: A survey. Softw. Test. Verif. Reliabil. 14, 2 (2004), 105--156.

Digital Library

[41]

P. Mcminn, C. J. Wright, and G. M. Kapfhammer. 2015. The effectiveness of test coverage criteria for relational database schema integrity constraints. ACM Trans. Softw. Eng. Methodol. 25, 1 (2015), 8.

Digital Library

[42]

P. McMinn, C. J. Wright, C. Kinneer, C. J. McCurdy, M. Camara, and G. M. Kapfhammer. 2016. SchemaAnalyst: Search-based test data generation for relational database schemas. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’16). IEEE, 586--590.

[43]

A. Mesbah, A. Van Deursen, and S. Lenselink. 2012. Crawling Ajax-based web applications through dynamic analysis of user interface state changes. ACM Trans. Web 6, 1 (2012), 3.

Digital Library

[44]

S. Newman. 2015. Building Microservices. O’Reilly Media, Inc.

Digital Library

[45]

A. Panichella, F. Kifetew, and P. Tonella. 2018. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Softw. Eng. 44, 2 (2018), 122--158.

[46]

J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri. 2015. Combining multiple coverage criteria in search-based unit test generation. In Proceedings of the International Symposium on Search Based Software Engineering (SSBSE’15). Springer, 93--108.

[47]

José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2016. Seeding strategies in search-based unit test generation. Softw. Test. Verif. Reliabil. 26, 5 (2016), 366--401.

Digital Library

[48]

J. Thomé, A. Gorla, and A. Zeller. 2014. Search-based security testing of web applications. In Proceedings of the International Workshop on Search-Based Software Testing (SBST’14). ACM, 5--14.

[49]

J. Tuya, M. J. Suárez-Cabal, and C. De La Riva. 2010. Full predicate coverage for testing SQL database queries. Softw. Test. Verif. Reliabil. 20, 3 (2010), 237--288.

Digital Library

Cited By

Liu WMondal SChen T(2024)An Empirical Study on the Characteristics of Database Access Bugs in Java ApplicationsACM Transactions on Software Engineering and Methodology10.1145/3672449Online publication date: 13-Jun-2024
https://doi.org/10.1145/3672449
Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652157
Ahsan FAnwer F(2024)A systematic literature review on software security testing using metaheuristicsAutomated Software Engineering10.1007/s10515-024-00433-031:2Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10515-024-00433-0
Show More Cited By

Index Terms

Handling SQL Databases in Automated System Test Generation
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
    2. Software verification and validation

Recommendations

Enhancing Search-based Testing with Testability Transformations for Existing APIs
Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag ...
Read More
SQL data generation to enhance search-based system testing
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference

Automated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links), or direct HTTP calls when dealing with web services (e.g., REST and SOAP). However, web/enterprise systems do ...
Read More
The Risks of Coverage-Directed Test Case Generation
A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 29, Issue 4

Continuous Special Section: AI and SE

October 2020

307 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3409663

Editor:
Mauro Pezzè
Università della Svizzera italiana and Università di Milano-Bicocca, Switzerland

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2020

Online AM: 07 May 2020

Accepted: 01 March 2020

Revised: 01 January 2020

Received: 01 September 2019

Published in TOSEM Volume 29, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

PICT
Norges Forskningsråd
Secretaría de Ciencia y Técnica, Universidad de Buenos Aires

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
598
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)8

Other Metrics

View Author Metrics

Citations

Cited By

Liu WMondal SChen T(2024)An Empirical Study on the Characteristics of Database Access Bugs in Java ApplicationsACM Transactions on Software Engineering and Methodology10.1145/3672449Online publication date: 13-Jun-2024
https://doi.org/10.1145/3672449
Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652157
Ahsan FAnwer F(2024)A systematic literature review on software security testing using metaheuristicsAutomated Software Engineering10.1007/s10515-024-00433-031:2Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10515-024-00433-0
Golmohammadi AZhang MArcuri A(2023)Testing RESTful APIs: A SurveyACM Transactions on Software Engineering and Methodology10.1145/361717533:1(1-41)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3617175
Belhadi AZhang MArcuri A(2023)Random Testing and Evolutionary Testing for Fuzzing GraphQL APIsACM Transactions on the Web10.1145/360942718:1(1-41)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.1145/3609427
Zhang MArcuri A(2023)Open Problems in Fuzzing RESTful APIs: A Comparison of ToolsACM Transactions on Software Engineering and Methodology10.1145/359720532:6(1-45)Online publication date: 30-Sep-2023
https://dl.acm.org/doi/10.1145/3597205
Zhang MBelhadi AArcuri A(2023)JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIsACM Transactions on Software Engineering and Methodology10.1145/359380132:6(1-29)Online publication date: 24-Apr-2023
https://dl.acm.org/doi/10.1145/3593801
Zhang MArcuri ALi YLiu YXue K(2023)White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case StudyACM Transactions on Software Engineering and Methodology10.1145/358500932:5(1-38)Online publication date: 21-Jul-2023
https://dl.acm.org/doi/10.1145/3585009
Golmohammadi A(2023)Enhancing White-Box Search-Based Testing of RESTful APIs2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW60843.2023.00034(9-12)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSREW60843.2023.00034
Guo HTao CHuang Z(2023)Multi-Objective White-Box Test Input Selection for Deep Neural Network Model Enhancement2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00051(521-532)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00051
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents