Cited By
View all- Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIs
Article No.: 139, Pages 1 - 29
Abstract
JavaScript is one of the most popular programming languages. However, its dynamic nature poses several challenges to automated testing techniques. In this paper, we propose an approach and open-source tool support to enable white-box testing of JavaScript applications using Search-Based Software Testing (SBST) techniques. We provide an automated approach to collect search-based heuristics like the common Branch Distance and to enable Testability Transformations. To empirically evaluate our results, we integrated our technique into the EvoMaster test generation tool, and carried out analyses on the automated system testing of RESTful and GraphQL APIs. Experiments on eight Web APIs running on NodeJS show that our technique leads to significantly better results than existing black-box and grey-box testing tools, in terms of code coverage and fault detection.
References
[1]
(n. d.). Babel. https://babeljs.io/. Online, Accessed May 20, 2022.
[2]
(n. d.). C8. https://github.com/bcoe/c8. Online, Accessed May 20, 2022.
[3]
(n. d.). cyclotron. https://github.com/ExpediaInceCommercePlatform/cyclotron. Online, Accessed May 20, 2022.
[4]
(n. d.). disease-sh-api. https://github.com/disease-sh/API. Online, Accessed May 20, 2022.
[5]
(n. d.). E-Commerce Server. https://github.com/react-shop/react-ecommerce. Online, Accessed May 20, 2022.
[6]
(n. d.). ECMAScript Specification. https://www.ecma-international.org/ecma-262/. Online, Accessed May 20, 2022.
[7]
(n. d.). Electron. https://www.electronjs.org/. Online, Accessed May 20, 2022.
[8]
(n. d.). EvoMaster Benchmark (EMB). https://github.com/EMResearch/EMB. Online, Accessed May 20, 2022.
[9]
(n. d.). Ionic. https://ionicframework.com/. Online, Accessed May 20, 2022.
[10]
(n. d.). JEDI. https://github.com/aelyasov/JEDI. Online, Accessed May 20, 2022.
[11]
(n. d.). Jest. https://jestjs.io/. Online, Accessed May 20, 2022.
[12]
(n. d.). JUnit. http://junit.sourceforge.net/. Online, Accessed May 20, 2022.
[13]
(n. d.). MongoDB. https://www.mongodb.com/. Online, Accessed May 20, 2022.
[14]
(n. d.). nestjs-realworld-example-app. https://github.com/lujakob/nestjs-realworld-example-app. Online, Accessed May 20, 2022.
[15]
(n. d.). NodeJS. https://nodejs.org/. Online, Accessed May 20, 2022.
[16]
(n. d.). React-Finland. https://github.com/ReactFinland/graphql-api. Online, Accessed May 20, 2022.
[17]
(n. d.). realworld API Specification. https://github.com/gothinkster/realworld. Online, Accessed May 20, 2022.
[18]
(n. d.). Redis. https://redis.io/. Online, Accessed May 20, 2022.
[19]
(n. d.). RestAssured. https://github.com/rest-assured/rest-assured. Online, Accessed May 20, 2022.
[20]
(n. d.). restler-fuzzer. https://github.com/microsoft/restler-fuzzer. Online, Accessed May 20, 2022.
[21]
(n. d.). RESTTESTGEN. https://github.com/resttestgenicst2020/submission_icst2020. Online, Accessed May 20, 2022.
[22]
(n. d.). SpaceX-API. https://github.com/r-spacex/SpaceX-API. Online, Accessed May 20, 2022.
[23]
(n. d.). The State of the OCTOVERSE. https://octoverse.github.com/.
[24]
(n. d.). SuperAgent. https://visionmedia.github.io/superagent/. Online, Accessed May 20, 2022.
[25]
S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Transactions on Software Engineering (TSE) 36, 6 (2010), 742–762.
[26]
Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Software Testing, Verification, and Reliability 16, 3 (2006), 175–203. DOI:
[27]
Esben Andreasen, Liang Gong, Anders Møller, Michael Pradel, Marija Selakovic, Koushik Sen, and Cristian-Alexandru Staicu. 2017. A survey of dynamic analysis and test generation for JavaScript. ACM Computing Surveys (CSUR) 50, 5 (2017), 1–36.
[28]
Andrea Arcuri. 2018. EvoMaster: Evolutionary multi-context automated system test generation. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.
[29]
Andrea Arcuri. 2018. Test suite generation with the many independent objective (MIO) algorithm. Information and Software Technology 104 (2018), 195–206.
[30]
Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 1 (2019), 3.
[31]
Andrea Arcuri. 2020. Automated black-and white-box testing of RESTful APIs With EvoMaster. IEEE Software 38, 3 (2020), 72–78.
[32]
A. Arcuri and L. Briand. 2011. Adaptive random testing: An illusion of effectiveness?. In ACM Int. Symposium on Software Testing and Analysis (ISSTA). 265–275.
[33]
A. Arcuri and L. Briand. 2014. A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability (STVR) 24, 3 (2014), 219–250.
[34]
Andrea Arcuri and Juan P. Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 4 (2020), 1–31.
[35]
Andrea Arcuri and Juan P. Galeotti. 2021. Enhancing search-based testing with testability transformations for existing APIs. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–34.
[36]
Andrea Arcuri, Juan Pablo Galeotti, Bogdan Marculescu, and Man Zhang. 2021. EvoMaster: A search-based system test generation tool. Journal of Open Source Software 6, 57 (2021), 2153.
[37]
Andrea Arcuri, ZhangMan, asmab89, Bogdan, Amid Golmohammadi, Juan Pablo Galeotti, Seran, Alberto Martín López, Agustina Aldasoro, Annibale Panichella, and Kyle Niemeyer. 2022. EMResearch/EvoMaster:. (June2022). DOI:
[38]
Andrea Arcuri, ZhangMan, Bogdan, asmab89, Amid Golmohammadi, Juan Pablo Galeotti, Alberto Martín López, Agustina Aldasoro, Annibale Panichella, and Kyle Niemeyer. 2022. EMResearch/EvoMaster:. (Feb.2022). DOI:
[39]
Andrea Arcuri, ZhangMan, Amid Golmohammadi, and asmab89. 2022. EMResearch/EMB:. (Feb.2022). DOI:
[40]
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API fuzzing. In ACM/IEEE International Conference on Software Engineering (ICSE). 748–758.
[41]
A. Baresel, D. Binkley, M. Harman, and B. Korel. 2004. Evolutionary testing in the presence of loop-assigned flags: A testability transformation approach. In ACM Int. Symposium on Software Testing and Analysis (ISSTA). 108–118.
[42]
A. Baresel and H. Sthamer. 2003. Evolutionary testing of flag conditions. In Genetic and Evolutionary Computation Conference (GECCO). 2442–2454.
[43]
Asma Belhadi, Man Zhang, and Andrea Arcuri. 2022. Evolutionary-based automated testing for GraphQL APIs. In Genetic and Evolutionary Computation Conference (GECCO).
[44]
Asma Belhadi, Man Zhang, and Andrea Arcuri. 2022. White-Box and Black-Box Fuzzing for GraphQL APIs. (2022). DOI:
[45]
D. W. Binkley, M. Harman, and K. Lakhotia. 2011. FlagRemover: A testability transformation for transforming loop-assigned flags. ACM Trans. Softw. Eng. Methodol. 20, 3 (2011), 12:1–12:33. DOI:
[46]
Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Empirical comparison of black-box test case generation tools for RESTful APIs. In 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 226–236.
[47]
Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2021. Replication Package: Automated Black-Box Testing of Nominal and Error Scenarios in RESTful APIs. (Dec.2021). DOI:
[48]
Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2022. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability (2022), e1808.
[49]
Domenico Cotroneo, Antonio Ken Iannillo, and Roberto Natella. 2019. Evolutionary fuzzing of Android OS vendor system services. Empirical Software Engineering 24 (2019), 3630–3658.
[50]
Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. 2018. Compiler fuzzing through deep learning. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 95–105.
[51]
Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A specification-based approach. In 2018 IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC). 181–190.
[52]
Alexander Elyasov, I. S. W. B. Prasetya, and Jurriaan Hage. 2018. Search-based test data generation for JavaScript functions that interact with the DOM. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 88–99.
[53]
Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE). 416–419.
[54]
G. Fraser and A. Arcuri. 2012. Sound empirical evidence in software testing. In ACM/IEEE International Conference on Software Engineering (ICSE). 178–188.
[55]
Matthew J. Gallagher and V. Lakshmi Narasimhan. 1997. ADTEST: A test data generation suite for ADA software systems. IEEE Transactions on Software Engineering (TSE) 23, 8 (1997), 473–484.
[56]
Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path sensitive fuzzing. In 2018 IEEE Symposium on Security and Privacy (SP). 679–696. DOI:
[57]
Vijay Ganesh, Tim Leek, and Martin Rinard. 2009. Taint-based directed whitebox fuzzing. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 474–484.
[58]
Patrice Godefroid. 2020. Fuzzing: Hack, art, and science. Commun. ACM 63, 2 (2020), 70–76.
[59]
Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In ACM Symposium on the Foundations of Software Engineering (FSE) (ESEC/FSE 2020). ACM, 725–736.
[60]
D. Gong and X. Yao. 2012. Testability transformation based on equivalence of target statements. Neural Computing and Applications 21, 8 (2012), 1871–1882. DOI:
[61]
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 269–280.
[62]
Mark Harman. 2018. We need a testability transformation semantics. In International Conference on Software Engineering and Formal Methods. Springer, 3–17.
[63]
M. Harman, A. Baresel, D. W. Binkley, R. M. Hierons, L. Hu, B. Korel, P. McMinn, and M. Roper. 2008. Testability transformation - program transformation to improve testability. In Formal Methods and Testing, an Outcome of the FORTEST Network, Revised Selected Papers. 320–344. DOI:
[64]
M. Harman, L. Hu, R. Hierons, A. Baresel, and H. Sthamer. 2002. Improving evolutionary testing by flag removal. In Genetic and Evolutionary Computation Conference (GECCO). 1351–1358.
[65]
M. Harman and B. F. Jones. 2001. Search-based software engineering. Journal of Information & Software Technology 43, 14 (2001), 833–839.
[66]
Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR) 45, 1 (2012), 11.
[67]
Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering (TSE) 37, 5 (2011), 649–678.
[68]
Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of OpenAPI described RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.
[69]
Bogdan Korel. 1990. Automated software test data generation. IEEE Transactions on Software Engineering 16, 8 (1990), 870–879.
[70]
Kiran Lakhotia, Mark Harman, and Hamilton Gross. 2013. AUSTIN: An open source tool for search based software testing of C programs. Information and Software Technology 55, 1 (2013), 112–125.
[71]
Y. Li and G. Fraser. 2011. Bytecode testability transformation. In Search Based Software Engineering - Third International Symposium, SSBSE 2011, Szeged, Hungary, September 10-12, 2011. Proceedings. 237–251. DOI:
[72]
Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. ACM SIGPLAN Notices 50, 6 (2015), 65–76.
[73]
Yun Lin, Jun Sun, Gordon Fraser, Ziheng Xiu, Ting Liu, and Jin Song Dong. 2020. Recovering fitness gradients for interprocedural Boolean flags in search-based testing. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 440–451.
[74]
Riyadh Mahmood, Jay Pennington, Danny Tsang, Tan Tran, and Andrea Bogle. 2022. A framework for automated API fuzzing at enterprise scale. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 377–388.
[75]
Valentin J. M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering 47, 11 (2019), 2312–2331.
[76]
Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for Android applications. In ACM Int. Symposium on Software Testing and Analysis (ISSTA). ACM, 94–105.
[77]
Bogdan Marculescu, Man Zhang, and Andrea Arcuri. 2022. On the faults found in REST APIs by automated test generation. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 3 (2022), 1–43.
[78]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-box constraint-based testing of RESTful Web APIs. In International Conference on Service-Oriented Computing.
[79]
P. McMinn. 2004. Search-based software test data generation: A survey. Software Testing, Verification and Reliability 14, 2 (2004), 105–156.
[80]
P. McMinn, D. Binkley, and M. Harman. 2009. Empirical evaluation of a nesting testability transformation for evolutionary testing. ACM Trans. Softw. Eng. Methodol. 18, 3 (2009), 11:1–11:27. DOI:
[81]
Barton P. Miller, Lars Fredriksen, and Bryan So. 1990. An empirical study of the reliability of UNIX utilities. Commun. ACM 33, 12 (Dec.1990), 32–44. DOI:
[82]
José Fragoso Santos, Petar Maksimović, Théotime Grohens, Julian Dolby, and Philippa Gardner. 2018. Symbolic execution for JavaScript. In Proceedings of the 20th International Symposium on Principles and Practice of Declarative Programming. 1–14.
[83]
Marija Selakovic, Michael Pradel, Rezwana Karim, and Frank Tip. 2018. Test generation for higher-order functions in dynamic languages. Proceedings of the ACM on Programming Languages 2, OOPSLA (2018), 1–27.
[84]
Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: A selective record-replay and dynamic analysis framework for JavaScript. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 488–498.
[85]
Wei Song, Xiangxing Qian, and Jeff Huang. 2017. EHBDroid: Beyond GUI testing for Android applications. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 27–37.
[86]
Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated black-box testing of RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.
[87]
Stefan Wappler, Joachim Wegener, and André Baresel. 2009. Evolutionary testing of software with function-assigned flags. Journal of Systems and Software 82, 11 (2009), 1767–1779. DOI:
[88]
J. Wegener, A. Baresel, and H. Sthamer. 2001. Evolutionary test environment for automatic structural testing. Information and Software Technology 43, 14 (2001), 841–854.
[89]
Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019. The Fuzzing Book. (2019).
[90]
Man Zhang and Andrea Arcuri. 2021. Adaptive hypermutation for search-based system test generation: A study on REST APIs with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021).
[91]
Man Zhang and Andrea Arcuri. 2022. Open Problems in Fuzzing RESTful APIs: A Comparison of Tools. (2022). DOI:
[92]
Man Zhang, Andrea Arcuri, Yonggang Li, Yang Liu, and Kaiming Xue. 2023. White-box fuzzing RPC-based APIs with EvoMaster: An industrial case study. ACM Transactions on Software Engineering and Methodology (TOSEM) (Feb.2023). DOI:Just Accepted.
[93]
Man Zhang, Asma Belhadi, and Andrea Arcuri. 2022. JavaScript instrumentation for search-based software testing: A study with RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.
[94]
Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2019. Resource-based test case generation for RESTful web services. In Proceedings of the Genetic and Evolutionary Computation Conference. 1426–1434.
[95]
Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2021. Resource and dependency based test case generation for RESTful Web services. Empirical Software Engineering 26, 4 (2021), 1–61.
[96]
Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022. Fuzzing: A survey for roadmap. Comput. Surveys 54, 11s, Article 230 (Sep.2022), 36 pages. DOI:
Index Terms
- JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIs
Recommendations
Random Testing and Evolutionary Testing for Fuzzing GraphQL APIs
The Graph Query Language (GraphQL) is a powerful language for application programming interface (API) manipulation in web services. It has been recently introduced as an alternative solution for addressing the limitations of RESTful APIs. This article ...
Open Problems in Fuzzing RESTful APIs: A Comparison of Tools
RESTful APIs are a type of web service that are widely used in industry. In the past few years, a lot of effort in the research community has been spent in designing novel techniques to automatically fuzz those APIs to find faults in them. Many real ...
White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study
Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks ...
Comments
Information & Contributors
Information
Published In
November 2023
949 pages
Copyright © 2023 Copyright held by the owner/author(s).
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 September 2023
Online AM: 24 April 2023
Accepted: 15 March 2023
Revised: 13 January 2023
Received: 20 May 2022
Published in TOSEM Volume 32, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- European Research Council (ERC)
- European Union’s Horizon 2020 research and innovation programme
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 1,669Total Downloads
- Downloads (Last 12 months)1,094
- Downloads (Last 6 weeks)69
Reflects downloads up to 06 Jan 2025
Other Metrics
Citations
Cited By
View all- Arcuri AZhang MGaleotti J(2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in