Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIs

The work presented in this article deals with the white-box fuzzing in the context of REST APIs. In the context of Web and Enterprise applications and technologies, there are several common terms and tools/libraries widely used in industry and that are specific for this domain. To make this article easier to read, here we list (in alphabetic order) some of those terms/tools/libraries with some brief description, as those are referenced often throughout the rest of the article.

Presently, a significant portion of web services are implemented following the REST (REpresentational State Transfer) architectural style [42]. Notable adopters of this approach among organizations and companies are Google,⁵ Amazon,⁶ X (formerly known as Twitter),⁷ Reddit,⁸ LinkedIn,⁹ among others. Beyond their role in furnishing internet-based functionalities (e.g., see API portals such as APIs.guru¹⁰ and RapidAPI Hub¹¹), the REST architectural style also holds sway within enterprise backends, particularly when microservices architecture are used [79, 81].

The REST architectural style is not a protocol per se, but rather a collection of principles governing and structuring resources accessible over HTTP(S) networks. Resources are pinpointed via URLs and can be manipulated using HTTP semantics, with GET requests for data retrieval, POST for data creation, PUT/PATCH for data modification, and DELETE for deletion. Inputs can be transmitted through path components in URLs, query parameters, HTTP headers, and payload bodies. Various data formats can be employed for transmission, with JSON (JavaScript Object Notation) currently ranking among the most prevalent such data format.

To enhance the understandability and usability of REST APIs, a common strategy involves providing schemas outlining the available endpoints and supported input formats (including details like query parameter names and types). Multiple schema standards exist, with OpenAPI/Swagger [9] standing out as the current industry leader. This format harnesses either JSON or YAML (Yet Another Markup Language) to specify these schemas. Schemas can be authored either manually or auto-generated from API source code (exemplified by tools like SpringFox and SpringDoc for the widely adopted enterprise framework Spring [12]).

In the context of REST APIs, a system-level test case is a sequence of HTTP calls to different endpoints. The expected result of each HTTP call can be asserted, checking that the value of the status code is within some given set of expected status codes values. As one might expect, writing a specific system-level test case that triggers a specific behaviour in the implementation of the API (such as creating a resource with a specific data format, or retrieving an existing resource against its expected value) is difficult and time consuming. Figure 1 presents a system-level test case (written using the RestAssured [11] library) that creates a new User resource using the endpoint /users. Subsequently, the test case retrieves a User resource by means of a GET HTTP call, and the retrieved data is compared against their expected values. The automation of creating such system-level test cases remains elusive [21].

The open-source tool EvoMaster [20, 28] aims at automatically generating system-level test cases for REST APIs. The existing version EvoMaster \(^O\) implements several evolutionary algorithms (such as MIO [22]) to evolve test cases towards maximizing code coverage and fault finding metrics. As API calls might depend on each other (e.g., in Figure 1 a typical GET template is exercised, where a unique PUT call is exercised followed by a GET call), EvoMaster can exploit dependencies among API resources [98, 99] to cover more API behaviour.

EvoMaster \(^O\) is divided in two main modules: (1) a core process and (2) a driver. The core process contains all the basic functionality for a SBST tool, like the search algorithms, fitness functions, writing the test generation outputs, and so on. On the other hand, the driver is provided as a library, which engineers need to use to specify how to start, stop and reset the SUT. This is done with short configuration classes, that need to be implemented manually. However, the driver is also responsible for instrumenting the bytecode, which is done automatically. This instrumentation allows EvoMaster \(^O\) ’s driver to retrieve several different SBST heuristics like the branch distance. This heuristic is not only retrieved for predicates in the control flow of the SUT, but also for all SQL commands executed over a database (if any) [26].

The core and the driver will run as separated processes, communicating over HTTP. This architecture will enable to support different programming languages, as a new supported language would just require a new driver library for it (such as JavaScript [96, 97] and C# [52]). Additionally, this architecture also allows EvoMaster to support both white-box [23] and black-box [24] testing. If black-box testing is chosen, EvoMaster \(^O\) ’s core process can directly interact with the deployed API. Therefore, there is no need for engineers to write any driver as EvoMaster could be run on any type of REST API regardless of their programming language.

If white-box testing is chosen, EvoMaster \(^O\) currently supports APIs that run on the JVM [22, 23, 27, 93] (e.g., written in Java or Kotlin), JavaScript/TypeScript [96, 97] and C# [52]. For JVM-based APIs, it can output test suites in JUnit format, using the library RestAssured [11] for making the HTTP calls toward the SUT. The generated tests will use the configuration classes for handling the SUT. This means that the generated tests are self-contained: the test suite files can start the SUT before any test is run, reset the state of the SUT before/after each test case execution (to make them independent), and stop the SUT once all tests are run. The generated test cases can be used as well for regression testing, as can be added to the repository of the SUT, and run as part of a Continuous Integration process.

In case of black-box testing, the generated test suites would still be output, e.g., in either Java or Kotlin using the RestAssured library, independently of the programming language in which the target APIs were written.

In [27] we have presented a series of testability transformations [57] to improve the fitness function of EvoMaster \(^O\) , which was built on top of some basic examples (e.g., for String.equals) from EvoSuite [43]. In particular, in the case of JDK APIs where method calls end-up providing no gradient to the search (e.g., they return either boolean values or throw an exception on invalid inputs), for several of those APIs we automatically replace those method calls with our own custom versions, at class-loading time via bytecode instrumentation. These custom method versions are semantically equivalent: for the same inputs they give the same outputs. However, they compute heuristic values to determine how far the test data was from returning either a true or false output, and how far they were from not throwing an exception. Such heuristic values are then used in the fitness function to guide SBST search toward generating test data to get the desired output.

Consider this following simple example:

Here, sampling the string variable x at random would have extremely low probability of satisfying the constraint of that if statement. In our bytecode instrumentation, that code would be replaced with:

Here, String.equals gets replaced with our own StringClassReplacement.equals version. Internally, before returning the same result as String.equals, it computes heuristic values for the two possible outcomes: true or false (which are then registered as targets in the search using the label targetID). Such heuristics are based on the distance values defined in [17]. The search is then rewarded in the fitness function to apply modifications to x that lead to have at least one test case in which such call returns true, and one that returns false.

This approach gives gradient to the search to evolve a desired x value. However, due to the length of the string, it can take several generations in the evolutionary process to evolve the desired value. But what if x is part of the test data? Once it is detected at runtime that such value is compared with an equals method, it would be more efficient to simply use the desired value (i.e., “A quite long string that it is unlikely to get at random” in this case) directly instead of evolving it. The problem is that x could be modified during the execution of the SUT (e.g., it is the result of a substring operation, or a concatenation of different strings). Also, in the testing of RESTful APIs, such x could be the value of a URL query parameter, or a nested field in a JSON body payload in a POST request.

To address this issue, in Reference [27] we presented a technique which is a form of taint analysis. When mutating strings in a test case (regardless of where they are, e.g., query parameters and fields in JSON objects), with a certain probability we replace such strings with tainted values in the form _EM_\d+_XYZ_. For example, _EM_0_XYZ_. In our method replacements, every time we detect that a string input is matching such regular expression, we can directly identify which part of the genotype it comes from. Then, the search can automatically replace such string values based on information from the replacement methods, making those constraints trivial to solve. That regular expression is written in that way to reduce the chances that a random string in the SUT would match it. Also most modifications on the inputs (apart from modifying digits in the middle of the string) would be detected as well.

This approach does not work in all cases (e.g., when strings are modified), but it is very effective for RESTful APIs [27]. The reason is that those APIs often have many string inputs that are not modified by the SUT (i.e., they are just read).

In [27] we provided several method replacements, e.g., for String.startsWith, Integer.parseInt and Collection.contains. For each one, we defined heuristic distances to provide gradient to the search. However, there are several more methods in the JDK APIs that could be handled this way.

OpenAPI schemas could be underspecified. For example, some query parameters and headers could be missing from the schema, albeit being handled by the API. A white-box approach could detect some of these cases, as it can analyze the source code of the API.

In the popular Spring framework (which is the most used enterprise framework for the Java language), parameters and headers can be automatically injected via annotations in the REST controllers (e.g., using @RequestParam and @RequestHeader). In those cases, automated tools that create OpenAPI schemas by analyzing annotations can detect those inputs (e.g., SpringFox and SpringDoc). However, a less common case is to pass as input to those REST controllers an object representing the whole HTTP call, such as WebRequest. In those cases, automated schema generators would have no information on the actual expected structure of the incoming HTTP requests. The generated schema definitions would hence have no useful info.

As this problem actually happens in some of the APIs in EMB [3, 32], in [27] we presented some techniques to handle some basic cases of this issue. In particular, we provided method replacements for getParameter(), getParameterValues(), getHeader() and getHeaders(). Everytime any of these methods is called, and the presence of a parameter/header not in the schema is detected, our instrumentation will inform the search engine about such value. Then, EvoMaster will expand the genotype of the evolving individuals, creating test cases that can set and use those newly discovered parameters and headers.

Similarly to parameters and headers, there can be issues when the body payload types are unspecified. For these cases, we provided a method replacement for getInputStream() on the object representing the incoming HTTP requests. If the execution of such call is then detected, we then trace the use of the library Gson for marshalling JSON payloads, in particular the calls to fromJson(). This way, we check which classes are used as DTO to map the incoming body payloads. This information can then be fed back to the search engine, which will now be able to evolve input objects matching those DTO structures. As there are few technical details at play here, we refer the interested reader to [27] for the full details on how this is achieved in EvoMaster \(^O\) .

Fuzzing is one of the most effective approaches to detect software faults, and to achieve higher code coverage [50, 70, 100]. Fuzzers operate using either a white-box or a black-box approach. In a white-box fuzzer, internal details of the system under test, such as its source code, binaries, bytecode, or SQL communication [26], will be accessed. This information can be used to design heuristics to improve the search to produce better test cases with better code coverage and fault detection capabilities. White-box fuzzers have been proven to be highly effective in numerous instances [24, 51, 64, 72, 93].

Fuzzing REST APIs have gained a major amount of attention among researchers in recent years (see for the example the survey [53]). There are multiple fuzzing tools available for RESTful APIs [64, 93]. Some of the aforementioned black-box fuzzers for REST APIs include (in alphabetic order): bBOXRT [66], EvoMaster [20], Dredd [8], Fuzz-lightyear [4], Morest [69], Quickrest [62], ResTest [76], RestCT [89], Restler [33], RestTestGen [86] Schemathesis [60], and Tcases [14].

For the testing of REST APIs, different studies having been carried out, including for example robustness testing [41], security testing [40], handling of inter-parameter dependencies [73, 74], generation of realistic test inputs [84], carving API tests from GUI interactions [90], and defining test coverage criteria [75]. For a more in-depth overview of such work, see our survey on this topic [53].

When dealing with under-specified API schemas, Kim et al. [63] provided a Natural Language Processing (NLP) approach for OpenAPI. Although a schema can be underspecified, it might still have natural language descriptions. For example, although a constraint might be not formally defined, the authors of the schema might still have added an informal description of such a constraint in the text comments. NLP techniques can then be used to analyze such text and then to extend the schema with extra inferred constraints (if any is present). One advantage of such a technique is that it is independent of the fuzzer, as it works directly to extend the given OpenAPI schema.

To the best of our knowledge, EvoMaster [20] is the only REST API fuzzer that can support both white-box and black-box testing of REST APIs. With the recent studies conducted for comparing the state-of-the-art REST API fuzzers [64, 93], EvoMaster in white-box mode achieved the best performance in code coverage and fault detection.

REST is not the only approach to define web services. Others are for example GraphQL [5] and different types of RPC frameworks (e.g., gRPC [6]). However, in contrast to REST [53], not much work has been done in the research literature on testing these other kinds of web services (e.g., [36, 61, 85, 91, 94]).

Search-based software testing (SBST) [15, 59] has been shown to be an effective technique to automatically generate test cases. In SBST, the problem of generating adequate test suites and fault-revealing test cases can be reformulated as an optimization problem. Examples include unit testing of Java software with open-source tools like EvoSuite [43], and testing of mobile applications with the Sapienz tool at Facebook [16].

When doing white-box testing, different techniques are used to define heuristics to smooth the search landscape. The most common in the literature of SBST is the so-called Branch Distance [65]. Given a boolean predicate in the code of the system under test (SUT) (e.g., x==y+10), the branch distance provides a heuristic value (e.g., minimize the value of |x-(y+10)|) to guide the search towards satisfying the boolean predicate.

Unfortunately, a common issue is the so-called flag problem [35], where the branch distance is not able to provide any gradient. For example, consider a scenario where our target program is dealing with string operations returning booleans [18], like comparisons of two strings for equality. By default, a boolean predicate like x.equals(“Hello”), would be just a flag, returning true or false. In such a scenario, the search algorithm has no guidance whether x is getting closer to become “Hello”. Inputs such as “ello” and “foo” will be considered indistinguishable. Flags in the code might depend on string operations [18], loop assignments [34, 37], nested predicates [78], calls to boolean functions [67, 68, 88] and non-integer comparisons [67].

An approach to address this issue is to transform the code of the SUT to improve the fitness function, using so-called Testability Transformations [58]. A testability transformation modifies the original SUT to produce a new version of the same SUT, but more amenable to the test generation process. This change might not necessarily preserve its original semantics, but it must “preserve test sets that are adequate with respect to some chosen test adequacy criterion” [55]. As an example, if a test generation targets a single goal (e.g., a specific branch in the SUT), a non-semantics preserving transformation might slice (i.e., remove) some lines/branches to alleviate a test generator. In contrast, semantics preserving testability transformation might not (by definition) affect the chosen test adequacy criteria. As an example, a semantic preserving testability transformation could replace the original boolean predicate x.equals(“Hello”) with a computation of a string distance, like the edit distance (number of insertions, deletions or changes to transform a given string into another) [18]. If such a transformation was applied, comparisons to “ello” and “foo” in x.equals(“Hello”) will no longer be considered indistinguishable by the heuristic.

Different transformations have already been proposed [55, 56], mainly to deal with (but not restricted to) flag conditions [18, 34, 35, 37, 54, 67, 67, 68, 78, 88]. Testability transformations have also been proposed to generate pseudo-oracles [77] (which can be helpful to detect numerical inaccuracies and race conditions). Furthermore, besides search-based test generation, slicing-based testability transformations can also be useful to improve Dynamic Symbolic Execution test generation [39].

EvoMaster performs whole test suite generation (as it is done by EvoSuite [45]) at the system level. In other words, it simultaneously targets all goals within the SUT. In this scenario, many of the aforementioned non-semantics preserving transformations would not be applicable as they would lead to generating test cases that do behave accordingly in the original SUT. For this reason, a main difference in [27] (i.e., EvoMaster \(^O\) ) and in this work (i.e., EvoMaster \(^N\) ), is that all of the proposed testability transformations must preserve the semantics of the SUT.

In this paper, one of the scientific contributions is to provide method replacements with SBST heuristics for several more APIs in the JDK and common libraries. This is a direct extension of what first presented in [27] (recall Section 2.4), and currently available in EvoMaster \(^O\) . The details of those new method replacements for EvoMaster \(^N\) will be discussed in the next sections.

When a new method replacement is designed, up to three different tasks need to be considered (depending on the method):

In EvoMaster \(^O\) , each testing target (e.g., lines and branches) we try to optimize for has a heuristic value \(h \in \left[0,1\right]\) . The value \(h=1\) means that the target is covered (equivalent to \(d=0\) ). Note that, internally, EvoMaster \(^O\) uses h instead of d to better handle the simultaneous optimization of many targets (e.g., with the MIO algorithm [22]). Given a branch distance value d, h is computed as \(h= b + \left(1 - b\right)\frac{1}{1+d}\) , where b is a small constant, e.g., \(b=0.1\) (see [27] for full details and rationale for these choices).

Note that, for reasons of space, we cannot go here into all the low level technical details of each of these new method replacements designed for EvoMaster \(^N\) . Those are thousands of lines of code, to deal with many edge cases (e.g., how to deal with instances of IdentityHashMap which does not use equals for comparisons, or how to detect at runtime maps that have \(\Omega (n)\) instead of \(O(log n)\) complexity for containsKey, which would result in computational bottlenecks when calculating the branch distances). What we present here is high level descriptions. For full details, we refer the reader to our open-source implementation on GitHub [2], specifically to version 1.6.1, which is stored on Zenodo [29] for long term storage. In particular, most of the implementation is under the package org.evomaster.client.java.instrumentation.coverage.methodreplacement.

In [27] we presented some initial work for EvoMaster \(^O\) to handle under-specified OpenAPI schemas (recall Section 2.5), which has been now extended here in EvoMaster \(^N\) by dealing with more classes and libraries (Section 4.1.4). However, still such work would only be able to deal with Spring and JEE/Jakarta, and, even for those, not all cases could be handled. Let us consider the snippet in Figure 4, coming from the SUT proxyprint used in our empirical analysis (Section 5). This is one of the cases of open problems that was discussed in detail in [93]:

Creating a method replacement for request.getParameterMap() would not help much here, as, at that point in time when it is called, there is no information yet on the parameter name mc_gross that is going to be read later on in the SUT’s business logic. There is the need for a more general solution that is able to handle also these cases.

Our novel solution in EvoMaster \(^N\) works as follows. First, when we make HTTP calls, we add a “fake” HTTP header (e.g., x-EMextraHeader123) and a “fake” query parameter (e.g., EMextraParam123). Then, each time any collection (e.g., lists, sets and maps) is queried with a key/value X, we check if such collection contains our extra header or parameter names. All data structures are already instrumented (e.g., recall Section 4.1.1) to enable heuristic computations and taint analysis, so this is just an extra check done inside those method replacements. Then, if there is a match, we check if X is an already known header/parameter name (e.g., from the OpenAPI schema definition). If not, then there might be a good chance that we are in a case like in Figure 4. The intuition here is that an HTTP server/framework would read all incoming headers/parameters and put them in a data structure, and then check for specific names in such data structure based on what the business logic of the SUT needs (e.g., mc_gross in this case). If that happens, then at the next fitness evaluation, we replace the name of the fake header/parameter with X, with a randomly initialized or tainted value. Then, with taint-analysis on the input of Double.valueOf we can further infer in the following fitness evaluation that the value of X should be turned into a double. Besides dealing with method calls on collections (e.g., search for a key in a map), we also check for these extra headers/parameters in every string comparison (e.g., in String.equals()).

This approach is fully automated, and it does solve the problem like in the example in Figure 4. However, there are two important aspects to consider here. First, it is performance. Sending extra query parameters and headers has a negligible cost, but checking for string matching might not, especially not on large collections. This cost can be justified if the OpenAPI schema is underspecified. However, it would be a clear performance loss if it is not. But we cannot really know if there is any missing header or parameter in the schema before applying our technique. What can be done is to try to minimize its computational cost. We apply two simple heuristics. First, the sending of extra parameters and headers is done only for a short period of time (e.g., currently 10% of the search budget, like for the first 6 minutes if the search budget is 1 hour). Second, if the analyzed collection is too large (e.g., more than 16 elements), then it would be unlikely that it represents the storing of HTTP headers or query parameters. If so, in these cases we simply skip any string matching checks in the instrumented method replacements.

Another rather peculiar issue is that, as we found out during some preliminary experiments, there might be some special headers automatically handled by the enterprise framework used by the API. For example, in older versions of Spring the HTTP filter HiddenHttpMethodFilter was active by default. This would handle a special query parameter named _method, to enable to change the HTTP verb of incoming requests. For example, a user could make a POST HTTP request with a query part ?_method=PUT to tell Spring to consider the request as a PUT. This feature comes from old time requirements, before the widespread use of JavaScript and AJAX, when it was possible to do only GET and POST requests from a HTML page (e.g., by clicking on a <a> link or submitting a <form>). What happened here is that our technique would automatically discover such hidden parameter _method, and use it in the evolving test cases. This provides no benefit to the search, and actually just leads to plenty of useless tests that return a 405 HTTP status code (“Method Not Allowed”), e.g., when sending random strings such as ?_method=Pfre5. The solution here is to simply ignore any newly discovered parameter that is named _method.

Data in an SQL database can have constraints, which are checked each time a new entry is added. Constraints could be as simple as stating that the entries of a column are unique, or that they might or might not be nullable. However, there might be more sophisticated cases, in which custom constraints can be written with CHECK directives in the SQL table schema.

For a white-box fuzzer, it is important to be able to create data directly into the SQL databases as part of the fuzzing process. This is needed to be able to test scenarios when the same SQL database is used by more than one application (e.g., in producer/consumer scenarios, or when data is populated by external batch jobs and the tested API is read-only). Also, it might improve the readability of the generated tests if some specific needed data in the SQL database could only be obtained with a long sequence of HTTP calls to the API. For this reason, EvoMaster \(^O\) is able to automatically generate data into SQL databases as part of the search [26]. During the search, it analyzes all executed SELECT commands, to see which ones return no data. In those cases, it will automatically insert data in the queried tables, with the fitness function aiming at solving the constraints in the WHERE clauses of those empty SELECTs [26]. Each test case will then be enhanced with extra initializing actions for the database, whose inputs (i.e., the data to insert) will be evolved during the search in the same way as any other element of the HTTP requests [26].

As adding invalid data would be pointless, as the SQL database would just straight-up reject them, when we insert SQL data in EvoMaster we make sure that all (linear) constraints are satisfied. However, there might be cases in which this is not enough. Figure 5 shows a snippet from the class VerificationAppSession in the SUT cwa-verification-server in our case study (Section 5). This class is marked with the JPA @Entity notation, and it is used to map data from the SQL table app_session. One problem here is that the numeric variable tan_counter in that table is nullable, i.e., NULL is a valid value for it. However, that entity class declares it with the primitive Java type int, instead of the nullable Java type Integer. If there is a null value for that variable, the parsing of that entity would crash, throwing an exception. Regardless of any JEE/Jakarta constrain annotation (e.g., @Min and @Max), primitive vs. nullable types do implicitly define further constraints. In this particular case, there is a mismatch of constraints between the SQL database and its ORM representation in the API. This is a fault, but whether the fault is in the API (i.e., wrong constraint mapping) or in the SQL database definition (i.e., underspecified constraints) is something that only the authors of that API can really comment on.

Another issue can be seen for columns such as teletan_type. In the database, it is declared as a varchar(10), i.e., a string of at most 10 characters. However, the JPA entity defines a further constraint that such string is actually an enumeration, where only a limited set of values is acceptable. In particular, the enumeration TeleTanType contains only the values TEST and EVENT. A random string in that column would crash the JPA parsing of entries from such table.

Our novel solution in EvoMaster \(^N\) for this problem is as follows. First, every single time a class is loaded into the JVM, we check if it is a JPA entity (e.g., by checking if the class has the @Entity annotation). If so, we analyze all JEE/Jakarta constraints defined on it (recall Section 4.1.5), including any implicit nullability check based on primitive types, and any enum declarations. Then we compare these sets of constraints with the constraints derived directly from querying the SQL database [26]. For this, we need to identify the corresponding table in the SQL database. We follow the same algorithms as done in JPA implementations such as Hibernate to resolve table and column names, which might be the same as the Java class/field names, or overridden with annotations, like VerificationAppSession vs. app_session (see Figure 5). If the JPA entity provides more constraints, then we use those constraints when inserting data into the database. For example, we would not insert a NULL into the column tan_counter. However, with a small probability, for robustness testing any now and then we also insert data into the relational database that does not satisfy these extra constraints.

In a backend system, it is possible to schedule tasks at precise time intervals. For example, in the API catwatch used in our case study (Section 5) a task is executed at 8:01am each day, to fetch project data from GitHub. In the API ocvn-rest data is imported each day at 3am and 9pm. These background tasks are set with Spring annotations such as @Scheduled, and are executed independently from the HTTP requests to the API.

These background tasks introduce a few complications for testing. First, generated tests could become flaky if the background tasks have side-effects (e.g., adding or deleting records from a database). Second, it might invalidate empirical comparisons of fuzzers (and their different settings) when measuring code coverage metrics. It is not uncommon that experiments for this kind of system takes days to run. For example, when running experiments with a fuzzer X at 2:59am on ocvn-rest, we might get better results than a fuzzer Y at 4am simply because the code executed by the background task would contribute to the measured code coverage.

This is a particular problem for black-box fuzzers, as there is no programmatic way (at least in Spring) to disable all the scheduled tasks when starting the API for testing. The API would need to be manually modified to enable disabling background tasks during fuzzing experiments. To handle this threat to the validity of the experiments, with a white-box approach using bytecode instrumentation in EvoMaster \(^N\) we simply remove any @Scheduled annotations when classes are loaded into the JVM (and so the tasks are not executed).

To evaluate our novel techniques presented in Section 4, we used all the 14 RESTful APIs present in the EMB corpus [32] at the time of these experiments, in particular version 1.6.1 [31]. EMB is a corpus of Web APIs (including GraphQL and RPC APIs), which we have collected and extended each year with new APIs since 2017. EMB includes the EvoMaster drivers for enabling white-box fuzzing on all of these APIs. To obtain more generable results, besides using open-source APIs, we also included in our experiments one API from one of our industrial partners. For this paper, to refer to this API we use the fictional name ind0. Table 1 shows some statistics on these 15 RESTful APIs, including number of source files, lines of code and number of REST endpoints in each of these APIs. Note these code statistics count only what is present in the business logic of those APIs. Statistics on code in third-party libraries (e.g., HTTP servers) are not included here.

EMB provides APIs of different size and complexity, coming from different domains, covering a variegated set of APIs needed for scientific experimentation [32]. There are two artificial APIs aimed at studying how to deal with numeric (rest-ncs) and string (rest-scs) constraints. The other 12 APIs are all taken from GitHub: some APIs come from public administrations (e.g., ocvn), or are widely popular tools that provide a REST interface (e.g., languagetool). A full description of these APIs can be found at [3, 32], including all the source code, build scripts, and links to the original repositories these APIs were collected from throughout the years.

In this paper, we have carried out two different sets of experiments. In the first set, we considered and compared six different configurations of EvoMaster \(^N\) , namely:

Fuzzing sessions were run for 1 hour each. To take into account the randomness of search-based fuzzing, each experiment was repeated 10 times. In total, considering six configurations and 15 SUTs, this required \(6 \times 15 \times 10 = 900\) hours, i.e., 37.5 days of computation. To be able to run all these experiments in reasonable time, they were run in parallel (15 at a time) on an HP Z6 G4 Workstation with Intel(R) Xeon(R) Gold 6240R CPU @2.40GHz 2.39GHz, 192 GB RAM, with 64-bit Windows 10 OS.

Ideally, the impact of each single new technique (e.g., each single testability transformation presented in Section 4.1) should be studied in isolation. However, that would have a non-trivial engineering cost (e.g., to support enabling only specific subsets of testability transformations in EvoMaster \(^N\) ’s code instrumentator), as well as making the running of all the experiments unfeasible in reasonable time. Using six configurations for the experiments was a viable compromise.

Still, it is important to check that each single technique is useful in its own right. EvoMaster is an industry-ready tool (e.g., used daily in large companies such as Meituan on hundreds of microservices [94, 95]), and particular care is taken to verify the correctness of its components. For this goal, EvoMaster has a sophisticated system of end-to-end tests, where it is run on a set of artificial API examples, carefully crafted to study and verify each single of its features [30]. These tests are automatically run in CI (e.g., GitHub Actions). To verify the correctness of our novel techniques, in EvoMaster \(^N\) we created artificial APIs and new end-to-end tests for all of them. An example is ValidEMTest,¹⁴ used to verify the handling of bean validation presented in Section 4.1.5. All these new system level tests pass, and are currently part of the daily CI testing of EvoMaster. As discussing all these end-to-end tests would take a considerable amount of space, we refer the interested reader to the code repository of EvoMaster (e.g., the module e2e-tests).

Based on the results of these experiments, after analyzing its results, to get more insight on a potential issue, we carried out a second set of experiments using a single SUT, namely ocvn. In this case, fuzzing sessions were run for 10 hours instead of just 1 hour. But only two settings were considered, Base and All, with 10 repetitions. This took \(10 * 10 * 2 = 200\) hours, i.e., 8.3 days. Experiments were run on the same hardware and configurations of the first set of experiments.

In total, the computation cost of our experiments was \(37.5 + 8.3 = 45.8\) days. Note that, although experiments can be run in parallel, there is a limit on how many can be parallelized, considering the specification of the employed hardware. For example, each single experiment requires to run few processes, like EvoMaster itself, the tested API and potentially its databases (e.g., Postgres through Docker) if any is in use. This can take a significant amount of OS resources, such as RAM and OS-level threads. Overloading the target machine with too many experiments in parallel could have a negative impact on time-based comparisons.

Table 2 shows the results in detail for All configuration compared to Base. We follow the statistical guidelines from [25], reporting p-values of Mann-Whitney-Wilcoxon U tests and Vargha-Delaney standarized \(\hat{A}_{12}\) effect sizes. Results are compared in term of line coverage and detected faults. Detected faults are based on 500 HTTP status codes (server errors) and mismatches of the responses from the OpenAPI schemas for each distinct endpoint (more details on what kind of faults can be detected this way can be found in [71]). Note that there exist also other kinds of metrics that could be used for comparisons, such as for example branch and path coverage. For the sake of simplicity and space, we just report the two most common metrics used by practitioners in industry.

As our novel techniques in EvoMaster \(^N\) could introduce some non-trivial computation overhead in the fitness function, in Table 2 we also report the number of HTTP calls done during the search. This could give some insight on the computational cost of our techniques. The less efficient (i.e., more computationally expensive) a technique is, the fewer number of HTTP calls can be done during the search budget (e.g., 1 hour in our case). However, this needs to be analyzed with care, as the cost of each HTTP call is not the same. For example, a call that returns immediately a 400 status code (user error) due to an invalid input would be faster than a successful 200 code call that executes large parts of the API’s code, including interactions with external databases. Better heuristics that lead to cover more code might result in executing HTTP calls that are more expensive to run, regardless of computation cost of the heuristics themselves.

To see the results of each of the six configurations in isolation, average results are summarized in Table 3. Detailed statistical comparisons of each configuration with Base are reported in the Appendix (Tables 4–7).

From these results, for line coverage we can see an average improvement of \(+1.7\) % (median \(+4.8\) %) over the 15 APIs. Results are statistically significant for only six APIs, with no statistically worse results. On these APIs improvement are either “small” (e.g., less than 1% for ocvn, rest-news and restcountries), “medium” ( \(+4.6\) % for catwatch), or “large” (i.e., \(+10.2\) % for cwa-verification and \(+10.7\) % for ind0). Note that the terms used here small, medium and large are subjective, and so are technically arbitrary.

Regarding fault detection, statistically better results are found in three APIs. However, there are statistically worse results for one API, namely ocvn. There is a large number of detected faults in ocvn (e.g., more than 500), which is related to the large number of endpoints in this API (i.e., 258, recall Table 1). This outlier significantly impacts the statistics over the whole 15 APIs. Although the average decreases from 62.0 to 60.4, the median increases from 20.0 to 21.0. This is a special case in which the average value is roughly three times the median one.

We can make a few hypotheses on why code coverage was significantly improved in only six out of 15 APIs.

In the first set of experiments, our novel techniques achieved better results in all but one case, namely ocvn. The computation cost of our novel techniques is not negligible on this API, as the number of HTTP calls is reduced from 136k to 103k. The question here is what would happen if the fuzzing would be run for longer. The choice of 1 hour for the experiments is technically arbitrary, as arbitrary as 42 minutes or 24 hours.

After running for a certain amount of time, a search algorithm will be stuck on an optimum (either global or more likely a local one). Running the search for longer would not help much if the algorithm cannot escape from the local optima. The search performance would reach a so called plateau, and stagnate. There is decades of research effort on addressing this problem, for example with techniques such as fitness sharing [83] to increase population diversity in population-based evolutionary algorithms. The tradeoffs between exploration and exploitation of the search landscape applied by a search algorithm do impact how, when and what type of local optimum would be reached. In this regard, the MIO [22] algorithm used in EvoMaster applies few of these techniques. Still, without a fitness function that can provide gradient to the search, such search would degenerate in a so called random walk on “fitness plateau” once a local optimum is reached, preventing further improvement. For all these reasons, the comparisons of two algorithms (or algorithm variants) might give very different results based on the used search budget, especially if using different fitness functions. A more sophisticated and expensive fitness function could give worse results for “low” time budgets, and better for “higher” budgets.

Figure 6 shows the results of the second set of experiments. Those focus only on the API ocvn, with a search budget of 10 hours. For low search budgets, the All configuration gives worse results. With increasing budget, both configurations Base and All improve in performance, although Base does plateau to lower values. For line coverage, All takes over after 1 hour. For number of detected faults, it takes over after 9 hours.

So, based on these results, which of the two configurations is “better”? In a software engineering context, it all depends on how common those time settings are in practice among practitioners. Unfortunately, we do not have such information. Most of the work done in the literature on automated testing of REST APIs has not considered its use among practitioners [53]. This is possibly because there is currently no popular REST API fuzzer widely used in industry. Even when considering mature fuzzers in other popular domains, such as data parsers, there is not much research work aimed at studying how practitioners use those fuzzers [80].

It can be speculated that, considering the life-time of the development of a SUT which can be in years, fuzzing sessions could be relatively long. This is particularly the case if the fuzzing can be integrated in remote CI servers (e.g., like done in [38] for unit test generation), especially if the outcome of a previous fuzzing session can be reused for following sessions (e.g., using different kinds of test seeding strategies [82]), even if parts of the tested code have been modified. Easy-to-use integration of fuzzers into CI servers is a common request among users of fuzzers [80]. In those scenarios, more expensive techniques could pay off better in the end. Still, hybrid approaches could be designed: e.g., simple, cheap techniques at the “beginning” (e.g., first few sessions on CI), followed by more expensive techniques afterwards.

Another point to consider is that, as we got better results with a 10 hour budget for ocvn, this might (or might not) happen as well for the other SUTs in our case study. Without empirical experiments, this is not possible to tell. Furthermore, even if there is no difference at 10 hours, perhaps there could be a difference at 24 hours, or more. Testing all different kinds of large test budgets is unfortunately not a viable option for academic experimentation, e.g., our experiments already took more than 45 days if run in sequence.