Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

As stated previously, each tool can self-report how many faults they find, but how such fault numbers are calculated might be quite different from tool to tool, making any comparison nearly meaningless. Manually checking (possibly tens of) thousands of test cases is not a viable option either, especially when they are generated in different programming languages (e.g., Java and Python).

Some authors, such as Kim et al. [53], use the number of unique exception stack traces in the SUT’s logs as a proxy of detected faults. On the one hand, a tool that leads the SUT to throw more exceptions could be considered better. On the other hand, using such a metric as proxy for fault detection has many shortcomings, such as the following: (1) not all exceptions reported in the logs are related to faults; (2) not all SUTs actual print any logs by default (e.g., this is the case for several SUTs in our study); and (3) crashes (which lead to responses with HTTP status code 500) are only one type of fault in RESTful APIs detected by these fuzzers [57], where, for example, all faults related to response mismatches with the API schema would leave no trace in the logs. For all of these reasons, we did not do this kind of analysis on the logs, as we do not believe that they provide much more information compared to line coverage—especially considering that the infrastructure to do such analysis would need to be implemented, which might not be a trivial task.

An alternative that we considered was to build a proxy HTTP server, to be used between the fuzzer and the SUT. With such a proxy, it would be possible to record all responses from the SUT—for example, to log all the endpoints returning a 500 HTTP status code—regardless of the employed fuzzer. We discarded this idea for two main reasons: (1) it could introduce a not trivial delay to all test case evaluations, which could bias the experiments (slower fuzzers would get a relative advantage if each HTTP call takes artificially longer to execute because going through a proxy), and (2) such an approach would not be able to distinguish between different faults triggered in the same endpoint.

Another alternative was to use some Mutation Testing [51] tool. However, we are not aware of any such tool aimed at system testing (i.e., they all seem to be tailored for unit testing). Furthermore, there could be major technical challenges in using a Mutation Testing tool when the generated tests and the SUT are written in different programming languages (e.g., Python and Java). Designing a novel Mutation Testing tool for RESTful APIs would be useful, but that is outside the scope of this work.

Existing work on comparing fuzzers in different domains, such as for data parsers (e.g., [41]), use SUT crashes as an oracle to detect faults. Those are easy to identify (the SUT is no longer running), although smart techniques need to be used to distinguish possible different faults in the same SUT when it crashes. This does not happen in RESTful APIs, as crashes in the business logic do not crash the HTTP servers they run on. The HTTP servers catch these exceptions and return a HTTP response with status 500. Additionally, as already stated, this is only one type of fault that can be automatically detected in RESTful APIs [57]. Different kinds of automated oracles have been defined in the literature [48]. Only focusing on that type of fault could be too restrictive.

In the end, for a more fair comparison on fault detection among the different fuzzers for RESTful APIs, we created an artificial API, with 10 distinctive types of artificial faults in it (as mentioned in Section 4.1). We then ran each black-box fuzzer on it and manually evaluated their generated test cases to check if the faults were identified.

The API is designed in a way that triggering faults in it should be trivial. In other words, just calling an endpoint would trigger a fault, regardless of any input. The goal here is to check if the fuzzers are able to identify if the obtained responses are indeed faulty. In other words, we are empirically checking which range of automated oracles each fuzzer employs.

Regardless of the achieved code coverage, a fuzzer could be extended with more automated oracles from the literature. But how to integrate them is not necessarily trivial. For example, consider robustness faults, in which wrong data (according to the schema) is sent, and a fuzzer then verifies if the HTTP response has status code in the 4xx family (i.e., representing a user error). Sending invalid data might negatively impact the efforts on maximizing code coverage in the SUT, as input validation is done in the first layer of the API. How often and when to send invalid inputs to check this type of automated oracle is hence a non-trivial design decision in a fuzzer.

In our artificial API, we have injected the following 10 different types of faults for which an automated oracle can be defined. Note that this is not meant to be an exhaustive list of all possible faults that can be automatically detected with an oracle. For example, we have not considered security-related faults (e.g., based on access policies). This selection is only meant as a starting point, to shed more light on this important issue.

The 10 fault types are the following:

We ran each of the seven black-box fuzzers on the artificial API. As the endpoints of such APIs are trivial to cover, we ran each tool only once, for just 6 minutes. Then, we manually checked all the generated test cases to see which of the 10 faults were correctly identified.

Table 7 shows the results of these experiments. On this API, RestCT did not manage to generate any test case, as it seems like it has issues when dealing with endpoints not having query or path parameters. Although the API is trivial, RESTest and RestTestGen did not manage to achieve full coverage, in contrast to the other four fuzzers.

Of the 10 faults, only #1 is reliable found by most tools. Fault #2 was found only by bBOXRT. For fault #3, it was found only by EvoMaster and Schemathesis. None of the other 7 faults was properly identified by any of the fuzzers. However, there are some clarifications to make:

The results of this experiment clearly show that current fuzzers only detect a small subset of possible fault types in RESTful APIs. As reported in a recent survey by Golmohammadi et al. [48], most fuzzers are able to find hundreds/thousands of faults in existing APIs. This is typically done by detecting 500 HTTP status codes (i.e., like in fault #1). However, not all faults have the same severity. For example, many cases of the 500 status code is simply wrong handling of invalid inputs [57]. Returning a 500 instead of 400 on wrong inputs is not as serious as a fault like #10. In this case, this latter type of fault can have catastrophic consequences, as it not only can corrupt the data of the API but also would be hard to reproduce and debug due to the non-determinism and low frequency of re-executing idempotent methods on the HTTP stack.

From the tool logs, at least four common issues are worthy of discussion. First, OpenAPI/Swagger schemas might have some errors (e.g., this is the case for cyclotron, disease-sh-api, cwa-verification, features-service proxyprint, and ocvn-rest). This might happen when schemas are manually written, as well as when they are automatically derived from the code with some tools/libraries (as those tools might have faults). In these cases, most fuzzers just crash, without making any HTTP call or generating any test case. It is important to warn the users of these issues with their schema, but likely fuzzers should be more robust and not crash (e.g., endpoints with schema issues could be simply skipped).

The second issue can be seen in languagetool. Most fuzzers for RESTful APIs support HTTP body payloads only in JSON format. However, in HTTP, any kind of payload type can be sent. JSON is the most common format for RESTful APIs [63], but there are others as well, like XML and the application/x-www-form-urlencoded used in languagetool. From the results in Table 2, it looks like only EvoMaster supports this format. On this API, EvoMaster achieves between 26% and 35.1% code coverage, whereas no other fuzzer achieves more than 2.5%.

The third issue is specific to scout-api, which displays a special case of the JSON payloads. Most tools assume a JSON payload to be a tree—for example, a root object A that can have fields that are object themselves (e.g., A.B), and so on recursively (e.g., A.B.C.D and A.F.C), in a tree-like structure. However, an OpenAPI/Swagger schema can define objects that are graphs, asis the case for scout-api. For example, an object A can have a field of type B, but B itself can have a field of type A. This recursive relation creates a graph that needs to be handled carefully when instantiating A (e.g., optional field entries can be skipped to avoid an infinite recursion, which otherwise would lead the fuzzers to crash).

The fourth issue is related to the execution of HTTP requests toward the SUTs. To test a REST API, the fuzzers build HTTP requests based on the schema and then use different HTTP libraries to send the requests toward the SUT—for example, JerseyClient is used in EvoMaster. By analyzing the logs, we found that for several case studies, some fuzzers seem to not have problems in parsing schemas, but they fail to execute the HTTP requests. For instance, javax.net.ssl.SSLException: Unsupported or unrecognized SSL message was thrown when RestTestGen processed realworld-app with OkHttpClient. For js-rest-ncs, js-rest-scs, and ind0, we found that 404 Not Found responses were always returned when fuzzing them with RESTler v8.5.0. By checking the logs obtained by RESTler, we found that it might be due to a problem in generating the right URLs for making the HTTP requests. For the SUT whose basePath is /, RESTler seems to generate double slash (i.e., //) in the URL of the requests. However, whether to accept the double slash to match a real path depends on the SUTs (i.e., js-rest-ncs, js-rest-scs, and ind0 do not allow it). In Figure 2, we provide the logs obtained by RESTler, representing the processed requests that contain the double slash and responses returned by js-rest-ncs (Figure 2(a)) and rest-ncs (Figure 2(b)).

For each SUT, we manually ran the best (i.e., highest code coverage) test suite (which are the ones generated by EvoMaster) with code coverage activated. Then, in an IDE, we manually looked at which branches (e.g., if statements) were reached by the test case execution but not covered, as well as looking at the cases in which the test execution is halted in the middle of a code block due to thrown exceptions. This is done to try to understand what are the current issues and challenges that these fuzzers need to overcome to get better results.

To be useful for researchers, these analyses need to be of a “low level”, with concrete discussions about the source code of these APIs. No general theories can be derived without first looking at and analyzing the single instances of a scientific/engineering phenomenon. As such software engineering discussions might depend on different groups of readers who might target different levels of detail in the software, or considering readers who might not be currently active in the development of a fuzzer, here we provide only a summary. Full analyses for each SUT are provided in Appendix A.

Table 9 summarizes all identified main issues for each SUT. There can be many reasons a higher coverage was not achieved. Here, based on our analyses, we have categorized six main issues:

Note that these are only the main issues we currently face. Solving them does not imply maximizing code coverage. As more code is executed, more issues could be identified. A concrete example is cwa-verification. Such an API does connect to an external service, but the code doing this is currently not executed by any of the fuzzers. Dealing with external services would hence currently give no improvement on cwa-verification. However, when the other issues in cwa-verification are solved (recall Table 9), then dealing with external services will be needed to improve coverage results further.

Based on the analyses of the logs and tests generated for the 19 APIs, some general observations can be made:

Many of the challenges we have identified (e.g., dealing with databases and underspecified schemas) are specific for RESTful APIs, although they would likely apply to the fuzzing of other types of web services as well (e.g., GraphQL [39, 40] and RPC [72, 73]). These challenges would not be meaningful in different testing contexts (e.g., unit test generation or fuzzing of parser libraries).

Issues with providing manual auth configurations would likely apply to any kind of software that needs authentication information. Dealing with constraints on string data is likely the most general challenge, as it applies as well, for example, to unit testing.

This means that solving these challenges would not only help in the testing of RESTful APIs, but it likely can have positive effects on other testing domains as well.

With an average line coverage of 50%, catwatch can be considered a non-trivial API. The main issue is that this SUT makes a call to an external service (more specifically, to the GitHub APIs to fetch project information). But such call seems to get stuck for a long time (and possibly timeout), which leads to none of the code parsing the responses (and do different kind of analyses) being executed. Calling external services is a problem for fuzzers. External services can go down or change at any time. They can return different data at each call, making assertions in the generated tests become flaky. Although testing with the actual external services is useful and should be done, the generated tests would likely not be suitable for regression testing. This is a known problem in industry, and there are different solutions to address it, like mocking the external services (e.g., in the JVM ecosystem, WireMock [23] is a popular library to do that). Enhancing fuzzers to deal with mocked services (e.g., to setup the mock data they should return) is going to be an important venue of future research.

On the cwa-verification API, achieved coverage is less than 50% (i.e., 46.9%). There are two interesting cases to discuss here, which have major impact on the achieved coverage.

First, Figure 3 shows a snippet of code for one of the endpoint handlers (i.e., External TanController, but the same issue happens as well in ExternalTestStateController and InternalTanController). Here, the condition actual.isPresent() is never satisfied, which leads to missing a large part of the code. A token is given as input as part of the body payload, and then a record matching such token in the database is searched for. However, none is found, and EvoMaster is unable to create it directly. In theory, a case like this should be trivial to handle with SQL support [32], but this was not the case. The reason is the peculiar properties of such token. This type of token has the given constraint in the OpenAPI schema:

This results in the token having a length of 36 characters. EvoMaster has no problem in sampling strings that match such a regular expression, like bfe8b80b-dca1-458d-B312-96ecae446be0. However, such constraint is not present in the SQL database, where the column for these tokens is simply declared with the following:

When EvoMaster generates data directly into the database, they would be random strings that do not satisfy such regular expression. Again, this should not a problem thanks to taint analysis [33]. The reason it is not working is due to the length of the string, which is 36 characters. By default, EvoMaster does not generate random strings with more than 16 characters. Even if with taint analysis we could inject the right string into the database, currently this does not happen due to 36 being greater than 16.

Note that having a constraint on the length of strings is essential. Sampling unbound random strings with billions of characters would have too many negative side effects. The choice of the value 16 is arbitrary: it could have been higher or lower. Still, whatever limit is chosen, an SUT could need strings longer than such a limit, as happens in this case for cwa-verification.

A solution to address this issue is to distinguish between two different max-length limits: a hard one that should never be violated (e.g., a constraint in the OpenAPI or SQL schemas) and a soft one (e.g., 16). The soft limit would be used when sampling random strings but could be violated in some specific circumstances (e.g., in taint analysis).

The second interesting case to discuss is present in the endpoint handler Internal TanController, shown in Figure 4. Here, an HTTP header with the value TELE_TAN_TYPE_ HEADER=”X-CWA-TELETAN-TYPE” can be provided as input. However, such information is missing from the OpenAPI schema. Therefore, the object teleTanType is always null. This is an example of underspecified schema.

On the cyclotron API, there is not much difference between white-box and gray-box testing. There are at least three major issues worthy of discussion.

First, a non-negligible amount of code is executed only if some configuration settings are on. But those are off by default (e.g., like config.analytics.enable and config.enableAuth in config.js). All code related to these functionalities cannot be tested via the RESTful endpoints.

Second, like for several other APIs in this study, the schema here is not fully correct/complete. For example, Figure 5 shows a snippet for the handler of the endpoint /data/{key}/upsert, where the two fields data and keys are read from the input object in the HTTP body payload of the request. However, the schema has no formal definition about those fields, as they are just mentioned as a comment (see line 17 in Figure 6).

Third, the API does several accesses to a MongoDB database. Several endpoints start by retrieving data from the database, such as with commands like Dashboards.findOne({ name: dashboardName }). If the data is missing, no following code is then executed. But it is hard to get the right ID (e.g., dashboardName in this example) by chance. Although EvoMaster has support for SQL databases (i.e., to analyze all executed queries at runtime), it does not for MongoDB, and not for NodeJS (i.e., SQL support is currently only implemented for JVM).

On disease-sh-api, there is not much difference between EvoMaster (both black-box and white-box) and Schemathesis. After an analysis of the generated tests, practically all achievable coverage is obtained, as the remaining code is not reachable through the REST endpoints in the API.

First, the schema refers only to version v3 of the API, but in the source code there is still all the implementation of the endpoints for version v2. Those endpoints are not executable based on the information in the schema. Second, large part of the code deals with the fetching of disease records from online databases, and storing them in a local Redis instance. But such code is executed only via scripts and not from the RESTful endpoints. Note that when the experiments are run, the database was populated with a selection of valid data, as there is no REST endpoint to add it.

With an average line coverage of 81.8%, features-service can be considered one of the easier APIs. Quite a bit of code cannot be reached with any generated system test, such as getters/setters that are never called or some functions that are only used by the manually written unit tests. There are still some branches that are not covered, related to properties of data in the database. In other words, when a GET is fetching some data, some properties are checked one at a time, but to be able to pass all of these checks, a previous POST should have created such data. Figure 7 shows one such case, where the code inside the if statement does not get executed. Tracing how database data is impacting the control flow execution, and which operations should be called first to create such data, is a challenge that needs to be addressed.

On the gestaohospital-rest API, there is not much difference between white-box and gray-box EvoMaster (i.e., average coverage 39.5% vs. 39.4%; recall Table 8). Furthermore, RestTestGen and Schemathesis give better results than EvoMaster (recall Table 2).

One specific property of this API compared to most of the other SUTs in our empirical study is that this type of API uses a MongoDB database. In contrast to SQL databases, white-box EvoMaster has no support for NoSQL databases (like MongoDB). As this API does many interactions with the database (e.g., there are a lot of calls like service.findByHospitalId(hospital_id)), not much coverage is achieved if such accessed data is not present in the database, and the evolutionary search has no gradient to create it. In this regard, it seems that RestTestGen and Schemathesis do a better job at creating resources with POST commands and use such created data for their following GET requests. Still, although quite good, the coverage is at most 62.3% (measured with JaCoCo for Schemathesis), which means that there remain some challenges to overcome.

We could not analyze in detail the output of these tools in an IDE for the Java program gestaohospital-rest (e.g., using a debugger), as RestTestGen generated only JSON files with the descriptions of the tests (and no JUnit file), and the outputs of Schemathesis are in YAML for the VCR-Cassette format (which is mainly for Ruby and Python, where ports in Java for handling tests with VCR-Cassette format seem to have not been under maintenance for several years).

The closed-source API ind0 was provided by one of our industrial partners. Therefore, we cannot provide any code example for it, but we can still discuss it at a high level. This API is a service in a microservice architecture. Although in terms of size it is not particularly big (recall Table 1), it is the most challenging API in our case study. No black-box fuzzer achieves more than 8.2% line coverage, and white-box testing goes only up to 18.8% (on average).

Out of 20 endpoints, all but 1 endpoint require string inputs satisfying a complex regular expression, but such information is not present in the OpenAPI schema. Therefore, a random string is extremely unlikely to satisfy such constraint. Furthermore, the constraint is expressed in a @ annotation, which is handled by the Spring framework before any of the HTTP handlers are executed.

Thanks to taint analysis, EvoMaster can handle these cases [33], even when constraints are evaluated in third-party libraries (and not just in the business logic of the SUT). Still, there is quite a bit of variability in the results—that is, given the average coverage 18.8%, the standard variation is 7.0, with coverage going from a minimum of 11.9% to a maximum of 32.6% (out of the 10 repeated experiment runs). EvoMaster manages to solve these constraints, although not in all runs, as such strings not only get matched with a regular expression, but they then are checked further with other constraints. However, as the regular expression is exactly the same for all of these 19 endpoints, applied on a URL path variable with the same name, it should be possible to design strategies to re-use such information to speed up the search, instead of resolving the same constraints each time for each endpoint. Even with the run with the highest coverage of 32.6%, many endpoints are not covered due to this issue.

The artificial API js-rest-ncs uses numerical computation functions (e.g., Triangle Classification [25]) behind REST endpoints. As this is a re-implementation in JavaScript of rest-ncs, we will defer to Section A.14 for the analyses.

One thing to notice, however, is that the coverage values are higher for a very simple reason: the fetching of the OpenAPI schema is part of business logic of the API, and code coverage is computed for it (as the schema is inside a JavaScript file as a string inside an HTTP endpoint declaration), whereas for Java, the schema is handled as a JSON resource.

Similarly to the case of js-rest-ncs, js-rest-scs is a JavaScript re-implementation of rest-scs, which involved string-based computation functions. We therefore will defer to Section A.16 for the analyses.

As for js-rest-ncs, the fetching of the schema impacts the computed code coverage. However, for white-box EvoMaster, coverage is worse for js-rest-scs than for rest-scs. The reason is rather simple: the support for white-box testing of JavaScript code in EvoMaster is a much more recent addition [74], and not all features implemented for JVM (e.g., different forms of taint analysis [33]) are supported currently.

On the languagetool API, coverage is up to 39.5%. This is the largest and most complex API in our study (recall Table 1). This is a CPU-bound (e.g., no database) application, doing complex text analyses. This API has only two endpoints: /v2/languages, which is a simple GET with no parameters (and so it is trivially covered by just calling it once), and /v2/check, which is a POST with 11 input parameters. However, looking at the source code of ApiV2.handleRequest, it seems that there are some more endpoints, although those are not specified in the OpenAPI/Swagger schema. Without an extensive analysis of this SUT, it is unclear how much of its code would not be coverable due to this issue.

Following the execution from the entry point /v2/check, there are a few missed branches worthy of discussion. In Figure 8, the last statement crashes due to mapper.readTree throwing an exception (and so all statements after it cannot be executed). Here, when a form in an application/x-www-form-urlencoded payload is received, 1 out of the 11 parameters is treated as JSON data (i.e., the input called data). Such information is written in the schema as a comment, although the type is registered as a simple string. It is unlikely that a random string would represent a valid JSON object. Another issue is when entering in the class TextChecker. Several if statements refer to input parameters that are undefined in the schema, such as textSessionId, noopLanguages, preferredLanguages, enableTempOffRules, allowIncompleteResults, enableHiddenRules, ruleValues, sourceText, sourceLanguage, and multilingual. Dealing with underdefined schemas is a major issue with fuzzers, especially black-box ones. Technically, these can be considered as faults in the schemas, which should be fixed by the users. However, schemas can be considered as documentation, and issues in the documentation might receive less priority compared to faults in the API implementation. Still, as fuzzers heavily rely on such schemas, a more widespread use of fuzzers might lead to changes in industrial practice by giving compelling reasons to timely update and fix those schemas.

As there are tens of thousands of lines that were not covered in this SUT, there are many other missed branches that would be interesting to discuss in detail. However, it is unclear how related or independent they are from the aforementioned issues. Once the issue of dealing of underdefined schemas is solved, it will be important to re-run these experiments to identify which major issues remain.

On the ocvn-rest API, with black-box testing line, coverage was up to 35.3%, which is significantly higher than the 24.8% achieved by white-box testing. It is the second largest API in our study, following languagetool, based on the number of LOCs. However, in terms of endpoints, it is the largest, with 192 of them. It would not be possible, given the space, to discuss each of these 192 endpoints. To analyze the achieved coverage, we hence used the test suites generated with both white-box and black-box testing. These have hundreds of test cases. Even calling each endpoint just once would result in at least 192 HTTP calls in the generated tests. Considering the complexity of this SUT, maybe using a larger search budget over 1 hour could be recommended.

More than 35% of the codebase (packages org.devgateway.ocvn and org.devgateway. ocds.persistence) deal with connections to databases such as SQL and MongoDB, but such code does not seem to be executed. This might happen if the endpoint executions fail before writing/reading from the databases (e.g., due to input validation). The definitions of HTTP handlers in the package org.devgateway.ocds.web.rest.controller take up more than 30% of the codebase. But only half of it (i.e., around 15%) gets covered by the tests. Few endpoints (e.g., in the class UserDashboardRestController) were not covered because they require admin authentication, which was not set up for these experiments (recall the discussion about authentication in Section 4.3). The class CorruptionRiskDashboardIndicatorsStatsController defines several HTTP endpoints, but none of them appears in the OpenAPI/Swagger schema. Therefore, they cannot be called by the fuzzers.

Around 5% of the codebase seems to deal with the generation of Excel charts, but that code is not covered. Out of the 14 HTTP endpoints dealing with this functionality, they all fail on the very first line (and so a substantial part of the codebase is not executed, possibly much higher than 5%), which is a variation of the following statement with different string inputs as the second parameter:

Here, the language is given as input in the HTTP requests, which then gets matched with a regular expression. This makes the translationService.getValue call crash when the regular expression is not satisfied. White-box testing can analyze these cases, but it can be hard for black-box techniques if the information on such regular expressions is not available in the OpenAPI/Swagger schemas.

Another large part of the code that is not covered (around another 5%) is in the package org.devgateway.ocds.web.spring. However, most of this code seems to only be called by the frontend of the OCVN application (and the jar file for ocvn-rest does not include this type of module). From the point of view of the HTTP endpoints, this can be considered dead code, because it cannot be reached from those endpoints.

It might be surprising that white-box testing gives significantly worse results than gray-box testing, by a large margin (i.e., 10.5%). In a combinatorial optimization problem, this can happen when the fitness function provides little to no gradient to the search, generating the so-called fitness plateaus. Mutation operators that only do small changes to the chromosome of an individual (i.e., an evolved test case in this context) would have lower chances to escape from such local optima. EvoMaster has some advance mechanisms to address these cases, such as adaptive hypermutation [70]. However, it was not enough to handle ocvn-rest. The problem here is that most endpoints take a JSON object as input, with several constraints that must be satisfied. However, those constraints are not evaluated in the business logic of the SUT but rather in a third-party library. These constraints are not specified in the OpenAPI schema. Figure 9 shows an example, in which the JSON input is unmarshalled into an YearFilterPagingRequest instance. Because this input is marked with the annotation @Valid, its validity is evaluated before the method countTendersByYear() is called. If it is not valid, then the Spring framework returns an HTTP response with status 400 (i.e., user error), without calling the method countTendersByYear().

As shown in Figure 10, the class YearFilterPagingRequest has nine fields on which javax.validation constraints are declared. Only one (out of nine) violated constraints is needed to invalidate the whole object. Because the constraints are evaluated in a third-party library, the fitness function has no gradient to guide the search to solve all of them. Looking at the results, it seems like generating valid instances at random is feasible, albeit with low probability. However, small mutations on an existing invalid instance have little to no chance to make the object valid.

To address these issues, there could be at least three strategies:

On the realworld-app API, there is a small difference between EvoMaster black-box and Schemathesis (c8 instrumentation: 69.4% vs. 69.7%), and between gray-box and white-box EvoMaster (EvoMaster instrumentation without bootstrap coverage: 24.6% vs. 24.4%).

Most of the code is covered. What is left are either unfeasible branches or branches related to fetching data from the database (MySQL in this case) with some given properties. Figure 11 shows one such example, where three queries into the database are executed, based on the two inputs id and slug. The condition of the if statement at the end of that code snippet is never satisfied, as it depends on these three SQL queries. Recall that although EvoMaster has support for SQL query analyses (which could help in this case), it is currently only for JVM and not NodeJS.

With a line coverage of up to 53.3%, proxyprint is an API in which reasonable results are achieved, but more could be done. Several functions in this API are never called, and therefore they are impossible to cover, such as the methods sendEmailFinishedPrintRequest and sendEmailCancelledPrintRequest in the class MailBox, and the private methods singleFile Handle and calcBudgetsForPrintShops in the class ConsumerController. Thus, 100% coverage is not possible on this API.

This API provides a selection of various branches that are not covered. However, it is not straightforward to discover the interesting challenges here. One problem is that the number of HTTP calls on this API is low, and therefore not much search is actually carried out compared to the other SUTs when using a 1-hour budget. For example, the generated statistic files of EvoMaster report an average of 37393 HTTP calls per experiment on proxyprint, whereas for rest-news, it is 372733 (i.e., nearly 10 times more). Thus, many of these missed branches could be covered if running EvoMaster for a longer amount of time. Still, some of these branches seem quite unlikely to be covered even with larger search budgets. Let us discuss a few of them. Some are related to authentication. For example, in Figure 12, the code inside the if statement is never executed. To reach that statement, an HTTP call with valid authentication information needs to be provided. Such authentication information is validated with the database when the Principal object is instantiated by the framework (Spring Security in this case). Authentication information needs to be available when the SUT starts (unless the API has some way to register new users directly on-the-fly from the API itself). Therefore, to deal with security (especially when involving hashed passwords), some initialization script is required (e.g., to register a set of users with valid username/password information). In the case of proxyprint, some other data in other tables is created as well (e.g., in the Consumer table). To be able to cover this type of branch, a fuzzer should find a way to either create new valid users or modify the existing data in the database (e.g., EvoMaster can add new data but not modify the existing one [32]).

Figure 13 shows an example in which the line defining the variable quantity throws an exception, due to Double.valueOf being called on a null input. Here, an HTTP object request is passed as input to the constructor of IPNMessage, which is part of the PayPal SDK library. Inside this type of library, request.getParameterMap() is called to extract all parameters of the HTTP request, which are used to populate the map object returned by ipnlistener.getIpnMap(). However, as such parameters are read dynamically at runtime, the OpenAPI/Swagger schema has no knowledge of them (as for this SUT, the schema is created automatically with a library when the API starts). Therefore, there is no information to use an HTTP parameter called mc_gross of type double. As this kind of parameter is used directly without being transformed, testability transformations with taint analysis might be able to address this problem [33], but such techniques would need to be extended to support getParameterMap() and Map.get.

Figure 14 shows another interesting example, where the value of a string field in a JSON payload is used directly to instantiate an enum value (i.e., the statement Item.RingType.valueOf(rti.getRingType())). This fails, as a random string is extremely unlikely to represent a valid value from a restricted set. This might be handled by providing such information in the Open- API/Swagger schema (which supports defining enumerations on string fields), or also by handling valueOf() in enumeration by extending the techniques in the work of Arcuri and Galeotti [33] to support it.

The handler calcBudgetForPrintRequest for the endpoint /consumer/budget is never called (and so all business logic related to this endpoint remains uncovered by the tests). This type of endpoint requires a payload with type multipart/form-data as input, but the schema wrongly specifies the application/json type. A fuzzer might be able to handle this case automatically, but it could be considered as a major issue in the schema, which should be fixed by the users like a fault in the SUT. In other words, there is a big difference between not providing all information (e.g., missing enum value constraints like in the case of /printshops/{id}/pricetable/rings) and providing wrong information (i.e., wrong payload type as in the case of /consumer/budget).

On the rest-ncs SUT, white-box testing with EvoMaster achieves an average of 93%. An analysis of the non-covered lines shows that those are not possible to reach (i.e., dead code). An example is checking twice if a variable is lower than 0, and returning an error if so. In this case, it is impossible to make the second check true. Therefore, as the maximum achievable coverage is obtained in each single run, this can be considered as a solved problem.

Regarding black-box testing, all but two fuzzers achieve more than 50% line coverage, with Schemathesis achieving 94.1% coverage (computed with JaCoCo), followed by RestCT achieving 85.5% average coverage. On this API, black-box EvoMaster gives worse results (i.e., 64.4%).

This is a rather interesting phenomenon, which is strongly dependent on some design choices these fuzzers make. For example, some branches in this API depend on whether two or more integer inputs are equal (e.g., in the case of Triangle Classification). Given two 32-bit integers \(X\) and \(Y\) , there is only a \(1/2^{32}\) chance that \(X=Y\) . On average, it will need to sample more than 2 billion values before obtaining \(X=Y\) . With such constraints, it will be quite difficult for a black-box fuzzer to cover this kind of branch if integer inputs are sampled at random. However, a fuzzer does not need to sample from the whole spectrum of all possible integers. For example, it could sample from a restricted range (e.g., \([-100,100]\) ). On the one hand, the shorter the range, the higher the chances to sample \(X=Y\) . On the other hand, a short range could make it impossible to cover branches that require values outside such range (e.g., 12345). This is a tradeoff that needs to be made.

On this API, it was possible to achieve up to 66.5% line coverage. As for the other SUTs, most of the missing coverage is due to dead code, which is impossible to execute. However, there are some branches that should be possible to cover, but they were not. All of these seem to be related to the same issue, which is the update of existing data in the database. Databases allow the defining of constraints on their data (e.g., a number should be within a certain range), using SQL commands such as CHECK. White-box EvoMaster can generated test data directly into SQL databases [32], taking into account all of these constraints (as all of this information is available in the schema of the database). If any constraint is not satisfied, then the INSERT SQL commands will fail. The problem here is that the SUT might define further constraints on such data. In JVM projects, this is commonly done with javax.validation annotations when using ORM libraries such as Hibernate [9]. But EvoMaster does not seem to handle this and generates data that is valid for the database (as all of the CHECK operations pass and the INSERTs do not fail) but not for these further constraints. Therefore, in an update endpoint, invalid (from the point of view of javax.validation constraints) data can be read from the database, which the SUT then fails to write back (as these javax.validation constraints are checked on each write operation) when the update endpoint has done its computations.

With a line coverage of up to 86.2%, rest-scs is an API in which white-box testing is highly effective. Two types of branches were not covered: the ones involving regex checks with Pattern.matches and the other involving the dot character (‘.’). In this latter case, it is due to EvoMaster not using this type of character when sampling random strings, which makes the if statement in Figure 15 impossible to cover. This might sound like something rather simple to fix. However, as soon as special characters are used in random strings, extra care needs to be taken into consideration when outputting test cases (e.g., the character ‘$’ has special meaning in Kotlin and would need to be escaped when used in strings, which is not the case for Java). Character escaping rules can be quite different based on the context in which they are used in a test case (e.g., inside a URL, inside a JSON object passed as HTTP body payload, or data injected into a SQL database). All of these cases have to be handled, as otherwise the tools would end up generating test cases that do not compile.

On the restcountries API, high coverage is achieved (i.e., 77.1%). Like the other SUTs, there is quite a bit of dead code due to getters/setters that are never called and catch blocks for exceptions that might not be possible to throw via test cases. The class StripeRest presents an interesting case, as it defines the endpoint /contribute, but its information is not in the schema (and thus this type of endpoint is never called).

Figure 16 shows a code snippet that is repeated several times with small variations in a few endpoints. Here, this API returns a list of countries based on different filtering criteria, such as country codes. However, the response with NOT_FOUND is never returned. The problem is that even if the HTTP requests provide invalid inputs (e.g., a country code that does not exist), the countries list is wrongly populated with null values, and thus the list is never empty. This is an interesting bug in the API, which then results in a crash (i.e., a returned 500 status code), as parsedCountries throws an exception. However, until this fault in the API is fixed, all of these statements with NOT_FOUND are technically dead code that cannot be reached with the tests.

Similarly to rest-ncs, the fuzzing of this API might be considered as a solved problem, at least for its current faulty version.

On scout-api, white-box EvoMaster achieves slightly worse results (53.4% vs. 54.7%), although the difference is not statistically significant (based on 10 runs). A large part of its codebase (more than a third) cannot be executed. For example, in the background, scout-api can start a thread to fetch and analyze some data, independently from the RESTful endpoints (i.e., the whole module data-batch-jobs). This type of thread is deactivated by default, and therefore nearly 30% of the whole codebase is not executed. Another non-negligible part of the codebase (around 5%) is related to authentication using OAuth via Google APIs (scout-api provides different ways to authenticate).

There is a large part of the codebase that could be executed, but it is not due to an if statement at the beginning of one of the HTTP endpoints. This is shown in Figure 17. Here an incoming JSON payload is unmarshalled into a MediaFile object, which has a string field that is treated as a URI. Most random strings are a valid URI, as a URI can just be a name. However, it is extremely unlikely that a random string would represent a URI with a valid scheme component, and so uri.getScheme() returns null. However, this is a case that likely would be trivial to solve with taint analysis (e.g., by extending the techniques in the work of Arcuri and Galeotti [33] to support URI objects). However, this does not mean that much higher coverage would be achieved; it depends on all remaining code that is executed once that predicate is satisfied.

Even without code analyses, this type of challenge could also be addressed with black-box techniques. For example, in this API, the string field representing the URI is called uri. An analysis of the names of the fields can be used to possibly infer their expected type. For example, any field whose name starts or ends with uri could be treated as an actual URI when data is generated. Additionally, some types are quite common in RESTful APIs, like strings representing URLs and dates. Instead of sampling strings completely at random, it could make sense to sample with some probability strings with given types that are commonly used. This might now work in all cases, but it is something worthy of investigation.

Similarly to realworld-app, on spacex-api, there is only a small difference between EvoMaster black-box and Schemathesis (c8 instrumentation: 84.7% vs. 85.4%), and between gray-box and white-box EvoMaster (EvoMaster instrumentation without bootstrap coverage: 41.5% vs. 41.5%, but with \(\hat{A}_{12}=0.58\) ). There are two main issues affecting the coverage results here.

First, this API requires authentication, where each API endpoint requires specific authorization roles for execution. For these experiments, a user with the right credentials was created in the database, and the fuzzers were given the authorization information to send HTTP messages on behalf of this user. The setting up of this user was done manually, in a script. However, after running the experiments and analyzing the generated tests, we realized that some authorization roles were missing (e.g., capsule:create). Therefore, a few endpoints were not covered due to misconfigured authentication.

The second main issue is related to the database. Figure 18 shows an example in which a record is searched by ID. No test generated by EvoMaster was able to make a call that returned a 200 HTTP status code. For white-box testing, EvoMaster fails to do this because it is not able to analyze queries on MongoDB databases. Even with black-box testing, it could be possible to create a new record with a POST and use its ID for a following GET. Fuzzers can exploit this kind of information to create sequences of HTTP calls where ids are linked. However, this all depends on how the link is defined. Figure 19 shows the implementation of the POST endpoint to create this type of record. EvoMaster has no issue in fully covering the code of such an endpoint and creating valid records. Additionally, EvoMaster uses different strategies to link endpoints that manipulate the same resources [29, 75]. However, those are based on the recorded interactions with the database (not done here for NodeJS and MongoDB), and on best practices in RESTful API design. For example, it is a recommended practice that POSTs on collections of resources (e.g., /cores) create a new resource, whose id (chosen on the server, to guarantee it is unique, unless it is a UUID) is present in the location HTTP header of the response (e.g., location: /cores/42). EvoMaster can use these location headers to create POST requests followed with the appropriate linked GET. However, on this API, the POST implementation in Figure 19 does not seem to follow best practices in REST API design. For example, it does not set up the location header in the response, and neither does it return the generated id (e.g., in the body payload). However, it could still be possible to test this kind of API by doing the following: create a new resource with POST /cores, followed by a GET of the whole collection (i.e., GET /cores), and then extract the id fields from this response to call GET /cores/:id with a valid id. Note that this is assuming that the collection is empty. If it is not, then the first POST is not even necessary. However, this also assumes that the name of the field representing the ID is the same (or at least very similar) in both the body payload and endpoint path parameter (so they can be matched).