Testing RESTful APIs: A Survey

API schemas are the main inputs that guide how to access the SUT and define what responses should be returned. For instance, as an example of the schema defined with the OpenAPI (Figure 1), an endpoint could be defined under a URI path with an HTTP verb (e.g., POST, GET), input parameters (e.g., Path Parameter), and possible responses (e.g., status code, response body). Then, there exist several black-box coverage metrics that are defined based on those elements [49, 52, 53, 54, 55, 56, 57, 59, 60, 63, 68, 85, 89, 104, 107, 109, 113, 118, 126, 130, 131, 145, 152, 155, 156, 157, 160, 166, 167, 171, 172, 176, 177]:

In addition, Martin-Lopez et al. [129] proposed 10 coverage metrics based on the schema that have been applied for assessing REST API testing approaches in Reference [63]. The 10 coverage metrics enable assessments of generated tests in fuzzing REST APIs with different inputs and outputs, such as parameter coverage, content-type coverage, and response body properties coverage. The metrics were also enabled in fuzzers, such as HsuanFuzz [155] and RESTest [131]. Restats [84] is a test coverage tool for assessing given tests based on the coverage metrics. In an empirical study [83], such schema-related coverage metrics were employed to compare black-box fuzzers for REST APIs.

Code coverage [49, 52, 54, 55, 56, 57, 59, 115, 118, 126, 145, 152, 155, 171, 172, 176, 177] is one typical white-box criterion for evaluating testing approaches. It was applied as well to evaluate REST API fuzzers for both black-box [59, 60] and white-box [53, 55, 171, 172, 173, 177]. Among different code coverage criteria, line/statement coverage is one of the most common and supported ones in industry-strength coverage tools (e.g., JaCoCo for Java programs).

Code coverage was also used as an evaluation metric to assess the black-box REST API testing approaches [60, 115, 118, 155]. For instance, in Reference [60], Atlidakis et al. reported accumulated code coverage achieved along with the number of requests to be executed. To collect the code coverage, TracePoint hook was configured in Ruby classes to trace the execution in this study. However, black-box testing of REST API could be performed on remote public APIs that are closed-source software. The code coverage in this context is applied only for evaluation purposes.

Code coverage could also be employed as a criteria to automate fuzzing, which is considered as white-box testing. EvoMaster enables runtime code coverage (with heuristics to maximize it) automatically during fuzzing with code instrumentation. However, such automated collection of runtime code coverage is specific to the programming language, and EvoMaster has enabled it for JVM (e.g., Java and Kotlin) [52] and JavaScript [175]. In addition, Pythia [59] employs the metrics of code coverage to produce tests. But, to enable such code coverage collection, it needs a pre-manual step to extract code information (such as block location) of the SUT.

Besides schema-based and code coverage, there also exist other specialized metrics, more specific (and possibly less general) to the proposed approaches [68, 77, 78, 98, 117, 118, 120, 130, 143, 156, 169].

For instance, in RESTler [60], a grammar is derived based on the API schema for driving following test sequence generation, e.g., selecting next HTTP request based on the derived producer-consumer dependencies among the endpoints. A specific metric employed in this approach is based on the grammar, i.e., grammar coverage. Martin-Lopez et al. [127, 128, 130] defined inter-parameter dependencies that represent constraints referring to two or more input parameters. Such dependencies are also used in RestCT to generate test data [166]. Alonso et al. [156] enabled test data generation with realistic inputs using natural language processing and knowledge extraction techniques. The performance of the data generation was evaluated from a percentage of valid API calls (i.e., 2xx status code) and valid inputs (i.e., Syntactically valid and Semantically valid). In Reference [100], Godefroid et al. introduced the error type metric, which is a pair of error code and error message. The metric was used to guide data fuzzing of REST API [100], i.e., maximizing error type coverage.

Chakrabarti and Rodriquez [77] defined POST Class Graph (PCG) to represent resources and connected relationships among them, then tests could be produced by maximizing coverage of the graph. UML state machine was applied in a model-based testing of REST APIs [143]. Metrics specific to model such as state coverage and transition coverage are used to guide test generation.

Lin et al. [115] constructed tree-based graph to analyze resource and resource dependency based on the API schema and responses. Then, test cases could be generated based on the graph with tree traversal algorithms.

To analyze security issues existing in REST API, Cheh and Chen [78] analyzed levels of sensitivity of data fields and API calls and also defined exposure level that calculates a degree of such data fields and API calls exposed to potential attacks.

The status code 5xx has been applied to identify potential faults in REST API testing [49, 52, 53, 54, 55, 56, 57, 59, 60, 63, 66, 85, 89, 104, 107, 113, 118, 126, 130, 131, 145, 152, 155, 156, 160, 166, 171, 172, 176, 177]. With HTTP, the status code 5xx indicates errors caused by the server, and the request could not be processed until the server has been fixed. For instance, the 500 status code is generic to represent that an internal server error occurs when performing the given request. The status code 503 is more specific, stating that the service is unavailable, e.g., due to down for maintenance or the server is overloaded.

In most HTTP frameworks, when there is a crash in the business logic due to some faults (e.g., an unhandled null-pointer exception), the entire server does not crash. In these cases, the server would still reply to the incoming HTTP request, responding with a status code of 500. Therefore, 500 status codes in the responses can be used as an oracle to detect faults in RESTful APIs [104]. However, not all 500 status codes are related to software faults. For example, if the API is connecting to a database, and the database is currently down, then the server would not be able to complete the request. In such a case, returning a 500 status code would be correct, although no software fault in the API is involved.

The API schema (such as OpenAPI) defines the response syntax for each operation [53, 54, 55, 56, 57, 85, 89, 113, 126, 152, 160, 171, 172, 177], e.g., status code and response body. Actual responses should be always consistent with the syntax specified in the schema. Thus, any inconsistency between the actual response and the syntax could be regarded as faults in the REST API. For instance, Viglianisi et al. defined such oracles in RestTestGen [160] by using an OpenAPI library¹⁴ to identify the mismatched responses. In QuickRest [107], Karlsson et al. formulated such consistency as properties.

Service errors (based on status code) and violations of schema (based on OpenAPI) are general oracles for fault finding in the context of REST API. Besides, there also exist some oracles to identify faults based on rules that characterize the REST APIs in terms of security, behavior, properties, and regression [61, 64, 66, 78, 92, 104, 107, 108, 113, 147, 156, 163, 169] (see Figure 5).

Security . As web services, security is critical for REST APIs. To enable test oracle relating to security, Atlidakis et al. [61] proposed a set of rules that formalize desirable security-related properties of the REST APIs. Any violation of the rules is identified as potential security-related bugs in the SUT. The rules are mainly defined based on assessing accessibility of resources, such as use-after-free rule: If a resource has been deleted, then it must not be accessible anymore.

Katt and Prasher [171] proposed a quantitative approach to measure the kinds of security related metrics (i.e., vulnerability, security requirement, and security assurance) for web services. To calculate the metrics, test cases are defined for validating whether the SUT meets security requirements and any kind of vulnerabilities exists. Masood and Java [133] identified various kinds of vulnerabilities that could exist in REST APIs, such as JSON Hijacking. Such vulnerabilities could be detected with both static analysis and dynamic analysis techniques. Barabanov et al. [64] proposed an approach specific to detection of Insecure Direct Object Reference (IDOR)/Broken Object Level Authorization (BOLA) vulnerabilities for REST APIs. The approach analyzes the OpenAPI specification by identifying its elements (such as parameters) relating to IDOR/BOLA vulnerabilities, then generates tests for verifying the API with such elements using defined security rules. Zha et al. [169] collected Common Sense Security Policies(CSSP), such as Access control, URL spoofing, and private messages for Team Chat system and defined CSSP violation scenarios. Security and privacy risks can be identified if an API under the CSSP violation scenarios can still work, e.g., return a valid response. Barlas et al. [66] studied regex-based denial of service (ReDoS) vulnerabilities led by handling of input sanitization in web services. The vulnerabilities are allowed to be identified by verifying consistency between client-side and server.

Behavior . Based on the provided API schema, Ed-douibi et al. [89] defined two rules to generate nominal test cases and faulty test cases. The nominal test cases take the inputs inferred based on examples or constraints in the schema, and the successful response is expected to return (e.g., assert that the status code is 2xx). Regarding the faulty test cases, it takes invalid inputs (e.g., missing required parameters, a string for a number parameter, a string violating its defined pattern), and the client error response is expected to return (e.g., assert that the status code is 4xx).

Liu et al. [117] constructed five constraints of REST guidelines with models. Then, such models could be used to verify design models of REST APIs. Any violation of the constraint models is considered as a potential defect in its architecture design.

Pinheiro et al. [143] defined UML state machines for modeling behaviors of REST APIs. The actual behavior (by executing the tests) should be consistent with the model (e.g., guard condition, invariant) as expected. Any inconsistency is recognized as potential faults of the SUT.

Properties . Most of the HTTP methods are idempotent, i.e., GET, PUT, DELETE, HEAD, OPTIONS, and TRACE. For such methods, the result of executing the method is independent of the number of repeated times, meaning that executing the method multiple times would not change the result compared to executing it once. Thus, assertions could be defined to check idempotency [157]. For example, executing multiple identical GET requests should result in the same response, and after a successful DELETE, the responses of all following identical DELETE requests should be the same. Connectedness is examined in Reference [77], which refers to accessibility among resources. For instance, assume that resource X owns resource Y; when performing a GET collection on Y referring to X, all available Y should appear in the response, otherwise the REST API is not “connected.”

The REST API could apply HATEOAS (Hypermedia as the Engine of Application State). Then, the response might contain hypermedia links for accessing itself or other resources. For such responses, Vassiliou-Gioles [157] defined assertions for validating availability of links in its response. In References [161, 162], Vu et al. also proposed a model-based approach that enables formalization of hypermedia behaviors of the REST API with \(\varepsilon\) -NFA, and the model could allow an identification of the faults by checking whether the SUT complies with it [163]. Fertig and Braun [92] also verified the REST APIs based on hypermedia constraints using model-based approaches.

Metamorphic relations capture necessary properties that the SUT should hold with multiple executions. To enable metamorphic testing of REST APIs, there exist several works to identify such metamorphic relations of the web services with abstract Metamorphic Relation Output Patterns (MROPs) [122, 147] (such as equivalence, disjoint) or specific properties of the API [120]. Then, faults could be detected by checking whether responses among the multiple requests conform to the identified relations.

Regression . Gazzola et al. [98] enable monitoring and tracing of the microservices to record their execution. Such recorded execution slices can be abstracted and considered as a metric for generating regression tests, i.e., verify whether the same response could be received with the same request in the further version of the SUT. Godefroid et al. [101] employed RESTler to produce tests and enabled detection of regression faults by comparing behaviors with the same tests among different versions of the REST APIs.

Performance-related metrics are also important for REST API testing [63, 66, 74, 92, 104], as the SUT provides services over the network. In Reference [63], Banias et al. measured the average response time of the requests generated by different strategies. Fertig and Braun [92] discussed potential solutions to enable performance testing with model-based approaches and existing techniques (such as Apache JMeter¹⁵). Schemathesis [104] defined the proprieties relating to performance metrics, i.e., identify slow response and request amplification with configured thresholds. Bucaille et al. [74] developed a testing framework that can assess and monitor performance-related properties, such as latency, by sending requests from various geographical locations using different cloud service providers.

Based on our survey, there exist two main types of testing techniques to automate testing of REST APIs, i.e., black-box and white-box. Regarding black-box testing, it enables to verify systems’ behaviors with exposed endpoints, e.g., checking responses of an HTTP request. The verification and validation of REST APIs with black-box technique are mainly driven by exploiting the API specification and returned responses. Based on the results of this survey (see Table 6), 72% of the existing approaches are in the context of black-box testing.

Regarding black-box fuzzers, bBOXRT [113] aims at testing of REST APIs in terms of its robustness. EvoMaster applies search-based techniques by defining fitness function with black-box heuristics in a black-box mode [53], such as status code coverage and faulty status code (i.e., 500), to generate tests. There is also a set of property-based testing approaches that identifies properties of REST APIs as test oracle problems, such as QuickRest [107] and Schemathesis [104]. For instance, in terms of the API specification, QuickRest and Schemathesis both identify consistency with the specification as the properties. Schemathesis also explores semantic properties of REST APIs, such as GET should fail after an unsuccessful POST when they perform on the same resource. Moreover, dependencies of REST APIs are studied for REST API testing. For instance, RESTler [60] infers and handles dependency among endpoints to generate effective tests based on API specification and runtime returned responses. RESTest generates tests by exploring inter-parameter dependencies of REST APIs. With RestTestGen [85, 160], Operation Dependency Graph is proposed to capture data dependencies among operations of a REST API. The graph is initially built based on its OpenAPI schema then further extended at runtime. Morest [118] formalized a property graph to construct relations of operations and object schema, and such a graph could be derived based on the API schema then dynamically updated based on responses at runtime. Within a specified time budget, test cases could be initially generated by traversing the graph, and any update of the graph would result in new tests. Furthermore, RestCT [166] is a combinational approach, integrating two phases that facilitate generating orders of operations then concertizing input parameters of operations. The orders are generated based on HTTP action semantics with greedy algorithm, e.g., for a specific resource, its GET operation should not appear before its POST and after its DELETE. The concrete values of input parameters could be produced in various ways, such as random, previous responses, specified examples, or inter-parameter dependencies. In the context of black-box testing, Cheh and Chen [78] enabled a semi-automatic approach to identify security issues in the API schema. Navas [136] proposed an approach to infer and validate the API schema, such as misnamed and duplicated elements in response schema.

Compared to black-box testing, white-box techniques could enable testing of internal behaviors of the REST APIs with additional metrics relating to source code, such as code coverage. For example, Pythia [59] employs code coverage heuristics to guide test generation. The code coverage could be collected by pre-configuring locations of basic blocks in its implementation with static analysis. EvoMaster is a fuzzer that has a white-box mode, and the white-box testing is enabled by code instrumentation for JVM [51, 52] and NodeJS programs [175] developed in the fuzzer. The code instrumentation allows to identify the code to cover and collect code coverage at runtime. With such information, EvoMaster defined white-box heuristics and applied search-based techniques to fuzzing REST APIs. With this survey, we found that all other white-box testing techniques are built on top of the EvoMaster platform (i.e., References [49, 52, 55, 145, 152, 154, 161, 171, 172, 176, 177]) with new algorithms and new techniques.

There also exist hybrid approaches that combine black-box and white-box [126]. For instance, Martin-Lopez et al. [126] proposed a solution by integrating two fuzzers (i.e., RESTest and EvoMaster). A motivation of having such a hybrid approach as described in the paper is that inputs of a REST API may be restricted with constraints, and the constraints are not specified in the API schema. Without the information of constraints, generating successful requests (i.e., receiving a response with 2xx status) is not trivial. However, white-box testing approach could tackle this problem by identifying the constraints with source code, such as EvoMaster [56, 57]. In Reference [126], RESTest is first employed to generate tests in the context of black-box testing, then the generated tests would be regarded as seeds that EvoMaster starts with. With the four selected REST APIs, the hybrid approach achieved the best results compared to isolated black-box and white-box solutions.

Search-based testing aims at solving software testing problems with metaheuristic search techniques, such as Genetic Algorithms and Swarm Algorithms. To enable the search techniques, it needs to reformulate the addressed testing problem as a search problem.

EvoMaster is a search-based fuzzer that reformulates test case generation as search problems supporting both black-box and white-box mode. The black-box mode is achieved with Random strategies with black-box heuristics, such as coverage of operations and status code. The white-box mode is enabled with novel techniques (e.g., code instrumentation [52], testability transformations [56, 57], and SQL handling [54, 55]), which allows to define white-box heuristics as part of fitness function. For instance, code instrumentation enables identification of lines and branches to cover as testing targets and collecting such coverage at runtime, and testability transformations provide better guidelines to search for maximizing the testing targets. Many independent objectives algorithm (MIO) is the default algorithm in white-box mode of EvoMaster. MIO is an evolutionary algorithm developed specific to white-box system test generation, and the algorithm is inspired by (1+1)EA [88], which contains sampling and mutation operators. Its effectiveness to test REST APIs has been demonstrated in several papers [51, 52, 57]. To be specialized for Web API testing and REST API domain, MIO is further extended with adaptive hypermutation [171], smart sampling [52], and resource-based techniques [176, 177].

Genetic Algorithms (GAs) were also enabled for REST API testing. For instance, the Whole Test Suite [95] employed genetic algorithm to enable automated test suite generation. With the GA, the approach evolved the test generation with mutation and crossover operators, and the fitness function is defined with an overall single objective using white-box heuristics. Instead of single-objective optimization, the Many-Objective Sorting Algorithm (MOSA) extends NSGA-II [87] for handling test generations with respects to maximizing many testing targets as many-objective optimization. WTS and MOSA have been integrated into EvoMaster, which allows to apply them to tackle REST API testing [51]. With the EvoMaster platform, Stallenberg et al. [152] employed Agglomerative Hierarchical Clustering (AHC) to identify patterns of tests, then extended MOSA (named LT-MOSA) to generate tests with the patterns. Aside from test generation, Liu and Chen [116] employed genetic algorithm to optimize test data of REST APIs in the context of mutation testing.

Moreover, Swarm Algorithms were also investigated. Sahin [145] proposed a Discrete Dynamic Artificial Bee Colony with Hyper-Scout (DABC-HS) algorithm that was introduced to address shortcomings of a basic Artificial Bee Colony (ABC) for REST API testing. Furthermore, a Greedy Algorithm was applied in RestCT to produce operation sequences [166].

Property-based testing is an approach that validates and verifies the SUT based on identified properties that should always abide by the SUT. In the context of REST API testing, one type of properties could be identified based on schema, such as check if the responses during testing conform to the schema [104, 107]. In addition, as REST, it defines a set of guidelines on how to access resources. Then, states of resources in REST API are captured as properties of REST APIs. For instance, QuickRest [107] defined stateful properties of REST API with documented responses for producing a stateful sequence of operations. Seijas et al. [112] generated finite state machines to construct changes of states of resources, then employed Quviq QuickCheck to enable property-based testing of REST APIs. Chakrabarti and Rodriquez [77] examined the connectedness of resources in testing REST APIs.

There also exist some works for exploring semantic of REST APIs. For instance, in Schemathesis, Hatfield-Dodds and Dygalo [104] derived structural and semantic properties of REST APIs relating to constraints, security, and performances. Metamorphic testing was also enabled in REST API testing that identified metamorphic relations of the REST API based on their semantics [120, 122, 147].

Model-based testing is to employ models to perform testing. The models are typically constructed manually before the testing for depicting the SUT, such as states, behaviors. For instance, Vu et al. [161, 162, 163] proposed a model-based testing approach that uses an \(\varepsilon\) -NFA to construct hypermedia behavior of the REST API. Fertig and Braun [92] developed a Domain Specific Language (DSL) for constructing models of REST APIs, then defined a set of test templates for test generations with such models.

In Reference [117], Liu et al. proposed a model-based approach for verifying REST service architecture using colored Petri Nets (CPN). Five REST feature constraints are defined with CPN models. Such constraint models allow a verification of architecture models using simulation. Pinheiro et al. [143] constructed behaviors of REST API with UML state machines.

Moreover, the models could also be used to construct the tests. Ed-douibi et al. [89] defined a metamodel for formalizing test suites of REST APIs. In the approach, the authors proposed a set of rules (e.g., infer parameter values based on examples, generate faulty test cases with invalid inputs) to generate the test suite model based on the OpenAPI. Then, the generated test suite model is further converted to execute code (e.g., JUnit) for performing the requests against the SUT.

Graphs were constructed in several works for capturing the structures and relationships of REST APIs. Then, for example, Breadth-First Search (BFS) can be employed for generating tests based on such graph [60].

Gazzola et al. [98] proposed an approach called ExVivoMicroTest that facilitates regression test generation by recording service interactions at runtime. Godefroid et al. [101] proposed an approach for detecting faults due to changes made among different versions of RESTful web services. Such faults could be identified by comparing behavior of different versions with the same inputs generated by a fuzzer RESTler.

Takeda et al. [154] developed a test case extraction approach that aims at identifying the impacts of migrating APIs to microservices from monolithic architecture. The impacts are derived by analyzing source code in the context of white-box testing.