A Survey on Automated Driving System Testing: Landscapes and Trends

According to the complexity and variety of the ADS, the society of automotive engineers (SAE) proposed the taxonomy and definitions of driving automation systems, known as SAE J3016,¹ which has become a classification standard in recent years. This standard categorizes driving automation systems into six levels, ranging from no driving automation (Level 0) to full driving automation (Level 5). These levels are usually referred to as L0 to L5.

The definitions of the systems from L0 to L5 are as follows:

These are shown in Table 1.

A system in L0 to L2 is also known as an advanced driver assistance system (ADAS), since it is only in charge of a part of the DDT, such as lateral control or longitudinal control, and the safety of the whole vehicle still relies on drivers. In contrast, a system in L3–L5 performs all of the DDT and drivers are not expected to interfere during the driving process, so it realizes the real automated driving.

Note that there exist other identified synonyms of Automated Driving, e.g., autonomous driving, self-driving, but in this article, we follow the relevant terminology from SAE J3016, in which the term “ADS” refers to Automated Driving System. In the literature [8], an ADAS is usually referred to as a system that belongs to L0–L2, while an ADS is referred to as a system that belongs to L3–L5. In this survey, since many testing techniques are independent of the automation levels of the systems under test, we sometimes mix the use of the terms and, by ADS, we refer to the systems over all of the levels of driving automation.

A common ADS is composed of four functional modules, namely the sensing module, the perception module, the planning module, and the control module, as shown in Figure 1. In the sensing module, intelligent sensors (e.g., camera, radar, and LiDAR) are used to collect the driving context from the physical world. The perception module extracts useful environmental information from the sensor data and sends it to the planning module for motion planning. Based on the information, the planning module generates the optimal driving trajectory. Last, the control module outputs the control commands to drive the vehicle along the trajectory. Moreover, some modern ADS adopt a special design named end-to-end module. In the remainder of this section, we elaborate on the functionalities of these modules in the typical architecture of an ADS. Sensing module. By adopting various physical sensors, the sensing module takes charge of collecting and preprocessing driving environmental information from the physical world. The common sensors used by an ADS include Global Positioning Systems (GPS), inertial measurement units (IMU), cameras, radar, and LiDAR. Specifically, GPS provides the absolute position data (e.g., latitude, longitude, and heading angle) while IMU provides the temporal data (e.g., acceleration and angular velocity). The combination of these two sensors can provide more accurate real-time positioning of the autonomous vehicles. Cameras are used to record and capture visual information on the driving road for the perception module, and radar is used to detect obstacles by radio waves. Nowadays, LiDAR has become an indispensable component in many leading ADS (e.g., Apollo and Autoware), since it can collect three-dimensional (3D) point cloud data and process it with higher measurement accuracy. In comparison with camera sensors that are sensitive to light conditions (e.g., shadows and bright sunlight), LiDAR sensors are more robust under these environments, and the generated 3D point cloud can be further utilized to build 3D models that better characterize the surrounding objects.

Perception module. With the help of deep learning techniques, the perception module processes sensor data (e.g., pictures and 3D point cloud) from the sensing module to accomplish a series of perception tasks, such as localization, detection, and prediction.

The prediction task also benefits from the perception module and is mainly used for trajectory planning. We leave the introduction of this task below in the planning module.

Planning module. By using DNNs and planning algorithms, the planning module takes perception data as input and makes decisions for the control module to control the vehicle. It has two submodules, namely the prediction submodule and the planning submodule.

Control module. Based on the trajectories planned by the planning module, the control module finally takes charge of the longitudinal and lateral control of the vehicle. By using control algorithms (e.g., proportional integral derivative (PID) control [13] and model predictive control (MPC) [14]), this module generates appropriate control commands (e.g., steering and braking) and sends them to the related hardware, i.e., the electronic control unit (ECU), via the protocol of controller area network (CAN) bus. Note that this module is critical for several functionalities provided by the ADS, including ACC, AEB, and Lane Keeping Assistance.

End-to-end module. As shown in Figure 1, besides the common modules mentioned above, there exists another end-to-end design that combines the perception, planning, and control processes in one module. To be specific, this module mainly consists of special deep learning models, which are trained by labeled data that maps information from sensors directly to the corresponding control commands. Consequently, these models could output the control commands based on the current driving environment.

In this section, we introduce four open source ADS, namely Apollo, Autoware, OpenPilot, and Pylot, and one simulation platform called BeamNG [15]. The first three ADS have been widely adopted for commercial usage in practice [16, 17, 18], and the last ADS is from academia.

Apollo . Apollo has been a popular open source ADS developed by Baidu since 2017; as of December 2021, it has been updated to version 7.0.0. The hardware platform of Apollo includes camera, LiDAR, millimeter wave radar, and Human-Machine Interface (HMI) device, and currently the communications over different components are managed by CyberRT [19]. The functionalities of Apollo include cruising, urban obstacle avoidance, and lane changing.

Autoware . Autoware is another open source L4 ADS developed by the research group of Nagoya University in 2015. Though it is mainly applicable to urban roads, it also suits highways and other road conditions. By using the sensors introduced in Section 2.2, it supports a series of functionalities including connected navigation, traffic light recognition, and object tracking. Unlike Apollo, which uses CyberRT, Autoware adopts robot operating system (ROS) [20] for communications over different components.

OpenPilot . OpenPilot is a popular open source L2 ADAS developed by Comma.ai, and it has been updated to version 0.8.12 as of December 2021. OpenPilot supports common L2 features, such as ACC, ALC, and Forward Collision Warning. Unlike other L2 ADAS, OpenPilot has a high degree of portability—it can be compatible with more than 120 types of vehicle models by using related hardware set (e.g., Car Harness [21] and Comma Two [22]).

Pylot . Pylot [23] is a modular and open source autonomous driving platform developed by UC Berkeley in 2021. For achieving the tradeoff between latency and accuracy, it is built on a deterministic dataflow system called ERDOS [24]. Pylot also has other built-in features such as modularity, portability, and debuggability, which allow researchers to implement or test ADS functions with higher efficiency.

BeamNG . BeamNG [25] is a popular image-generating simulation platform, which has been widely used in the Search-Based Software Testing competition [26]. Specifically, it is based on a physically accurate engine that can support customized vehicle models and realistic damage. For example, different components of a vehicle can have different degrees of deformation after a collision. In addition, BeamNG also contains a driving agent called BeamNG.AI [27], which could take over one or more vehicles and drive in several different modes.

This section introduces the methodology adopted in our paper collection process, which is illustrated in Figure 2. The overall process consists of five main steps, namely database search, abstract analysis, full-text analysis, backward & forward snowballing, and data extraction. The intermediate results of each step during the process are reported in our supplementary website² and are also available in Zenodo [28]. We now describe the details of each step as follows.

In this section, we analyze the collected papers from three perspectives, namely the publication venues, the targeted system modules, and the publication years. We show the distribution of the publication venues of all the papers in Figure 3(a) and the distribution of the targeted modules/system in Figure 3(b). Moreover, we present the number of papers published in different years in Figure 3(c).

Publication venues. In Figure 3(a), we can see that (i) many of the papers, up to 38 \(\%\) , are published in transportation venues such as ITSC and IV; (ii) 25 \(\%\) of the papers are published in software engineering venues such as ICSE and ASE; (iii) the adversarial attack methods for vulnerability detection of ADS are related to the security of the systems, and hence 10 \(\%\) are published in the security venues, such as CCS and USENIX Security Symposium; and IEEE Symposium on Security and Privacy; and (iv) since the ADS and artificial intelligence are closely related, 7 \(\%\) of the papers are published in artificial intelligence venues such as CVPR and AAAI.

Target modules. In Figure 3(b), we can see that, obviously, the papers on system-level testing dominate the largest percentage, up to 54 \(\%\) . These papers involve testing techniques that span over both simulation-based testing and mixed-reality testing. Moreover, the number of the papers concerning with the perception module is the second largest, up to 22 \(\%\) . The perception module takes charge of object detection and image semantic segmentation using deep learning, which is important but vulnerable to safety and security threats and thus becomes a popular research direction. Compared to the perception module, there are fewer papers concerning other modules, such as the planning module or the control module.

Publication year. In Figure 3(c), we can see that the number of the papers related to ADS testing shows a general ascending trend, from 2015 to 2021. This trend indicates that the safety and security of ADS are attracting more and more research attention from researchers. The reason for fewer relevant papers in 2022 is that only partial papers had been published by the time we collected the papers.

Because of the high complexity of the system architectures of ADS, it is necessary to have a comprehensive understanding of the systems before performing their evaluation. The system studies, e.g., on Apollo [31], build the logical architectures that disclose the connections over different modules in ADS. As a result, these lines of work can bring insights into the potential vulnerabilities and suggest useful metrics for system testing.

Peng et al. [31] investigate the collaboration between the code and the DNN models in Apollo; specifically, they study which roles are played by the code and the underlying DNN models, respectively. They find that the 28 DNN models used in Apollo interact with each other in diverse ways, e.g., the output of one DNN can be used as the input of another DNN, and the outputs of multiple DNNs can be combined as the input of another DNN. Moreover, the code also plays an important role in the system workflow, e.g., it can be used for filtering out invalid output of DNNs, and it can complement the imperfect outcome of DNNs.

For those open source ADS, issues and bugs reported in their public repositories (e.g., GitHub) reflect real problems encountered by users and developers during the development and deployment. Therefore, systematic analysis on these issues [32, 33] can provide insights into the root causes of system failures. In this section, we review two studies [34, 35] in the field of ADS testing.

Garcia et al. [34] present a comprehensive study of bugs in two ADS, namely Apollo and Autoware. Specifically, they collect bugs from the commits across the Apollo and Autoware repositories in GitHub and perform a manual analysis on these bugs and commits. As a result, they obtain 13 root causes (e.g., algorithm, data, memory) for system crashes, 20 symptoms (e.g., speed and velocity control, vehicle trajectory) and 18 bug-related components (e.g., perception, planning, control), based on their analysis of 499 bugs in the two ADS.

Tang et al. [35] perform a study on issue analysis for OpenPilot. They collect 235 bugs from 1,293 pull requests and 694 issues of the OpenPilot project in GitHub and Discord.⁵ These bugs are then classified into five categories, including (DNN) model bugs, plan/control bugs, car bugs, hardware bugs, and UI bugs. Among these different types of bugs, they find that the car bugs related to the interface with different car models dominate \(31.48\%\) , and plan/control bugs related to the control of car behaviors account for \(25.95\%\) .

The following works all perform analysis on public reports, i.e., CADMV [30], which is a database involving disengagement and crash records on public roads. Specifically, a disengagement refers to a failure that requires a human driver to take over control of the vehicle; a crash refers to a collision with other traffic participants. These empirical studies investigate the relevant factors, such as the causes, the correlations, and the impacts of these system failures, and they also shed light on future system developments.

Analysis of disengagement reports. In References [36, 37, 38], the authors analyze the disengagements based on different metrics. Lv et al. [36] classify the disengagement events into two types, namely active disengagement and passive disengagement, and investigate the root causes of each group. Boggs et al. [37] apply the binary logistic regression [39] to categorize the cause of the disengagements in more detail. The results show that the planning discrepancy (e.g., improper localization, motion planning) accounts for \(41\%\) of ADS disengagements. Khattak et al. [38] investigate the relationship between disengagements and crashes and find relevant factors that could increase the likelihood of a disengagement without a crash.

Analysis of crash reports. References [40, 41, 42, 43, 44, 45, 46] analyze the crash reports and identify the contributing factors. Leilabadi et al. [40] apply text analysis to the crash reports, and they find that the crashes mostly occur when vehicles run in the automated mode, and the most frequent ADS crash type is the rear-end collision. Favaro et al. [41] focus on the dynamics aspect and present the speed distribution of those crash vehicles. Wang et al. adopt regression and classification tree to investigate the types and severity of these crashes. They find that the severity increases significantly when an automated vehicle is responsible for the event. Das et al. [43] utilize Bayesian latent class model to perform the analysis and identify six collision patterns. Aziz et al. [44] investigate both crash data involving ADS and without ADS and build a spatial-temporal mapping of the contributing factors between them. Song et al. [45] conclude that the most representative crash pattern is the “collision following ADS stop,” i.e., an automated vehicle stops suddenly and gets hit by other vehicles on the road. Besides CADMV, the crash data in other databases such as UK’s STATS19 [47] are also analyzed with statistical approaches [46, 48].

Table 2 summarizes the collected papers that empirically study the issues in ADS testing. Several existing system studies focus on Apollo, and there are also studies that cover other open source ADS, such as Autoware and OpenPilot. Moreover, there are many works [36, 37, 38, 40, 41, 43, 44, 45, 46, 49] that target the disengagement and crash reports for identifying the root causes or failure types, as these investigations are critical to understanding the ADS safety performances in the real world.

The sensing module is the frontier module of an ADS and the performance of the physical sensors (e.g., camera, radar, and LiDAR) in this module is critical to the safety and security of the whole ADS. Relevant studies on the test methodology of this module can be divided into physical testing (shown in Section 5.1.1) and deliberate attack (shown in Section 5.1.2). Physical testing aims to test the performance of the sensors under different physical conditions, while deliberate attack interferes with the input signals of the sensors to diminish the sensing quality.

The perception module receives and processes sensor data; based on that, it perceives external environments. The literature we collected includes the test methodologies (shown in Section 5.2.1), the test oracles (shown in Section 5.2.2), and the test adequacy criteria (shown in Section 5.2.3) for testing the DNN models used in the perception module of ADS.

The planning module takes the information from the perception module as input and produces a suitable driving trajectory as a reference for the control module to make decisions. In the planning module, we introduce the studies on test methodology (shown in Section 5.3.1), test oracle (shown in Section 5.3.2), and test adequacy (shown in Section 5.3.3).

Based on the trajectories produced by the planning module, the control module takes charge of the lateral and longitudinal control of the ADS. By using various control algorithms, such as MPC and PID control, it generates control signals, e.g., acceleration, deceleration, and steering angle, to the CAN bus for the control of the whole system. In this module, we introduce the works on the testing of the control module, from the perspectives of test methodology (shown in Section 5.4.1) and test oracle (shown in Section 5.4.2).

The end-to-end module is a special design adopted by many modern ADS, which integrates the functionalities of perception, planning, and control in a single DNN-based model. The DNN model is often developed by supervised learning, which is trained by using a training dataset consisting of realistic driving data. Each element of the dataset is a pair \(\langle I, c\rangle\) , which maps the information I at the end of the sensor to a label c that indicates the desired control decision at the end of the controller. After training, a model can infer control decisions based on the driving environment at runtime to drive the ADS properly. For instance, in some modern ADS that perform steering angle control, the end-to-end DNN model takes as input the sensing information including road conditions and the status of other cars and outputs a series of predicted steering angles for controlling the ADS. In this section, we introduce the collected studies on the testing of the end-to-end module from the perspectives of test methodology (shown in Section 5.5.1), test oracle (shown in Section 5.5.2), and test adequacy (shown in Section 5.5.3).

In total, we survey over 80 papers that study the testing of different modules of ADS. Various testing techniques have been proposed for testing different modules. Based on our survey, we can draw the following conclusions: (1) for the sensing module, physical testing and deliberate attack on the sensors could effectively find their abnormal behaviors; (2) for the perception module and the end-to-end module, adversarial attack is the most widely used approach, since the two modules mainly rely on the use of DNN-based models; (3) for the planning module, though the relevant studies are not so many, search-based testing has been extensively adopted; and (4) for the control module, main testing techniques include fault injection, sampling, and falsification.

Despite the numerous techniques dedicated to different modules, we also find some open challenges for the testing of these modules. For example, the neuron coverage in Reference [107] may not be effective for testing the perception module and the end-to-end module. More details about these open challenges are also discussed in Section 8.

We first introduce system-level testing conducted with the help of software simulators. Similarly to the module-level works, we also present these studies from three perspectives, namely test methodology (shown in Section 6.1.1), test oracle (shown in Section 6.1.2), and test adequacy (shown in Section 6.1.3).

Due to the expensiveness of the real-world testing of ADS, most approaches in Section 5 and in Section 6.1 test ADS in software simulators. Although modern simulators can be powerful and high-fidelity, simulation-based testing is not sufficient to reveal all the problems of ADS, due to the gap between simulators and the real world. As a tradeoff, mixed-reality testing combines simulation-based testing with real-world testing. In this section, we introduce several special testing schemes, which replace certain parts of the components in the testing loop with physical components. Specifically, these schemes include hardware-in-the-loop (HiL), vehicle-in-the-loop (ViL), and Scenario-in-the-Loop (SciL); their mechanisms are illustrated in Figure 4.

Regarding the efforts in simulation-based testing, a natural question arises that, how far is the simulation-based testing still from the real-world testing. Moreover, Kalra et al. [287] find that the ADS should be driven hundreds millions of miles to demonstrate their reliability. As an emerging issue, this topic has attracted increasing research attention; here, we introduce the latest progress from two perspectives, namely the realism of test cases and the realism of simulators.

Realism of test cases. One question arises in the simulation-based testing whereby the virtual scenarios generated by testing algorithms that lead to system failures may never happen in the real world. Indeed, simulators create a wide range of traffic participants of which, nevertheless, only a subset can really happen.

There is a line of work that aims to bridge this gap and thus generate natural scenarios for ADS testing. Nalic et al. [288] propose a co-simulation framework using two simulation tools, CarMaker (for vehicle dynamics) and VISSIM (for traffic simulation); their framework can generate scenarios based on calibrated traffic models derived from real-world data. In their later work [289], a stress testing method, which has been introduced in Section 6.1.1, is applied for increasing the number of detected critical scenarios under the co-simulation environment. Klischat et al. [290] utilize OpenStreetMap to extract real-world road intersections and combine with SUMO to generate realistic traffic scenarios. Wen et al. [291] focus on triggering the events in a specific area near the ego vehicle, and a convolutional neural network (CNN) based selector is utilized to choose those scenario agents, which could achieve more realistic results.

The following works [292, 293, 294, 295, 296] focus on reconstructing scenarios from public crash reports. Mostadi et al. [295] utilize a distance metric, i.e., Manhattan distance, to align the virtual scenarios to real-world scenarios. Computer vision algorithms, i.e., object detection and tracking, are adopted in References [292, 296] to extract the trajectories of the vehicles from the crash videos. Gambi et al. [189, 293, 294] utilize natural language processing techniques to extract the relevant information and then calculate the abstract trajectories for recreating the crash. Experimental results show that the method could accurately reconstruct the crashes in public reports, and the generated test cases are able to expose faults in open source ADS, i.e., DeepDriving [207].

There is also a line of work [297, 298, 299, 300] that focuses on narrowing the reality gap in the training process. By including components such as augmented data, small-scale cars, and real-world tracks, they could generate more realistic cases to train the perception model or reinforcement learning algorithms for automated driving.

Realism of simulators. A comparative study on the assessment of testing in different levels of simulation is performed by Antkiewicz et al. [301]. In their work, the authors study simulation-based testing, mixed-reality testing, and real-world testing on two scenarios, i.e., car following and surrogate actor pedestrian crossing. They propose various metrics, e.g., realism, costs, agility, scalability, and controllability, and based on these metrics, they compare the different testing schemes under evaluation. As their conclusion, they quantitatively show the performance difference among the testing schemes: although real-world testing is better in terms of realism, it is more costly, and less agile, scalable, and controllable, compared to simulation-based testing; the performance of mixed-reality testing is in the middle of them. Testing ADS in different simulators is studied by Borg et al. [302], in which they utilize search-based testing techniques to generate scenarios in two simulators, i.e., PreScan [303] and Pro-SiVIC [304]. They find notable differences of the test outputs, e.g., they detect different safety violations. Consequently, they recommend involving multiple simulators for more robust simulation-based testing in the future.

Although simulation-based testing cannot achieve the same realism as real-world testing, to what extent can the results of simulation-based testing benefit real-world testing? This question is investigated in Reference [305], where the authors perform simulation-based testing to identify critical scenarios and map them to a real-world environment. Their key insights show that \(62.5\%\) of the unsafe scenarios detected by the simulators translate to real collisions and \(93.3\%\) of the safe scenarios with the simulators are also safe in the real world. Another question is whether the simulator-generated dataset can substitute real-world datasets for DNN-based ADS testing, which has been studied in Reference [306, 307]. Moreover, they also compare offline testing, e.g., module-level testing, and online testing, e.g., system-level testing, in terms of their pros and cons. Experiments on DNN-based ADS show that the average prediction error difference on two datasets is less than 0.1, which means the simulator-generated dataset can serve as an alternative to the real-world dataset. Online testing is more suitable than offline testing for DNN-based ADS testing, since online testing could detect more errors, i.e., those errors caused by accumulation over time, than offline testing. Reway et al. [308] evaluate the simulation-to-reality gap by testing an object detection algorithm under three different environments, namely a real proving ground and two simulation software, considering four weather conditions. The gap is quantitatively calculated by considering metrics such as precision and recall on each platform. One of their experimental results is that the gap between real and simulation domains under nighttime and rainy conditions is larger than that under daytime conditions.

Overall, we have surveyed more than 90 papers dedicated to the system-level testing of the ADS. We find that those module-level testing techniques, such as search-based testing, sampling, and adversarial attack, are also widely adopted for finding failures arising from collaborations over different modules at the system level. Besides, more metrics, which can be found in Sections 6.1.2 and 6.1.3, are proposed or utilized for facilitating the testing process. Another observation is that more than 30 papers focus on bridging the gap between the simulation and the real-world environments, e.g., by introducing real components into the testing loop or by making a comparison between the simulation-based testing and the real-world testing.

Similarly to module-level testing, there still remain several open challenges for system-level testing of the ADS. For example, since the system executions during testing are expensive and time-consuming, it needs future exploration on how to accelerate the testing process. More discussions about the challenges and future research directions can be found in Section 8.

In this section, we construct a threat model to summarize the safety and security threats that each module may confront based on our survey results. To build the threat model, we first summarize the threats discovered in the papers that we survey in the module-level testing in Section 5; then, as a complement, we review the bugs shown in the empirical studies [34, 35] on open source ADS to understand the concrete issues encountered in each module during system development. Our threat model is shown in Figure 5.

Threats to sensing. In this module, existing studies mainly concern about the hardware aspect, e.g., those physical sensors that are critical hardware used in an ADS for collecting the information of the external environments. A common threat such as harsh weather conditions could reduce the capabilities of the intelligent sensors. There are also many deliberate attack techniques, such as jamming attack [52, 53, 54] and spoofing attack [55, 56, 57, 58] (see details in Section 5.1.2) that target this module and could interfere with these sensors and harm their normal functionalities.

Threats to perception. The perception module is the most investigated, and we collect 23 testing techniques dedicated to this module. Common threat comes from adversarial examples that are generated by adding perturbations to normal images, which can fool the deep learning models in the perception module to make incorrect predictions, as shown in References [61, 62, 63, 64, 65, 65, 66, 67, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 83]. Another type of threat is called Trojan attack [88, 89], in which malicious data are injected into the training data of the deep learning models. Moreover, in the case when the ADS requires an HD map from the cloud service, Denial of service [309] or fake HD map data [310] can interfere with perception tasks such as localization.

Threats to planning. With the data produced by the perception module, the planning module takes charge of several tasks, e.g., object trajectory prediction and path planning, and we collect eight testing techniques for this module. A common threat comes from the unwanted maneuvers of NPCs [137, 138, 139, 141, 142, 143, 144, 145], which can interfere with the prediction for moving objects and thus lead to an unsafe trajectory plan. Moreover, improper localization from the perception module can also threaten the accuracy of output trajectories.

Threats to control. This module mostly adopts those mature control techniques, e.g., MPC and PID, and thus are relatively hard to attack. One threat comes from the injected faults [151, 152], which could affect the longitudinal/lateral control of the vehicle. Another threat emerges when an emergency situation is encountered, e.g., when an emergency braking is needed, and the control module may fail to handle these cases. Moreover, the output signals are sent via the CAN bus to the ECU for controlling the vehicle. Since this process involves a data transmission between software and hardware, a potential threat is the interface mismatch [311], e.g., an inappropriate steering angle rate, in practical usage.

Datasets. In the context of ADS, deep learning components handle safety-critical tasks, e.g., perception and end-to-end control, so it is necessary to validate their robustness under various scenarios. This process typically relies on data from the real world, which is, however, hard to obtain in general. Fortunately, there are a collection of publicly available datasets to solve the problem, which involve large quantities of real-world pictures and videos recorded by onboard sensors. For example, the KITTI dataset [313] contains over 10,000 images of traffic scenarios, collected by a variety of sensors including high-resolution RGB/grayscale stereo cameras and a 3D laser scanner.

In this section, we summarize the scenario-driven datasets for ADS testing in Table 10. The first column shows the time when each dataset was released. The next three columns give the name, brief description, and the size of each dataset, and the last column indicates the related works that adopt these datasets. Note that the datasets for other machine learning testing tasks [337] that have nothing to do with ADS testing are not listed here; in other words, all the datasets listed here are dedicated to ADS testing.

As shown in Table 10, we collect 27 datasets released from 2004 to 2022, including popular ones like the KITTI dataset [313] and emerging ones like the CrashD [336] dataset. One observation is that these datasets span over various physical conditions, e.g., different times of the day [317], different weather conditions [329, 332], and different traffic density [327]. They also span over various application scenarios, such as urban street [316, 320, 322], highway [325, 326, 338], and intersection [331]. In addition, we find that some of these datasets are specific to a certain task, e.g., pedestrian detection [319, 321] and traffic sign detection [125, 314, 315].

As the column of references in Table 10 shows, several datasets such as the KITTI dataset [313] and the Udacity dataset [317] are frequently used in ADS testing due to the diverse tasks they support, such as object detection and semantic segmentation. However, we also find that a number of datasets have not been widely used, due to their own limitations. For example, the rounD [333] and openDD [334] can only be used for validating the behavior planning of ADS in the scenario of roundabouts; SYNTHIA [318] and GTA5 [339] contain synthetic images from virtual environments, which may be not realistic enough for ADS testing.

Tool stacks. As mentioned, simulation-based testing has become an important alternative approach for real-world testing. Simulators usually provide vehicle dynamics, e.g., longitudinal and lateral motion of the vehicle, and virtual traffic scenarios. Moreover, simulators can help generate those extreme scenarios for testing, e.g., harsh weather, which are rarely encountered in the real world. There have been many advanced simulation platforms developed for ADS testing in recent years. For example, Carla [345] is an open source simulator for ADS training and testing, which supports various sensor models and environmental conditions.

In this section, we summarize the simulation platforms, namely the simulators usually used for ADS testing in Table 11. As shown in the table, we collect 20 simulation platforms, including classical platforms such as Matlab/Simulink [340] and CarSim [341], and emerging popular simulators, such as Carla and LGSVL [351]. Since these simulators have their own pros and cons, we compare them in the table and focus on several aspects of interest, e.g., their gap from real environment. Specifically, the first column lists the name and the second column shows the accessibility of each simulator. The third column is relevant to physical aspects, that is, whether the simulator allows for customizing a dynamic model and whether it is a soft-body or rigid-body based simulator. The fourth column indicates the level of support for mixed-reality testing, including model-in-the-loop (MiL), software-in-the-loop (SiL), HiL, and ViL. The fifth column presents the capability of these simulators to complement each other, e.g., whether they support co-simulation with other simulators. The last column indicates the related works that adopt these simulators in their research.

Based on the table, we can draw the following conclusions:

Overall, it can be seen from the reference column that Matlab/Simulink and BeamNG have been widely used for ADS testing. Simulators like NVIDIA Drive Sim and Gazebo also have great potential for future research, since they cover multiple features we list in the table, e.g., whether they could perform ViL testing or co-simulation with other simulators.

Moreover, we also introduce several publicly available systems under test in Table 12. OpenPilot, Apollo, Autoware, and Pylot are all modular systems and have already been described in Section 2.3, so we will not repeat them here. In addition to modular ADS systems, there also exist open source end-to-end based systems: LBC [356] is an imitation learning controller that uses camera images and direction commands as input to control the direction of the vehicle in the lane and intersection; DeepDriving [357] and Nvidia CNN Lane Follower [358] are also widely used CNN-based end-to-end controllers; and Open source DNN models from the Udacity self-driving challenge, such as Chauffeur [179] and Epoch [180], are another line of end-to-end driving controllers. Besides, there are also driving agents and controllers from simulators, such as BeamNG and Carla. BeamNG.AI, which has been mentioned in Section 2.3, is an AI agent in BeamNG simulator and could accept virtual images in the simulator as input for path planning and trajectory tracking. Carla PID is a specific module in Carla that performs calculations at the motion planning stage and estimates the acceleration, braking, and steering inputs required to reach target positions.

Programming languages. To systematically generate test cases, it has become a trend to propose new programming languages for testing scenario description. In this way, the generation of a new test case boils down to writing a program that describes the scenario. Also, researchers can make use of existing coverage criteria for programs, such as the code coverage criteria, to assess the adequacy of the generated tests.

To define such a programming language, researchers need to formally express the basic elements in an ADS scenario, e.g., the ego car, other cars, pedestrians, and static objects. Since these languages are usually dependent on existing formats, they vary in their ways of expressing those elements, based on their dependent formats. For instance, Scenic [361], a pythonlike language, requires users to define those objects as variables; in contrast, GeoScenario [360] provides users with a graphical interface where users can drag the icons to describe a scenario. Moreover, these languages usually do not emerge independently; instead, they come with specific simulators or even specific ADS.

In this section, we summarize the state-of-the-art programming languages for test case generation in Table 13, and introduce their dependent formalism, their bonded simulator, other features, and their adoption in ADS testing. There exists literature, e.g., Reference [367], that surveys programming languages for the test generation of ADS. Compared to Reference [367], our main aim is to show the ecosystems and the landscape of the use of these languages in ADS testing, as a reference for the readers to better understand the testing techniques in Sections 5 and 6. Also, our study includes some latest achievements, e.g., paracosm [366] and SceGene [365], in this direction.

As shown by Table 13, we collect eight representative programming languages, including the classic ones, such as OpenScenario, that have been widely used in different stages of the development of ADS, and emerging ones, such as paracosm [366]. As our findings, first, these languages offer distinct capabilities for different purposes, e.g., languages are designed for different purposes and attached with different features, e.g., Scenic [361] allows probabilistic sampling for testing driving systems with machine learning components; SceGene [365] designs bio-inspired operations, such as crossover and mutation, for scenario generation. Second, some of these languages provide more user-friendly features; for instance, some of the languages, e.g., SceML [363] provide a graphical user interface for users to define their scenarios. However, as the column of reference shows, most of these languages have not been widely adopted in practice. This can be due to several reasons: One possibility is that some languages are still too specialized for practitioners to adopt them in their work; also, since many of the languages, such as GeoScenario [360], are designed for specific systems, they are still ad hoc and not easily extensible to be adopted in other systems.

In conclusion, programming languages are increasingly deemed as powerful weapons for test case generation in ADS testing, but currently they have not been widely adopted in practice.