Testing, Validation, and Verification of Robotic and Autonomous Systems: A Systematic Review

In the domain of autonomous cars, qualitative models have been used to reason about Human-Machine Interactions [162, 231] and high-level decision-making, particularly regarding ethical concerns [60] and safety [81, 199]. Most quantitative interventions propose use of formal models in their methodology [58, 60, 71, 72, 147, 162, 197, 205, 229, 231]. For instance, Yun et al. [231] propose a strategy that formalises Human-Machine interaction into SysML language to help steer the testing process. In the same vein, Naujoks et al. [162] provide a DSL using a taxonomy of use cases to cover transitions and modes in Human-Machine Interaction interfaces used in the verification process. Sun et al. [205] employ Satisfiability-Modulo-Convex encoding to build finite state abstractions of the learning component and formally verify it. Dennis et al. [60] focus on formalising and verifying ethical concerns in BDI agents. More informal approaches include the use of 3D simulation by describing scenarios in UML notation modelling [208] and the use of graphical notations for safety assurance analysis [81, 199]. A mix of formal and informal models is employed by Heitmeyer et al. [98], where they provide two new tools to be included into their toolset (FORMAL [97]). The first tool synthesises formal models from scenarios written in Event Sequence Charts, and the second tool incorporates a 3D simulation tool (eBotworks [92]) into their toolset.

In the aerial domain, there are only a handful of interventions that use qualitative models for testing of aerial vehicles, mostly regarding safety and security concerns. For instance, linear temporal logic is used more than once to formalise safety assurance cases [17, 41]. Hagerman et al. [93] make use of finite state machines to extract security test suites, and Bhattacharyya et al. [31] focus on formally verifying the boundaries beyond which the agent are designed to operate by translating models from a cognitive architecture (Soar [124]) into UPPAAL.

Only two studies have been found in this category. Andrews et al. [10] model autonomous systems and their environment using Petri nets to generate test cases and apply their technique to a case study in the human-robot interaction domain. Furthermore, in the context of software product lines, Mansoor et al. [149] conduct a case study on a family of surgical robots by employing formal analysis, feature modelling, and testing; they discuss the key challenges and lessons learned from the case study.

Most of the included interventions in this category concerned abstract representations of multi-agent autonomous systems and provided efficient algorithms for parametric (formal) verification or state-space reduction techniques [8, 22, 23, 24, 119, 120, 121, 122, 134, 136, 167, 187]. Similar to the previous item, most of the interventions used Linear Temporal Logic or variants thereof to model safety properties [88, 222]. Formal modelling and verification of human-machine interaction is also a common theme in this category [69, 221, 222].

In this domain, we see a strength in integrating code-level abstractions (e.g., for individual components or functions) with system-level specifications (for vehicles and fleets of vehicles and using); such integrations are then tested using simulation and formal verification frameworks. The quantitative models used for such testing intervention often pertain to vehicle dynamics and also probabilities arising from communication frameworks. AbdElSalam et al. [1] present a framework for verification of ADAS and autonomous vehicles that uses SystemC TLM models for virtual ECUs. Transaction-Level Models provide a high-level abstraction of the SystemC components that are used in virtual ECUs. These models are then integrated with the vehicle and traffic models for simulation. Parametric modelling of CAVs as a network of timed automata is used by Arcile et al. [16]. In this work, VerifCar tool is applied to asses the impact of communication delays on the decision algorithms of CAVs and to check the robustness and efficiency of such algorithms. Similarly, variations of extensions of timed-automata, probabilistic timed automata, and stochastic timed automata are used [12, 137, 226] for modelling the behaviour of autonomous vehicles to verify properties of different decision-making and collision avoidance algorithms. Barbot et al. [19] use statistical model checking to verify an autonomous vehicle controller where the controller is specified in C++. A set of safety properties specified in HASL, a quantitative variant of linear temporal logic, are verified for a controller. Betts et al. [30] compare the effectiveness of two search-based testing methods, genetic algorithms and surrogate-based optimisation, for test case generation for UAV flight control software. There are several works in this category that provide and use simulation platforms [51, 51, 70, 87, 128, 131, 133, 172, 206].

In this domain timing information Human-machine interaction is formalised (through a formalisation of a cognitive architecture) in terms of networks of timed automata, and UPPAAL is subsequently used for their verification. [18, 33, 42, 63, 103, 139, 161, 201, 234]

There are a few studies in this domain that consider quantitative models. Among these studies, the majority employ formal models [12, 37, 137]. Statistical model checking is one of the techniques that is used [12] to verify performance of transport robots based on behavioural models, stochastic timed automata, using UPPAAL SMC. Furthermore, model checking of Markov models is used [37] to verify PCTL properties of swarm robotics behaviour in the design phase. The models then are used as a blueprint for implementation and simulation. Probabilistic model checking of unmanned surface vehicles is another technique used [137]. PRISM model checker verifies PCTL properties of USVs on probabilistic timed automata as behavioural model. Other work in this category [176] use a designed DSL (graph-based models) to describe the system behaviour that is used as test model for generating test cases.

As a general observation a considerable portion of all interventions (23% of papers) in this category have been an out-of-the-box application of model checking tools (mostly using PRISM [123], in some cases UPPAAL [125] and FDR [85]) to specify small-scale robotic case studies; another prevalent category (30% of papers) concerns theoretical papers on various logics and model-checking algorithms for multi-agent robotic systems. Notable exceptions of this general theme include languages and toolsets for rigorous simulation [47, 48, 156, 182] using formal verification as part of the design of robotic interaction protocols [64, 228], using formal verification to analyse human-robot interaction [69, 158, 175, 216, 221, 222], and generating test cases from formal models [14, 46, 132, 163, 192]. Another interesting intervention concerned comparison of different hybrid-systems solvers [39, 40] for formal verification of robotic applications. Also, the theorem prover Isabelle/HOL has been used to formalise safety assurance claims [88]. Also, model checking has been used to train policies in reinforcement learning [171]. Compositional techniques has been used to reduce the complexity of the model checking problem [170, 189]. Another noteworthy attempt is in formalising and verifying ethical concerns [60, 62].

There are a number of non-exhaustive testing from formal models [133, 208] and scenario generation [157] techniques proposed in this domain. Model synthesis from scenarios has also been studied [98]. The applications of different verification techniques such as formal verification based on supervisory control, model checking, and deductive verification have been studied in industrial context [197]. Also in this domain, a framework for validating ethical policies has been developed [229], and human-machine interaction for user interfaces have been validated [231].

Apart from applying traditional model checking [103, 201] and simulation [56] techniques to this specific domain, we observe notable attempts to combat the huge state-space of domain-specific models by employing statistical model checking [18] and runtime monitoring [63]. Most papers in this domain have focused on constructing safety models/properties or even coming up with safety specification frameworks [17, 41]; however, energy-efficiency [234] and to a limited extent, security [93], have been addressed as well.

The landscape in this domain is much more sparse and scattered. As usual for this category, there are a number of applications of model-checking tools. There is a single paper on test description and test-case generation [176]; also, variability is an under-studied aspect in robotics that has been handled in this context [149] and, finally, there is an industrial case study on the application of model checking [12].

Formal verification is prevalent in this category; besides parametric verification of multi-agent systems using variations of epistemic logic [8, 22, 24, 121, 135, 187], formal verification using timed automata has also been used in a strategy that composes verification problems into smaller ones [189] and in the verification of path planning [7]. Furthermore, Araiza-Illan, Pipe, and Eder [14] use BDI models and model checking of probabilistic timed automata (in UPPAAL) to generate test sequences for human-robot collaboration tasks. Another use of UPPALL in this category is for model checking of ROS applications that makes use of an ad hoc translation from ROS to UPPAAL [94].

Verification of probabilistic aspects can be found in a few studies. Zhao et al. [235] employ Bayesian inference to estimate the distribution of parameters of Markov chains. Then, they combine formal verification, synthesis, and runtime monitoring to check that the estimated parameters are not violated. Pathak et al. [171] make use of probabilistic properties of Markov chains for self-repair capabilities in robots and tie those into a formal verification process (of PCTL formulae). Araujo, Mota, and Nogueira [15] apply probabilistic model checking to verify whether a robot trajectory (described in terms of an algorithm) satisfies specific behaviours or properties (stated as temporal formulas).

The combination of CSP and FDR can be seen in a few studies. Cavalcanti et al. [46] generate mutations on RoboCart models and feed the mutated CSP [101] (obtained from the RoboChart models) into FDR, which yields a counter-example that is used for testing. Yueng et al. [228] detail a process to contribute to the design of simulation experiments by analysing variations of timing parameters in CSP. They show that the simulation experiment can yield different results in performance and in behavioural terms. Sumida et al. [204] demonstrate a case study in which they model a LEGO robot (EV3) and verify it in FDR for deadlock and livelock.

Several other studies employ different strategies to achieve different goals. Gainer et al. [77] synthesise formal models from control rules of robots in a DSL and input those models into NuSMV for model checking. Doan et al. [64] employ MAUDE to formally verify gathering of robots in a RING network. Santos et al. [192] work on generating unit testing for ROS components and property-based testing for ROS using the Hypothesis tool [154]. Bresolin et al. [39, 40] apply reachability analysis of Hybrid Automata in ARIADNE [26] to analyse the dynamics of surgery robots.

A majority of papers in this category introduce a simulation tool [1, 51, 54, 70, 129, 155, 206, 232] combining vehicle kinematics with other aspects of vehicle modelling such as communications [51], vision-based algorithm [129], and fuel consumption [70]. There are two interventions that use search-based testing [25, 30]. Surrogate modelling, where a higher-level model is used to steer the search, is used in both approaches. Another approach uses past data to identify challenging situations and embed them into test-cases (using an XML structure) [200].

There are also a number of process interventions describing a process for safety assurance [81, 236] and testing Human-Machine Interfaces [162]. Some papers do use a well-defined syntax or a mathematical notation, but are classified as informal; in our classification, if a model does not have a rigorous formal syntax, semantics, and reasoning method, then it is classified as informal. These include using XML as a formal model [200], mathematical descriptions of vehicle kinematics (see simulation tools above), and probabilistic descriptions for risks [236].

All interventions reported here concern simulation tools for modelling dynamics and control [42, 139, 161] and communication of aerial vehicles [67]. It is remarkable that these simulation tools rely on entirely different context tools, which will be analysed further in RQ3.

In this category, the majority of interventions again propose simulation tools [117, 148, 184, 202] or a simulation architecture [224]. (Note that there are two simulation tools that address marine robots [148, 212], but since we did not have a separate class for such robots, we classified it here.) There are, however, a couple of interventions concerning test-case prioritisation [127] and automated unit-test execution [34], and runtime monitoring [102]. The runtime monitoring environment [102] provides an integration with ROS implementations.

Concerning the effectiveness measures for road vehicles, collision analysis is the most prominent metric, including analysis on the number of collisions [11, 16, 49, 54, 113, 130, 227], probability of collision [19, 128], and severity [209]. Furthermore, a few studies focus on the analysis of deviation from the intended path in terms of spatial and rotational deviation [30, 32, 50, 78]. As for measures for test adequacy, probability of faulty and rare events [155, 166], and the number of tests generated [208] have been studied; however, these latter measures are not domain-specific and similar measures have been used in other sub-domains.

Only four studies have been included in this category. As measures of SUT efficiency, Desai et al. [63] measure obstacle avoidance and plan execution. Similar to road vehicles, studies on probability of completing a task for aerial vehicles are found in the literature [18, 234]. Zhao et al. [234] take a further step by analysing expected mission time and expected number of battery recharges during a mission. Conversely, as a test adequacy metric, Reference [198] measures the accuracy of its simulation.

With respect to mobile robots, only a handful of papers’ collective metrics related to effectiveness. As in other sub-domains, collision avoidance [137] and probability of satisfying requirements [37] can also be found here. Brambilla et al. [37] also measures the improvement in the behaviour of a robot by analysing the number of objects retrieved. Arai et al. [12] also consider a similar measure of improvement in the behaviour but in terms of device utilisation. The only metric regarding testing adequacy is number of failures detected studied by Proetzsch et al. [176].

As a unique measure of effectiveness, Ruijten [188] detects anthropomorphism in its subjects, which is a measure of human likeliness in robots. Several studies focus on analysing the probability of completing a mission successfully [15, 141, 158, 235]. Studies on safety [7, 163, 216] are also commonly found in this category. For instance, Viventini et al. [216] focus their efforts on analysis of hazards, such as the number of hazards identified, number of types of hazard, and number of risk reduction measures taken.

Mullins et al. [159] use precision, convergence, and resolution for efficiency in testing. In a number of studies on verification of autonomous systems [71, 72, 110] the size of the state space as well as the total memory footprint [71] in their evaluation is measured to evaluate efficiency. Sun et al. [205], in the verification of finite state models, use abstraction and verification time to estimate the efficiency in their work. Verification time is used in a number of studies [16, 71, 72, 91, 110, 147, 165] to measure efficiency. Gladisch et al. in Reference [87] use simulation time to measure efficiency. Similarly, Bi et al. use simulation time [32] for measuring the efficiency of their work. Fayaz et al. [70] measure test duration in their evaluation. To evaluate the efficiency of the work, Reference [172] measures CPU usage and network bandwidth. Bode et al. [35] measure the cost (euros) of application of their approach as a notion of efficiency. Li et al. in Reference [131] measure computational time in testing and comparing various autonomous vehicle decision and control systems.

Sirigineedi et al. [201] use verification time and number of states in Kripke structures, as system model, for measuring efficiency in their work. Urso et al. [67] use simulation time once using Gazibo as notion of efficiency. Zhao et al. [234], in their work that is using prism model checker, consider the expected measure time as a notion of efficiency.

Andrews et al. [10] consider the number of states and transitions as a measure of efficiency for the testing method. Arai et al. [12] present results on the response time, throughput, and device utilisation as measures of efficiency for the system under test.

Verification time is commonly used in a number of studies [77, 136, 141, 171, 182, 189] to measure efficiency. The number of states is another common notion measured in evaluations [15, 77, 122, 135, 182, 228]. Furthermore, testing and simulation time is used to measure efficiency in a set of studies [83, 117, 192]. Althoff et al. in Reference [6] measure the reduced idle time in self-verifying robots. Munawar et al. [160] measure latency and number of steps in simulation of surgical robots. Search time and optimisation time are used by Collet et al. [53] in evaluation of testing single arm robots. Gerstenberg et al. in Reference [83] measure task completion time in simulation. Vicentini et al. in Reference [216] present the total time for risk reduction and hazard identification in their evaluation of formal verification techniques. Probability of task completion, number of executed instructions, and time for completing the task are other measures used in evaluation of model checking autonomous systems [15]. Muhammad et al. [158] measure time and probability of task completion in probabilistic model checking of cooperative robot interaction.

Variations of structural coverage can be found in studies within this sub-domain. Neves et al. [58] developed a tool that conducts post-analysis (based on meta-models) on outputs collected from field testing of autonomous vehicles. Their tool aims to expand test coverage by exploring functionalities in the meta-model that were not covered during the field testing. Majzif et al. [143] devise a process that guarantees coverage of safety standards. They abstract results from component testing and make use of meta-models and situation graphs to compute a system-wide degree of test coverage and derive new scenarios to cover unexplored situations. Tatar [209] presents a method (implemented in TestWeaver [106]) for testing and validation of ADAS systems. The tool generates scenarios to cover relevant system states and feeds back previous executions to guide the next round of testing.

In the only paper in this category, Bicevskis, Gaujen, and Kalnins [33] developed new methods for testing and validation of autonomous processes collaboration. They build a collaboration model using an extended finite state machine and employ symbolic execution and feasibility tree analysis to check that all relevant states can be reached in the model. They have evaluated their strategy using a UAV case study.

Tian et al. [211] propose DeepTest, a tool for testing Deep Neural Networks in autonomous vehicles. It generates tests that explore different parts of the DNN logic with the goal of maximising neuron coverage. Araiza-Illan, Pipe, and Eder [14] propose a methodology for generating test cases that achieve high code coverage in human-robot collaborative tasks. They developed a testbench for ROS that makes use of belief-desire-intention (BDI) agents to generate valid and human-like tests. Structural coverage of Petri nets models have been utilised by Sagglietti in different contexts, such as in the generation of test cases for autonomous agents [132] and the verification of reconfiguration behaviour of autonomous agents [190, 191].

There are a handful of open-source tools for formal verification [16], testing [54], and simulation [4, 80, 211] of autonomous vehicles. Testing and simulation seem to be gaining some strength with respect to open-source tools, and we see tools for scenario generation, testing, and simulation of autonomous vehicles within various traffic scenarios. For connected vehicles, VerifCar [16] is a framework based on timed automata and is dedicated to modelling and verifying and validating connected autonomous vehicles policies.

Garzón and Spalanzani [80] present a tool that combines 3D simulation (for ego-vehicle control) with a traffic simulator (which controls the behaviour of other vehicles). The goal is to test the ego-vehicle in realistic high-traffic situations. The FLPolyVF tool [4] connects functional verification, sensor verification, diagnostics, and industry/regulatory communication of autonomous vehicles while checking the effects of using different scenario abstraction levels. The MoVE tool [54] provides the possibility of modelling pedestrian behaviour. The framework focuses on testing autonomous system algorithms, vehicles, and their interactions with real and simulated vehicles and pedestrians.

Gambi, Mueller, and Fraser present the AsFault prototype tool [78]. The tool combines procedural content generation and search-based testing to automatically create challenging virtual scenarios for testing self-driving car software. Tian et al. [211] propose DeepTest, a tool for testing Deep Neural Networks in autonomous vehicles. It generates tests that explore different parts of the DNN logic with the goal of maximising neutron coverage

All new tools reported here concern simulation tools for modelling dynamics and control [18, 139] and communication of aerial vehicles [67].

Lugo-Cárdenas, Luzano, and Flores [139] introduce a 3D simulation tool for UAVs whose focus is on assisting the development of flight controllers. Analogously, D’Urso, Santoro, and Santoro [67] also present a simulator for UAVs, called GzUAVChannel. The framework combines Gazebo, Autopilot, and NS-3 network simulator to provide a 3D visualisation engine, a physics simulator, a flight control stack, and a network simulator to handle communications among unmanned aerial vehicles. On the stochastic side of software verification, Bao et al. [18] present a prototype tool for parametric statistical model checking that can cope with complex parametric Markov chains where state-of-the-art tools (such as PRISM) have timed out. They provide evidence of their tool efficiency by conducting an industrial case study.

Several open-source tools have been proposed in this category, with a majority of them being simulators. The only exceptions are a formal verification tool [175] for human-robot interactions and two runtime verification tools [63, 102].

Rohmer, Singh, and Freese introduce VREP [184] a popular robotics physics simulator that is now known as CoppeliaSIM. The tool uses a kinematics engine and several physics libraries to provide rigid body simulations (including meshes, joints, and multiple types of sensors). Brambilla et al. have developed ARGOS [37], which is a multi-physics robot simulator that can simulate large-scale swarms and can be customised via plug-ins. In the Matlab environment, the FROST tool [100] is an open-source Matlab toolkit for modeling, trajectory optimisation, and simulation of robots, with a particular focus in dynamic locomotion. Munawar and Fischer [160] present the Asynchronous Framework, which incorporates real-time dynamic simulation and interfaces with learning agents to train and potentially allow for the execution of shared sub-tasks.

For underwater robots, three new tools have been introduced: Manhaes and Rauschenbach present UUV simulator [148], which is an extension of Gazebo accommodating the domain-specific aspects of underwater vehicles. Cieslak et al. introduce Stonefish, a geometry-based simulator [52] that can be integrated with ROS. Last, the MARS [212] tool provides simulation environments for marine swarm robots. As for tools in the human-robot interaction (HRI) domain, RoVer [175], provides visual authoring of HRI, formalisation of properties in temporal logic, and verification that the interactions abide by a set of social norms and task expectations.

Huang et al. present ROSRV [102], which is a runtime verification framework that can be used with ROS. Desai et al. [63] present a runtime verification framework based on Signal Temporal Logic [144], where an online monitor checks robustness on partial trajectories from low-level controllers (in the context of surgical robots).

Mullins et al. [159] developed a tool (RAPT - Range Adversarial Planning Tool) for generating test scenarios. The tool employs an adaptive search method that generates new scenarios based on the performance and results of the previous one. A clustering algorithm ranks the scenarios based on the performance type and how close they are to the boundaries of each cluster. The boundaries are based on notions of efficiency, diversity, and scaling.

Shah et al. [198] introduce the AirSim simulator that generates training data for building machine learning models used in autonomous air crafts. It offers physical and visual simulation, including models of physics engine, vehicle, environment, and sensors.

There are two tools included in this category: a formal specification and verification tool [46, 47, 48, 156] and a simulation tool [233]. Cavalcanti et al. [46, 47, 48, 156] introduce RoboTool, supporting graphical modelling, validation, and model checking (via FDR [85]) of robotic models written in RoboChart [156] and RoboSim [48]. Zhang et al. [233] introduce CyberEarth, a framework for program-driven simulation, visualisation, and monitoring of robots. The tool integrates modules from several other open-source tool such as ROS [177] and OpenSceneGraph (OSG) [43].

The three tools included in this category comprise two tools for formal analysis [98, 165] and a simulation tool [232], each of which are explained further below.

Heitmeyer and Leonard [98] introduce two tools integrated into the FORMAL framework; The tools synthesise and validate formal models. The first tool synthesises a formal Software Cost Reduction (SCR) requirements model from scenarios, and the second tool combines the existing SCR simulator [96] with eBotworks 3D simulator to allow for simulation of continuous components.

O’Kelly introduces APEX [165], which is a tool for formally verifying the trajectory planning and tracking stacks of ADAS in vehicles. Zhang et al. present RoadView [232], a photo-realistic simulator that tests performance of autonomous vehicles and evaluates their self-driving tasks.

The three tools included in this category are diverse and range from simulation [215] to formal verification [75] to model-based testing [53].

Verma et al. [215] present a Flight Software simulator that is used to simulate MARS Rover missions. The simulator assists in predicting the behaviour of semi-autonomous systems by providing the capability for human operators to check if their intent is correctly captured by the robot prior to execution in different scenarios and environments. Foughali et al. [75] implement an automatic translation from GenoM [145], a robotics model-based software engineering framework, to the formal specification language Fiacre [28], which can be fed into TINA [29] for formal verification. Collet et al. [53] introduce RobTest, a tool for generating collision-free trajectories for stress testing of single-arm robots. It employs constraint programming techniques to solve continuous domain constraints in its trajectory generation process.

A few papers employ case studies with a focus on collision avoidance for road vehicles [4, 49, 71, 128]. They all focus on detecting imminent collision using built-in sensors. Similarly, Gaurehof, Munk, and Burton [81] conduct a case study where a machine learning function detects pedestrians using a video analysis.

Several case studies concentrate on driving scenarios and manoeuvres, such as lane changing [19, 165], lane and path following [11, 232], merging [80], roundabouts [227], traffic scenarios [131, 155], parking [50], and overall cruise control [72, 110, 130, 147, 217]. A focus on the actual decision-making and path planning can be seen in a handful of the case studies [50, 51, 226] as well. Hardware-in-the-loop simulations [95] and human-machine interaction for driving assistance [231] can also be seen among the included case studies.

The small-scale case studies in this domain only present theoretical or very limited models of UAVs, such as a surveillance drone [63] or a model for UAV launch [93]. Moreover, Brunel et al. [41] conduct a safety case analysis, while Bu et al. [42] explore simulation and realistic testing of vision-based object tracking for UAVs. Aerial drones in co-operative scenarios are also the subject of two case studies [103, 201].

Only two small-scale case studies were included here: Lu et al. [137] make use of PRISM model checker to investigate three collision avoidance algorithms in an unmanned surface vehicle model with a dynamic intruder, and Arai and Schlingloff [12] employ a model checking technique on a transport robot model.

Several case studies can be found within this category, with the vast majority being small models that are applied to demonstrate the respective intervention. Here, we briefly present and discuss some of them.

Walter, Täubig, and Lüth [218] provide an algorithm that increases safety through formal verification using the theorem prover Isabelle; the case study is a small robot. Nguyen et al. [163] provide a multi-step process to verify correctness of autonomous agents and apply it to a cleaner robot. Fu and Drabo [230] model a humanoid robot in an extension of Petri nets (called Predicate Transition Reconfigurable Nets - PrTR Nets) and formally verify it. Lill et al. [132] also make use of Petri nets, however, they develop models of cooperative forklifts and simulate scenarios where the robots decide which one has the priority when passing through narrow pathways.

Farulla and Lamprecht [69] conduct a case study on human-robot interaction processes that have been modelled in DIME and show how they can be verified with the GEAR model checker. Zhang et al. [233] have built a virtual simulation platform, CyberEarth, for robotics and cyber-physical systems. A visual coverage task for UAVs is also introduced to demonstrate the platform. Dennis and Fisher [62] apply an agent verification approach to verify the correctness of an agents ethical decision-making. Doan, Bonnet, and Ogata [64] specify and formally verify, using the model checker Maude, a robotic gathering model.

The industrial case studies involving road vehicles included in our survey typically involve verifying specific components of such systems.

In the context of advanced driver assistance systems (ADAS), Abdessalem et al. [25] generate test cases for such a system that can visually detect pedestrians. Zhou et al. [237] introduce a framework for virtual testing of advanced driver assistant systems that uses real world measurements. Kluck et al. [113] consider virtual driving scenarios for testing automated emergency breaks. AbdElSalam et al. [1] use Hardware Emulation-in-the-loop to verify Electronic Control Units (ECUs) for ADAS systems.

Fayazi, Vahidi, and Luckow [70] implement a vehicle-in-the-loop verification environment and conduct field testing in the International Transportation Innovation Center (ITIC).

Gladisch et al. [87] select case studies that use industrial automated driving (adaptive cruise control, lane keeping, and steering control scenarios) to evaluate their search-based testing strategy. Abdessalem et al. [2] generate test cases for the SafeDrive system, which contains the following four self-driving features: autonomous cruise control, traffic sign recognition, pedestrian protection, and automated emergency braking.

Shah et al. [198] build a model of quadrotor with pixhawk controller in their newly developed simulator, AirSim, that includes a physics engine and supports real-time hardware-in-the-loop. Rooker et al. [185] demonstrate their validation framework for autonomous systems in a farming context with simulations and field testing. They employ both UAVs and ground mobile systems. Bhattacharyya et al. [31] apply formal verification methods to an autonomous CoPilot agent.

The only study in this category is the study conducted by Rooker et al. [185], which is also mentioned above. In summary, they demonstrate their validation framework for autonomous systems in a farming context with simulations and field testing.

Jacoff et al. [105] conduct field testing for the performance evaluation of robots used in disaster scenarios. Verma et al. [215] present a Flight Software simulator that is used to simulate MARS Rover missions. They demonstrate their approach with a case study. Satoh [194] conducts a case study using a physical transport robot to demonstrate their framework that can emulate the robot’s physical mobility.

Manhaes and Rauschenbach [148] model the Sperre SF 30k ROV underwater robot (RexROV) in the demonstration of the simulator for unmanned underwater vehicles. Uriagereka et al. [214] conduct simulation-assisted fault injection to assess safety and reliability of robotic systems. The feasibility of their method is demonstrated by applying it to the design of a real-time cartesian impedance control system. Gainer et al. [77] conduct a case study in the context of verification of human-robot interaction using the Care-O-Bot robotic assistant.

Many different case studies have been included in this category. We briefly discuss the most distinguished ones. For instance, Neves et al. [58] developed a tool that conducts post-analysis (based on meta-models) on outputs collected from field testing of autonomous vehicles. Five field testings involving a program to control the navigation of an autonomous vehicle, CaRINA I, were performed. Zofka et al. [238] present the framework Sleepwalker for verifying and validating autonomous vehicles and demonstrate the benefits of their framework using different instances stimulating an autonomous vehicle.

Mullins et al. [159] have developed a tool (RAPT - Range Adversarial Planning Tool) for generating test scenarios to be employed on the System Under Test. Their tool is applied to realistic underwater missions. Heitmeyer et al. [98] synthesise software cost-reduction models of multiple autonomous systems to be used in a simulator integrated with the eBotworks simulation tool. Gruber and Althoff [91] present a reachability analysis tool (Spot) that finds counter-example to property violations. Their tool is evaluated using the CommonRoad benchmark PM1:MW1:DEU_Muc-3_1_T-1.

Pereira et al. [172] employ several small case studies in their attempt to couple two simulators, namely, SUMO and USAR-Sim. Pasareanu, Gopinath, and Yu [170] present a compositional approach for the verification of autonomous systems and apply the technique on a neural network implementation of a controller for a collision avoidance system on the ACAS Xu unmanned aircraft. Bi et al. [32] present a deep Learning-based framework for traffic simulation and execute several scenarios of intersections with and without pedestrians.

Not many studies have applied their intervention to benchmarks of aerial systems. Bicevskis, Gaujens, and Kalnins [33] develop models for the testing of UAV and UGV collaboration in the Simulink environment. Mutter et al. [161] also explore the simulation of UAV models in Simulink and discuss the results when combining the platform and environment models. D’Urso, Santoro, and Santoro [67] simulate leader-follower UAV scenarios in their framework. Their goal is to combine four simulation environments: a 3D visualisation engine, a physics simulator, a flight control stack, and a network simulator.

Wang and Cheng [219] present a hardware-in-the-loop simulator for drones that can generate synthetic images from the scene as datasets, detect and verify objects with a trained neural network, and generate point cloud data for model validation. They simulate and conduct field testing on a physical UAV. Zhao et al. [234] model an unmanned aerial vehicle (UAV) inspection mission on a wind farm and, via probabilistic model checking in PRISM, show how the battery features may affect verification results.

Two studies fit this category. Proetzsch et al. [176] use a designed DSL (graph-based models) to describe the system behaviour of the autonomous off-road robot RAVON. The model is used as test model for generating test cases. Brambilla et al. [37] model a probabilistic swarm that is checked in PRISM to evaluate their property-driven design method.

Several studies have applied their intervention to benchmark case studies of generic/immobile robots. We briefly discuss the most distinguished ones.

Tosum et al. [213] present a design framework that facilitates the rapid creation of configurations and behaviours for modular robots. They have demonstrated their framework on the SMORES robot. Halder et al. [94] use the physical robot Kobuki as a case study, over which properties are automatically verified using the UPPAAL model checker. The focus of their approach is to model and verify ROS systems using real-time properties. Laval, Fabrese, and Bouraqadi [127] introduce a methodology to support the definition of repeatable, reusable, semi-automated tests and apply it to a two-wheel differential drive robot.

Bohlmann, Klinger, and Szczerbicka [36] automatically generate a model of a parallel delta robot on-the-fly. Their method for model generation is based on machine learning and symbiotic simulation techniques. Mariager et al. [150] design and field-test a robot that interacts with adolescents with cerebral palsy. Althoff et al. [6] propose a framework (IMPROV) for self-programming and self-verification for robots, which is demonstrated on a physical robotic arm. Wingand et al. [224] have developed CoSiMA, which is an architecture for simulation, execution, and analysis of robotics system. They conduct experiments on the humanoid robot COMAN.

With respect to swarms, a number of theoretical studies [119, 120, 121] focus on scaling up the parametrised model checking problem to large swarm sizes. They employ various types of epistemic extensions of CTL as property specification languages. Their models include case studies on clustering of swarms, which synchronise to gather in a certain area. Cybulski et al.’s contribution [56] to the field is a simulation framework for the behaviour of UAV swarms. The framework also allows for performing simulations with a user-defined map of the environment.

Regarding the use of qualitative models of cooperative systems, our search only resulted in three studies, two of which employ variations of Petri nets as their models. Lill et al. [132] make use of Petri nets to develop models of cooperative forklifts. The forklifts communicate to decide which has the priority when passing through narrow pathways. Sagglieti et al. [191] employ Coloured Petri nets and classify cooperation in three distinct levels: perception-based, reasoning-based, and action-based cooperation. To demonstrate their strategy, they model platooning-like scenarios where different robots follow each other. The third study, by Doan et al. [64], is a more theoretical study of parametric model checking that employs model checking for multiple small robots gathering in circular configurations using a ring-based topology. Their study focuses on model checking the underlying distributed system and the properties written in LTL.

Three out of four papers in this category deal with probabilistic behaviour: Lomuscio et al. [135] perform model checking on probabilistic LTL, while Anmin et al. [7] and Brambilla et al. [37] both describe properties in PCTL. Cavalcanti et al. [47], however, models timed dynamics in CSP.

The studies that have been classified in this category employ variations of temporal logic as their properties, such as LTL [103, 107, 158] and CTL [16]. As an exception to that list, Bicevskis et al. [33] provide a simulation environment in Simulink.

Kouvaros et al. [119, 120, 121] provide several theoretical studies in the field of formal verification and model checking of autonomous systems. They have demonstrated the applicability of their strategy on a case study of gathering of UAVs swarms. With respect to probabilistic systems, three studies have been found: Lomuscio et al. [135] offer a strategy for parameterised model checking of probabilistic LTLs. Furthermore, Brambilla et al. [37] provide a property-driven design method for probabilistic swarms that is checked using Prism. Last, Amin et al. [7] verify probabilistic behaviour expressed via PCTL properties using UPPAAL. They assert deadlock freedom, safety liveness checks, and reachability computations.

The majority of studies employing formal methods in analysing RAS used model checking [16, 33, 64, 103, 107, 158, 158]; after model checking, the most frequently used technique is model-based testing [132, 190, 191]. Arcile et al. [16] and Kamali et al. [107] investigate car platooning manoeuvres. In the former, the vehicles are modelled as timed automata, and UPPAAL is used as a model-checking tool. In the latter, joining and exiting operations are modelled in Belief Desire Intention models and model-checked using AJPF. They focus on abstracting a formal (untimed) model from an agent (timed) model and checking the correspondence between the agent model and the code. With respect to probabilistic model-checking, Muhammad et al. [158] model robots that synchronise to position themselves in an attempt to guarantee coverage of a certain area. The models are Markov Decision Processes that are checked using PRISM. Humprey et al. [103] make use of the model checker Spin to investigate cooperation between UAVs and sensors and collaboration among sensors as well.

The only study in this category [67] provides a simulation environment that integrates existing solutions for simulation of multi-UAV applications such as Gazebo (for robotic simulation), ArduPilopt (for UAV control algorithms), and NS3 (for network simulation). Their case study is a model of a leader-follower application for large convoys of UAVs.

Amin et al. [7] use very generic notions of effectiveness for the system under test, namely, deadlock freedom, safety, liveness, and reachability. Brambilla et al. [37] go beyond that and in addition to some domain-agnostic properties, such as probability of satisfying the requirements (safety), they measure aggregation time of the swarm and improvement of the behaviour (in terms of objects retrieved).

Muhammad et al. [158] measure the probability of task completion and human interaction to measure efficiency in wireless sensor networks. Arcile et al. [16] measure the number of collisions in its vehicle verification approach.

Lomuscio et al. [135] count the number of sates and transitions as a measure of efficiency of their formal verification methodology.

Muhammad et al. [158] also measure the time of task completion and human interaction as a measure of efficiency for the system under test. Arcile et al. [16] measure the travel time for the system under test and the verification time as efficiency metrics for the respective testing technique. D’Urso et al. [67] measure simulation time as a measure of efficiency of their integrated simulator.

Saglietti, Winzinger, and Lill [191] consider state coverage in the analysis of their model-based reconfiguration testing strategy, while Bicevkis et al. [33] conduct testing of collaborative UAV and UGV and consider “the complete test set” as a measure test coverage. Lill and Saglietti [132] model Petri nets entities and address the maximisation of interaction coverage while minimising the amount of test cases.

With respect to swarms, two open-source tools have been reported: the PSV-CA tool [135] can model-check probabilistic PLTL properties for swarm systems, and ARGOS [37] is a multi-physics robot simulator that can simulate large-scale swarms and can be customised via plug-ins.

Only two open-source tools have been found for cooperative robots: VerifCar [16] allows for fault injection in models for UPPAAL model checking. GzUAV [67], meanwhile, is a simulation tool for connected UAVs.

The only public, non-open source tool that as been employed in the testing of swarms is the RoboTool by Cavalcanti et al. [48]; RoboTool supports modelling and model checking (through the FDR tool [85]). The tool has been applied to a UAV swarm case study.

Kouvaros and Lomuscio [119, 120] study parameterised verification of robot swarms against temporal-epistemic specifications and model a small, theoretical, robot swarm. Cavalcanti et al. [47] introduce Robochart, which allows for modelling and verification of interacting robots. Cybulski [56] provides mathematical models of a UAV swarm that can be simulated in their proposed framework. Amin et al. [7] present a formal verification approach using timed automata for the verification of path planning of robot swarms.

Poncela and Aguayo-Torres [174] conduct a case study where they test underwater robots’ wireless communication. Lill et al. [132] make use of Petri nets to develop models of cooperative forklifts and simulate scenarios where the robots decide which one has the priority when passing through narrow pathways. Humprey et al. [103] make use of the model checker Spin to investigate cooperation between UAVs and sensors and collaboration among sensors as well. Saglietti, Winzinger, and Lill [191] use coloured Petri nets to model interacting autonomous agents and generate test cases for reconfiguration scenarios.

The only reported benchmark for this category is by Brambilla et al. [37], where they investigate aggregation and foraging manoeuvres on large-scale swarms of multiple sizes.

D’Usrso et al. [67] evaluate their methodology on a number of test programs using different UAV sizes. They aim their evaluation at testing (i) the scalability of the solution and (ii) its performances by comparing the simulation time with respect to physical execution time.

The only reported industrial case study is by Rooker et al. [185]. They make use of a simulation tool in the smart farming domain. Land and air robots are modelled using real dynamics and cooperate to complete farming tasks.

Heitmeyer et al. [98] synthesise state-based formal models (Software Cost Reduction tabular models) from scenarios specified in Mode diagrams (extensions of Message Sequence Charts). The models are used in a simulator integrated with the eBotworks simulation tool. Cybulski et al. [56] developed a simulation tool for UAV based on class- and activities diagrams. Further, their framework allows for user-defined maps of the environment.

Two search-based testing approaches are employed in this category: Lill and Saglietti [132] employ genetic algorithm to maximise coverage in the Petri nets models for generating test cases. Analogously, Nguyen et al. [163] provide a multi-step process to verify correctness of autonomous agents. They make use of multi-objective evolutionary algorithms to cover stakeholder soft goals. Araiza et al. [14] and Andrews et al. [10] focus on human robot interaction. The former generate test cases from BDI models, while the latter focus on coverage to generate test cases from Petri nets. Another model-based testing tool that uses Petri nets is Reference [191], which uses Coloured Petri nets and structural coverage metrics to generate test cases for reconfiguration scenarios. Finally, Hagerman et al. [93] combine a behavioural model with attack and mitigation analyses to generate a security test suite for UAVs.

The vast majority of papers in this category perform formal verification based on properties specified in variations of temporal logic; since autonomous systems specifications typically involve aspects such as beliefs and intentions, several studies are dedicated to studying the theoretical boundaries (e.g., (un)decidability) of verifying epistemic extensions of temporal logics [119, 120, 121, 122, 134, 136]; a notable tool used in this context is the MCMAS model checker [136], which is also evaluated on a small-scale benchmark against the general-purpose model checker NuSMV.

Many theoretical studies study the issue of abstraction for parameterised specifications, where parameters can be the number of autonomous agents [119, 120, 121, 122, 134, 136] or the size and shape of the arena [8, 9]. Aminof et al. [9] investigate the decidability problem for parameterised grid sizes. They found that restricting the grid size results in the problem being solved in Pspace. In the same vein, Rubin et al. [8] establish a framework in which to model and automatically verify autonomous agents. The framework contains an algorithm tailored to solve a parameterised verification problem where they use the model graphs as parameter.

Coming up with temporal logic specifications is known to be difficult and requires some level of formal training. To alleviate this, a few papers focus on writing LTL properties. Webster et al. [221, 222] model scenarios for a robot in the healthcare sector; they use Brahms as the language to describe human-robot interaction scenarios, and the properties are written in LTL. Babiceanu et al. [17] combine LTL and Event-B to build models of trustworthiness for small unmanned aerial systems (sUAS).

Formalisation of and applying formal verification on different cognitive architectures have been the focus of many studies.

Bhattacharyya [31] formalise a rule-based representation of cognitive architecture using Soar framework [124] UPPAAL and connect the verification agent to the simulation environment. They model an auto-pilot avionic system and analyse contingency situations during takeoff.

The Belief-Desire Intention framework is another natural cognitive architecture for specifying autonomous agents, and it has been extensively used in the literature. Several studies make use of the MCAPL framework [61]. Dennis et al. [60, 62] verify ethical aspects of autonomous agents’ interactions with people by modelling their behaviour using the BDI models and capturing ethical priorities; these ethical models are subsequently model-checked against LTL specifications using the MCAPL framework. Furthermore, Ferrandes et al. [71] model autonomous vehicle components and also use MCAPL to formally verify their BDI models. Last, Ferrando et al. [72] go further and provide an approach that combines formal verification and runtime monitoring by specifying trace behaviour in Prolog and connecting that with a JAVA implementation (using the JPL framework³) for runtime monitoring.

Sun et al. [205] study the effect of neural network components in the behaviour of autonomous systems; in particular, Rectified Nonlineary in Neural Networks, and analyse it using Satisfiability Modulo Convex (SMC). To mitigate the verification effort, they perform a pre-processing and evaluate the effect of pre-processing on the verification time, measured in the number of neurons in the Neural Network.

With respect to the verification of safety on human-robot interaction, Vicentini et al. [216] provide safety assessments by formally verifying models written in TRIO temporal logic [84]. Their strategy aims to identify hazardous situations associated with non-negligible risks. Analogously, Farulla and Lampretc [69] focus on model checking security properties formulated in Computational Tree Logic (CTL).

Selvaraj et al. [197] evaluate the application of different formal techniques to verify control software for an autonomous vehicle. They investigate the application of Supervisory Control Theory, Model Checking, and Deductive Verification and provide insights on how these different approaches can address different industrial challenges.

Most interventions in this category focus strictly on introducing a simulation tool for a specific sub-domain, such as underwater robots [52, 148, 212], vehicles [54, 200, 232], robots [100, 126], and UAVs [139, 161].

However, a number of interventions combine a simulation approach with other testing aspects. Li et al. [131] employ a game-theoretical approach where vehicles have different levels of knowledge about other vehicles. For instance, a level 0 car has no knowledge about the other cars, and a level k car has information about level k - 1 cars. Strangely, they show that, in some instances, lower-level cars effect less constraint violations.

Szalay et al. [206] provide a scenario-in-the-loop simulation using SUMO and Unity engine. They simulate simplified platooning and valet parking scenarios in both the simulation and in a real smart city (Salazone).

Verma et al. [215] present a Flight Software simulator that is used to simulate MARS Rover missions. The simulator assists in predicting the behaviour of semi-autonomous systems by providing the capability for human operators to check if their intent is correctly captured by the robot prior to execution in different scenarios and environments.

Two studies present supporting libraries: Koolen et al. [117] implement robotic simulation library in the Julia programming language. The library offers support for robot dynamics, visualisation, and control algorithms. Rohmer et al. [184] developed libraries to integrate VREP and other programming languages (Lua, C++, Java, Python, Matlab and Ruby) with support for different types of 3D objects and modules for kinematics and dynamics.

Multi-objective search is an increasingly popular technique for coping with complex robotics systems. Betts et al. [30] employ the Monte Carlo search heuristic to verify the lateral distance between the outcome of surrogate-based models compared to a known ground truth in UAV applications. A similar approach is used by Nejati et al. [25] on pedestrian detection using vision-based system. They employ NSGA-II [59] using minimum distance and minimum time to collision as fitness functions and compare the performance of the heuristic with and without surrogates.

Sagliatelli and Meitner [190, 191], however, propose multiple notions of coverage to help generate test cases from Petri nets models of autonomous, cooperative, and reconfigurable robots. Furthermore, they employ statistical testing techniques, which intend to evaluate the degree of acceptance of the behaviour observed.

A framework for automated testing using metamorphic testing principles combined with model-based testing is employed by Lindvall et al. [133]. Test cases are generated from test models and multiple variations of scenarios that are programmatically generated based on metamorphic relations.

In this category, verification checks at runtime have been typically coupled with model checking strategies, where properties are being checked during the system execution. For instance, Desai et al. [63] present an STL-based framework where assumptions used during model checking hold at runtime, where the online monitor checks robustness on partial trajectories from low-level controllers.

In the context of obstacle avoidance, Luo et al. [140] employ JavaMOP to verify that a robot does not behave against requirements written in FSM and PTLT languages. Temporal properties are also employed by Wang et al. [220], where the RoboticSpec specification language for robotic applications is translated into a framework for online monitoring that also uses PLTL properties.

Huang et al. [102] present ROSRV, which is an online monitoring framework that runs on top of ROS. They make use of a public-subscribe communication architecture and intercept commands and messages passing through the communications channel. This way, they are able to verify safety and security requirements at runtime using a domain-specific language.

Manhaes and Rauschenbach present UUV simulator [148], which is an extension of Gazebo accommodating the domain-specific aspects of underwater vehicles. They assist with modelling of underwater hydrostatic and hydrodynamic effects, thrusters, sensors, and external disturbances and demonstrate their tool on a case using a modified model of the Sperre SF 30k ROV robot (RexROV).

As another tool for underwater robots, MARS [212] provides simulation environments for marine swarm robots that allows for hardware-in-the-loop simulation. The tool has a Java interface and has been applied to the MONSUN and HANSE autonomous underwater robots.

In the Matlab environment, the FROST tool [100] is an open-source Matlab toolkit for modeling, trajectory optimisation, and simulation of robots, with a particular focus in dynamic locomotion. In the study, they model the ATLAS and DRC-HUBO as examples.

Munawar and Fischer [160] present the Asynchronous Framework, which incorporates real-time dynamic simulation and interfaces with learning agents to train and potentially allow for the execution of shared sub-tasks. Due to the asynchronous nature of the communication, they measure the number of packets against latency. Furthermore, they focus on surgical robots as part of their application domain, and they employ the CHAI3D haptics framework. They connect their tools with ROS, which allows them to connect to learning libraries such as TensorFlow.

D’Urso, Santoro, and Santoro [67] also present a simulator for multi-UAV applications, called GzUAVChannel. It works as a middleware that combines Gazebo, Autopilot, and NS-3 network simulator to provide a 3D visualisation engine, a physics simulator, a flight control stack, and a network simulator to handle communications among unmanned aerial vehicles. They model a leader-follower example.

The MoVE tool [54] provides the possibility of modelling pedestrian behaviour. The framework focuses on testing autonomous system algorithms, vehicles, and their interactions with real and simulated vehicles and pedestrians. They conduct three case studies: traffic wave observation, medical evacuation, and virtual vehicles avoiding real pedestrians.

Rohmer, Singh, and Freese introduce VREP [184] a popular robotics physics simulator that is now known as CoppeliaSIM. The tool uses a kinematics engine and several physics libraries to provide rigid body simulations (including meshes, joints, and multiple types of sensors).

Koolen et al. [117] implement robotic simulation library in the Julia programming language. The library offers support for robot dynamics, visualisation, and control algorithms.

Brambilla et al. have developed ARGOS [37], which is a multi-physics robot simulator that can simulate large-scale swarms and can be customised via plug-ins.

Cieslak et al. introduce Stonefish, a geometry-based simulator [52] that can be integrated with ROS. Last, the MARS [212] tool provides simulation environments for marine swarm robots.

Gambi, Mueller, and Fraser present the AsFault prototype tool [78]. The tool combines procedural content generation and search-based testing to automatically create challenging virtual scenarios for testing self-driving car software.

Garzón and Spalanzani [80] present a tool that combines 3D simulation (for ego-vehicle control) with a traffic simulator (which controls the behaviour of other vehicles). The goal is to test the ego-vehicle in realistic high-traffic situations.

Lugo-Cárdenas, Luzano, and Flores [139] introduce a 3D simulation tool for UAVs whose focus is on assisting the development of flight controllers.

Parametric modelling of CAVs as a network of timed automata is used by Arcile et al. [16]. In this work, VerifCar tool is applied to assess the impact of communication delays on the decision algorithms of CAVs and to check the robustness and efficiency of such algorithms.

Gruber and Althoff [91] present a reachability analysis tool (Spot) that finds counter-example to property violations. It starts with a coarse model of the system dynamics but it can refine the abstraction levels for precision/scaling.

Desai et al. [63] present a runtime verification framework (DRONA) based on Signal Temporal Logic [144], where an online monitor checks robustness on partial trajectories from low-level controllers.

RoVer [175] provides visual authoring of HRI, formalisation of properties in temporal logic and verification (via model-checking with PRISM [123]) that the interactions abide by a set of social norms and task expectations whose goal is to identify social norms violation.

Althoff introduces IMPROV [6], a tool that is used to formally verify human-robot interaction for modular robots.

Gainer et al. [77] provide a tool (CRutoN) for translation to formal models from control rules of robots in a DSL and input those models into NuSMV for model checking. Their main emphasis is the verification of human-robot interaction.

Bao et al. [18] present a prototype tool for parametric statistical model checking that can cope with complex parametric Markov chains where state-of-the-art tools (such as PRISM) have timed out. They provide evidence of their tool efficiency by conducting an industrial case study.

The FLPolyVF tool [4] connects functional verification, sensor verification, diagnostics, and industry/regulatory communication of autonomous vehicles while checking the effects of using different (matrix-based) scenario abstraction levels.

Lomuscio et al. have developed the MCMAS model checker [136]. They used a logic (ATL-k) alternating temporal logic (epistemic), one of the few tools that uses CTL. They demonstrate their strategy in a couple of small-scale examples, where they compare their strategy against alternatives (NuSMV and MCTK).

Huang et al. present ROSRV [102], which is a runtime verification framework that can be used with ROS.

Desai et al. [63] present a runtime verification framework based on Signal Temporal Logic, where an online monitor checks robustness on partial trajectories from low-level controllers (in the context of surgical robots).

Cavalcanti et al. [46, 47, 48, 156] introduce RoboTool, supporting graphical modelling, validation, and model checking (via FDR [85]) of robotic models written in RoboChart [156] and RoboSim [48].

Shah et al. [198] introduce the AirSim simulator that generates training data for building machine learning models used in autonomous aircraft. It offers physical and visual simulation, including models of physics engine, vehicle, environment, and sensors. Further, it connects to an API for planning and control.

Zhang et al. [233] introduce CyberEarth, a framework for program-driven simulation, visualisation and monitoring of robots. The tool integrates modules from several other open-source tool such as ROS [177] and OpenSceneGraph (OSG) [43].

Mullins et al. [159] developed a tool (RAPT - Range Adversarial Planning Tool) for generating test scenarios to be employed on the System Under Test. The tool employs an adaptive search method that generates challenging scenarios based on the performance and results of the previous ones. A clustering algorithm ranks the scenarios based on the performance type and how close they are to the boundaries of each cluster. The boundaries are based on notions of efficiency (precision and convergence), diversity (how many performance boundaries are being covered), and scaling.

The only tool in this category is RoboTool [48], which has also been described above in the Simulation category. It provides formal verification via translated CSP models fed into the FDR model checker [85].

Heitmeyer and Leonard [98] introduce two tools integrated into the FORMAL framework; the tools synthesise and validate formal models. The first tool synthesises a formal Software Cost Reduction (SCR) requirements model from scenarios, and the second tool combines the existing SCR simulator [96] with eBotworks 3D simulator to allow for simulation of continuous components. They focus on the verification of human-machine interaction.

Verma et al. [215] present a Flight Software simulator (SSIM - part of Rover Sequencing and Visualisation Program (RSVP) suite) that is used to simulate MARS Rover missions. The simulator assists in predicting the behaviour of semi-autonomous systems by providing the capability for human operators to check if their intent is correctly captured by the robot prior to execution in different scenarios and environments.

Zhang et al. present RoadView [232], a photo-realistic simulator that tests performance of autonomous vehicles and evaluates their self-driving tasks. They make use of driving scenarios where they compare autonomous vehicle to a human-driven scenario to demonstrate their tool.

Schöner presents a simulation tool that is part of the (industrial) Pegasus framework [195]. It integrates sensors, traffic, and road models (in open-drive Format) into the simulation where different scenarios and situations are executed.

Collet et al. [53] introduce RobTest, a tool for generating collision-free trajectories for stress-testing of single-arm robots. It employs constraint programming techniques to solve continuous domain constraints in its trajectory generation process. The efficiency of such a process is evaluated in a controlled experiment where the generation time of acceptable near-optimal trajectories.

O’Kelly introduces APEX [165], which is a formal verification tool for verifying vehicle dynamics, trajectory planning, and tracking stacks of ADAS in vehicles. Property specifications are written in metric interval temporal logic. The tool calls DReach [116] in the background to perform reachability analysis on the vehicle trajectories.

Foughali et al. [75] implement an automatic translation from GenoM [145], a robotics model-based software engineering framework, to the formal specification language Fiacre [28], which can be fed into TINA for model checking (on the Petri net models). They apply their tool to an autonomous ground vehicle (RMP 400 Segway).

Regarding small-scale case studies for simulation environments, the vast majority conduct a simple demonstration to illustrate features of their simulation tools [11, 42, 48, 50, 56, 95, 139, 232, 233], such as Mars [212] (for underwater robots) and VREP [184] (for generic robots) case studies.

Differently, Li et al. [131] employ a game-theoretical approach where vehicles have different levels of knowledge about other vehicles. For instance, a level 0 car has no knowledge about the other cars and a level k car has information about level k - 1 cars. Strangely, in their case study, they show that, in some instances, lower-level cars effect less constraint violations.

Two of the case studies in this category consider generating test cases from formal models [132, 163] of autonomous agents. Furthermore, Andrews et al. [10] model autonomous systems and their environment using Petri nets to generate test cases and apply their technique to a case study in the human-robot interaction domain. In Hagerman’s case study [93], finite state machines are used to extract security test suites. Sagglietti et al. [190, 191] conduct a case study in which the reconfiguration behaviour of autonomous agents is verified. Betts et al. [30] compare the effectiveness of two search-based testing methods, with a case study involving a UAV flight control software.

Several of the included case studies in this category concern abstract representations of multi-agent autonomous systems and provided efficient algorithms for parametric (formal) verification or state-space reduction techniques [18, 22, 24, 119, 120, 122, 135, 136].

Several other case studies [19, 46, 47, 48, 137, 204] concern systems used in model checking tools such as Prism [123] and FDR [85]. Another use of these case studies is to demonstrate usage of introduced tools such as APEX [165] and MDE [147]. Differently, Dennis et al. [60, 62] focus on formalising and verifying ethical concerns in BDI agents and provide corresponding small case studies. Aminof et al. [9] investigate the decidability problem for parameterised grid sizes. In their case study, they found that restricting the grid size results in the problem being solved in Pspace.

In the only paper in this category, Desai, Tomasso, and Seshia [63] make use of an STL-based (signal temporal logic) online monitoring system to ensure that the assumptions about the low-level controllers (discrete models) used during model checking hold at runtime. They demonstrate the strategy in a surveillance application case study.

Zhou et al. [237] introduce a framework for virtual testing of advanced driver assistant systems that uses real-world measurements. Shah et al. [198] build a model of quadrotor with pixhawk controller in their newly developed simulator, AirSim, that includes a physics engine and and supports real-time hardware-in-the-loop. Schöner presents a simulation tool that is part of the (industrial) Pegasus framework [195]. It integrates sensors, traffic, and road models (in open-drive Format) into the simulation where different scenarios and situations are executed. Reference [185] demonstrates their validation framework for autonomous systems in a farming context with simulations and field testing.

Uriagereka et al. [214] conduct simulation-assisted fault injection to assess safety and reliability of robotic systems. The feasibility of their method is demonstrated by applying it to the design of a real-time cartesian impedance control system. Manhaes and Rauschenbach [148] model the Sperre SF 30k ROV underwater robot (RexROV) in the demonstration of the simulator for unmanned underwater vehicles. Verma et al. [215] present a Flight Software simulator that is used to simulate MARS Rover missions. They demonstrate their approach with a corresponding case study. AbdElSalam et al. [1] use Hardware Emulation-in-the-loop to verify Electronic Control Units (ECUs) for ADAS systems.

In the only industrial case study in this category, Abdessalem et al. [25] generate test cases for a system that can visually detect pedestrians in the context of advanced driver assistance systems (ADAS).

Gainer et al. [77] conduct a case study in the context of verification of human-robot interaction using the Care-O-Bot robotic assistant. Bhattacharyya et al. [31] apply formal verification methods to an autonomous CoPilot agent.

Gladisch et al. [87] select case studies that use industrial automated driving (adaptive cruise control, lane keeping, and steering control scenarios) to evaluate their search-based testing strategy.

Several benchmark-scale case studies can be found in this category. In what follows, we briefly discuss some of them. Wigand et al. [224] have developed CoSiMA, which is an architecture for simulation, execution, and analysis of robotics system. They conduct experiments on the humanoid robot COMAN. Tosum et al. [213] present a design framework that facilitates the rapid creation of configurations and behaviours for modular robots. They have demonstrated their framework on the SMORES robot. Pereira et al. [172] employ several small case studies in their attempt to couple two simulators, namely, SUMO and USAR-Sim. Brambilla et al. [37] model a probabilistic swarm that is checked in PRISM to evaluate their property-driven design method. Bohlmann, Klinger, and Szczerbicka [36] automatically generate a model of a parallel delta robot on-the-fly. Their method for model generation is based on machine learning and symbiotic simulation techniques. Mutter et al. [161] also explore the simulation of UAV models in Simulink and discuss the results when combining the platform and environment models. Bi et al. [32] present a deep Learning-based framework for traffic simulation and execute several scenarios of intersections with and without pedestrians. D’Urso, Santoro, and Santoro [67] simulate leader-follower UAV scenarios in their framework. Their goal is to combine four simulation environments: a 3D visualisation engine, a physics simulator, a flight control stack, and a network simulator. Wang and Cheng [219] present a hardware-in-the-loop simulator for drones that can generate synthetic images from the scene as datasets, detect and verify objects with a trained neural network, and generate point cloud data for model validation. They simulate and conduct filed testing on a physical UAV. Heitmeyer et al. [98] synthesise software cost-reduction models of multiple autonomous systems to be used in a simulator integrated with the eBotworks simulation tool.

Proetzsch et al. [176] use a designed DSL (graph-based models) to describe the system behaviour of the autonomous off-road robot RAVON. The model is used as test model for generating test cases. Mullins et al. [159] have developed a tool (RAPT - Range Adversarial Planning Tool) for generating test scenarios to be employed on the System Under Test. Their tool is applied to realistic underwater missions. Furthermore, in their case study, Araiza-Illan, Pipe, and Eder [14] use BDI models and model checking of probabilistic timed automata (in UPPAAL) to generate test sequences for human-robot collaboration tasks.

Here, we briefly discuss some of the benchmarks that involve formal verification. Halder et al. [94] use the physical robot Kobuki as a case study, over which properties are automatically verified using the UPPAAL model checker. The focus of their approach is to model and verify ROS systems using real-time properties. Brambilla et al. [37] model a probabilistic swarm that is checked in PRISM to evaluate their property-driven design method. Bicevskis, Gaujens, and Kalnins [33] develop models for the testing of UAV and UGV collaboration in the Simulink environment. Althoff et al. [6] propose a framework (IMPROV) for self-programming and self-verification for robots, which is demonstrated on a physical robotic arm. Zhao et al. [234] model an unmanned aerial vehicle (UAV) inspection mission on a wind farm and, via probabilistic model checking in PRISM, show how the battery features may affect verification results. Gruber and Althoff [91] present a reachability analysis tool (Spot) that finds counter-example to property violations. Their tool is evaluated using the CommonRoad benchmark PM1:MW1:DEU_Muc-3_1_T-1.

Pasareanu, Gopinath, and Yu [170] present a compositional approach for the verification of autonomous systems and apply the technique on a neural network implementation of a controller for a collision avoidance system on the ACAS Xu unmanned aircraft. Temporal properties are employed in Wang’s case study [220], where the RoboticSpec specification language for robotic applications is translated into a framework for online monitoring that also uses PLTL properties. Huang et al. conduct a case study using a model of the LandShark UGV to demonstrate their tool, ROSRV [102], which is a runtime verification framework that can be used with ROS. In the context of obstacle avoidance, Luo et al. [140] employ JavaMOP in their case study to verify that the robot does not behave against requirements written in FSM and PTLT languages.