An Empirically Grounded Path Forward for Scenario-Based Testing of Autonomous Driving Systems
Pages 232 - 243
Abstract
Testing of autonomous driving systems (ADS) is a crucial, yet complex task that requires different approaches to ensure the safety and reliability of the system in various driving scenarios. Currently, there is a lack of understanding of the industry practices for testing such systems, and also the related challenges. To this end, we conduct a secondary analysis of our previous exploratory study, where we interviewed 13 experts from 7 ADS companies in Sweden. We explore testing practices and challenges in industry, with a special focus on scenario-based testing as it is widely used in research for testing ADS. Through a detailed analysis and synthesis of the interviews, we identify key practices and challenges of testing ADS. Our analysis shows that the industry practices are primarily concerned with various types of testing methodologies, testing principles, selection and identification of test scenarios, test analysis, and relevant standards and tools as well as some general initiatives. Challenges mainly include discrepancies in concepts and methodologies used by different companies, together with a lack of comprehensive standards, regulations, and effective tools, approaches, and techniques for optimal testing. To address these issues, we propose a `3CO' strategy (Combine, Collaborate, Continuously learn, and be Open) as a collective path forward for industry and academia to improve the testing frameworks for ADS.
References
[1]
Houssam Abbas, Matthew E. O’Kelly, Alena Rodionova, and Rahul Mangharam. 2017. A Driver’s License Test for Driverless Vehicles. Mechanical Engineering, 139, 12 (2017), 12, S13–S16. issn:0025-6501 https://doi.org/10.1115/1.2017-Dec-9
[2]
Jubril Gbolahan Adigun, Matteo Camilli, Michael Felderer, Andrea Giusti, Dominik T Matt, Anna Perini, Barbara Russo, and Angelo Susi. 2022. Collaborative Artificial Intelligence Needs Stronger Assurances Driven by Risks. Computer, 55, 3 (2022), 52–63. https://doi.org/10.1109/MC.2021.3131990
[3]
Mert Aydin and Mustafa Ilhan Akbas. 2021. Identification of Test Scenarios for Autonomous Vehicles Using Fatal Accident Data. SAE International Journal of Connected and Automated Vehicles, 4 (2021), 119–132. https://doi.org/10.4271/12-04-01-0010
[4]
Felix Beringhoff, Joel Greenyer, Christian Roesener, and Matthias Tichy. 2022. Thirty-one challenges in testing automated vehicles: Interviews with experts from industry and research. In 2022 IEEE Intelligent Vehicles Symposium (IV). IEEE, 360–366. https://doi.org/10.1109/IV51971.2022.9827097
[5]
Jinkang Cai, Weiwen Deng, Haoran Guang, Ying Wang, Jiangkun Li, and Juan Ding. 2022. A Survey on Data-Driven Scenario Generation for Automated Vehicle Testing. Machines, 10, 11 (2022), 1101. https://doi.org/10.3390/machines10111101
[6]
Daniela S Cruzes and Tore Dybå. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. 275–284. https://doi.org/10.1109/ESEM.2011.36
[7]
Iván García Daza, Rubén Izquierdo, Luis Miguel Martínez, Ola Benderius, and David Fernández Llorca. 2022. Sim-to-real transfer and reality gap modeling in model predictive control for autonomous driving. Applied Intelligence, 1–17. https://doi.org/10.1007/s10489-022-04148-1
[8]
Erwin de Gelder and Olaf Op den Camp. 2020. Procedure for the Safety Assessment of an Autonomous Vehicle Using Real-World Scenarios. https://doi.org/10.48550/arXiv.2012.00643
[9]
2022. Autonomous Vehicles Tests Without a Driver. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/testing-autonomous-vehicles-without-a-driver/ Accessed on April 12, 2023
[10]
Francesca M Favarò, Nazanin Nader, Sky O Eurich, Michelle Tripp, and Naresh Varadaraju. 2017. Examining accident reports involving autonomous vehicles in California. PLoS one, 12, 9 (2017), e0184952. https://doi.org/10.1371/journal.pone.0184952
[11]
Shlomi Hacohen, Oded Medina, and Shraga Shoval. 2022. Autonomous Driving: A Survey of Technological Gaps Using Google Scholar and Web of Science Trend Analysis. IEEE Transactions on Intelligent Transportation Systems, 23, 11 (2022), 21241–21258. https://doi.org/10.1109/TITS.2022.3172442
[12]
Florian Hauer, Tabea Schmidt, Bernd Holzmüller, and Alexander Pretschner. 2019. Did we test all scenarios for automated and autonomous driving systems? In 2019 IEEE Intelligent Transportation Systems Conference (ITSC). 2950–2955. https://doi.org/10.1109/ITSC.2019.8917326
[13]
2022. Road vehicles — Safety of the intended functionality. International Organization for Standardization. https://www.iso.org/obp/ui/##iso:std:77490:en
[14]
2021. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. SAE International. https://www.sae.org/standards/content/j3016_202104/
[15]
Pengliang Ji, Ruan Li, Yunzhi Xue, Qian Dong, Limin Xiao, and Rui Xue. 2021. Perspective, survey and trends: Public driving datasets and toolsets for autonomous driving virtual test. In IEEE International Intelligent Transportation Systems Conference (ITSC). 264–269. https://doi.org/10.1109/ITSC48978.2021.9564428
[16]
Yue Kang, Hang Yin, and Christian Berger. 2019. Test your self-driving algorithm: An overview of publicly available driving datasets and virtual testing environments. IEEE Transactions on Intelligent Vehicles, 4, 2 (2019), 171–185. https://doi.org/10.1109/TIV.2018.2886678
[17]
Dhanoop Karunakaran, Julie Stephany Berrio, Stewart Worrall, and Eduardo Nebot. 2022. Parameterisation of lane-change scenarios from real-world data. In IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). 2607–2613. https://doi.org/10.1109/ITSC55140.2022.9921981
[18]
Michael Kläs, Lisa Jöckel, Rasmus Adler, and Jan Reich. 2022. Integrating Testing and Operation-related Quantitative Evidences in Assurance Cases to Argue Safety of Data-Driven AI/ML Components. https://doi.org/10.48550/arXiv.2202.05313
[19]
Alessia Knauss, Jan Schröder, Christian Berger, and Henrik Eriksson. 2017. Paving the roadway for safety of automated vehicles: An empirical study on testing challenges. In 2017 IEEE Intelligent Vehicles Symposium (IV). 1873–1880. https://doi.org/10.1109/IVS.2017.7995978
[20]
Nicola Kolb, Florian Hauer, and Alexander Pretschner. 2021. Fitness Function Templates for Testing Automated and Autonomous Driving Systems in Intersection Scenarios. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). 217–222. https://doi.org/10.1109/ITSC48978.2021.9564591
[21]
Kristofer D Kusano, Kurt Beatty, Scott Schnelle, Francesca Favaro, Cam Crary, and Trent Victor. 2022. Collision Avoidance Testing of the Waymo Automated Driving System. https://doi.org/10.48550/arXiv.2212.08148
[22]
Guannan Lou, Yao Deng, Xi Zheng, Mengshi Zhang, and Tianyi Zhang. 2022. Testing of Autonomous Driving Systems: Where Are We and Where Should We Go? ESEC/FSE 2022. ACM, New York, NY, USA. 31–43. isbn:9781450394130 https://doi.org/10.1145/3540250.3549111
[23]
Jing Ma, Xiaobo Che, Yanqiang Li, and Edmund M-K Lai. 2021. Traffic scenarios for automated vehicle testing: A review of description languages and systems. Machines, 9, 12 (2021), 342. https://doi.org/10.3390/machines9120342
[24]
Till Menzel, Gerrit Bagschik, and Markus Maurer. 2018. Scenarios for development, test and validation of automated vehicles. In IEEE Intelligent Vehicles Symposium (IV). 1821–1827. https://doi.org/10.1109/IVS.2018.8500406
[25]
Christian Neurohr, Lukas Westhofen, Tabea Henning, Thies de Graaff, Eike Möhlmann, and Eckard Böde. 2020. Fundamental considerations around scenario-based testing for automated driving. In IEEE intelligent vehicles symposium (IV). 121–127. https://doi.org/10.1109/IV47402.2020.9304823
[26]
2022. Commission Implementing Regulation (EU) 2022/1426. European Parliament and the Council. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32022R1426
[27]
Safety Pool TM. 2021. Safety Pool. https://www.safetypool.ai/ Accessed on April 12, 2023
[28]
Felix Reisgys, Johannes Plaum, Andreas Schwarzhaupt, and Eric Sax. 2022. Scenario-based X-in-the-Loop Test for Development of Driving Automation. 14. Uni-DAS e.V. Workshop Fahrerassistenz und automatisiertes Fahren.
[29]
Stefan Riedmaier, Thomas Ponn, Dieter Ludwig, Bernhard Schick, and Frank Diermeyer. 2020. Survey on scenario-based safety assessment of automated vehicles. IEEE Access, 8 (2020), 87456–87477. https://doi.org/10.1109/ACCESS.2020.2993730
[30]
Francisca Rosique, Pedro J Navarro, Carlos Fernández, and Antonio Padilla. 2019. A systematic review of perception system and simulators for autonomous vehicles research. Sensors, 19, 3 (2019), 648. https://doi.org/10.3390/s19030648
[31]
Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering, 14 (2009), 131–164. https://doi.org/10.1007/s10664-008-9102-8
[32]
2022. Scenario-based verification and validation of self-driving vehicles: relevant safety metrics. Siemens Digital Industries Software. https://resources.sw.siemens.com/en-US/white-paper-scenario-based-verification-and-validation-of-autonomous-driving-systems
[33]
Qunying Song, Emelie Engström, and Per Runeson. 2021. Concepts in testing of autonomous systems: Academic literature and industry practice. In IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN). 74–81. https://doi.org/10.1109/WAIN52551.2021.00018
[34]
Qunying Song, Emelie Engström, and Per Runeson. 2024. Industry Practices for Challenging Autonomous Driving Systems with Critical Scenarios. ACM Trans. Softw. Eng. Methodol., 33, 4 (2024), 35 pages. issn:1049-331X https://doi.org/10.1145/3640334
[35]
Qunying Song, Emelie Engström, and Per Runeson. 2023. Supplementary materials. https://doi.org/10.5281/zenodo.10436801
[36]
Qunying Song, Emelie Engström, and Per Runeson. 2024. Supplementary materials. https://doi.org/10.5281/zenodo.11091933
[37]
Qunying Song and Per Runeson. 2023. Industry-academia collaboration for realism in software engineering research: Insights and recommendations. Information and Software Technology, 156 (2023), 107135. https://doi.org/10.1016/j.infsof.2022.107135
[38]
Qunying Song, Kaige Tan, Per Runeson, and Stefan Persson. 2021. An Industrial Workbench for Test Scenario Identification for Autonomous Driving Software. In 2021 IEEE International Conference on Artificial Intelligence Testing (AITest). 81–82. https://doi.org/10.1109/AITEST52744.2021.00024
[39]
Qunying Song, Kaige Tan, Per Runeson, and Stefan Persson. 2022. Critical scenario identification for realistic testing of autonomous driving systems. Software Quality Journal, 1–29. https://doi.org/10.1007/s11219-022-09604-2
[40]
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2022. Mind the gap! a study on the transferability of virtual vs physical-world testing of autonomous driving systems. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2022.3202311
[41]
Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, and Yinxing Xue. 2022. A Survey on Automated Driving System Testing: Landscapes and Trends. https://doi.org/10.48550/arXiv.2206.05961
[42]
Alexander Tenbrock, Alexander König, Thomas Keutgens, and Hendrik Weber. 2021. The ConScenD Dataset: Concrete Scenarios from the highD Dataset According to ALKS Regulation UNECE R157 in OpenX. In IEEE Intelligent Vehicles Symposium Workshops (IV Workshops). 174–181. https://doi.org/10.1109/IVWorkshops54471.2021.9669219
[43]
Haoxiang Tian, Yan Jiang, Guoquan Wu, Jiren Yan, Jun Wei, Wei Chen, Shuo Li, and Dan Ye. 2022. MOSAT: finding safety violations of autonomous driving systems using multi-objective genetic algorithm. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 94–106. https://doi.org/10.1145/3540250.3549100
[44]
Simon Ulbrich, Till Menzel, Andreas Reschka, Fabian Schuldt, and Markus Maurer. 2015. Defining and substantiating the terms scene, situation, and scenario for automated driving. In IEEE 18th international conference on intelligent transportation systems. 982–988. https://doi.org/10.1109/ITSC.2015.164
[45]
Xinhai Zhang, Jianbo Tao, Kaige Tan, Martin Törngren, José Manuel Gaspar Sánchez, Muhammad Rusyadi Ramli, Xin Tao, Magnus Gyllenhammar, Franz Wotawa, Naveen Mohan, Mihai Nica, and Hermann Felbinger. 2023. Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study. IEEE Transactions on Software Engineering, 49, 3 (2023), 991–1026. https://doi.org/10.1109/TSE.2022.3170122
[46]
Ziyuan Zhong, Yun Tang, Yuan Zhou, Vania de Oliveira Neves, Yang Liu, and Baishakhi Ray. 2021. A survey on scenario-based testing for automated driving systems in high-fidelity simulation. https://doi.org/10.48550/arXiv.2112.00964
Index Terms
- An Empirically Grounded Path Forward for Scenario-Based Testing of Autonomous Driving Systems
Recommendations
Industry Practices for Challenging Autonomous Driving Systems with Critical Scenarios
Testing autonomous driving systems for safety and reliability is essential, yet complex. A primary challenge is identifying relevant test scenarios, especially the critical ones that may expose hazards or harm to autonomous vehicles and other road users. ...
A Scenario Distribution Model for Effective and Efficient Testing of Autonomous Driving Systems
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software EngineeringWhile autonomous driving systems are expected to change future means of mobility and reduce road accidents, understanding intensive and complex traffic situations is essential to enable testing of such systems under realistic traffic conditions. ...
ISS-Scenario: Scenario-Based Testing in CARLA
Theoretical Aspects of Software EngineeringAbstractThe rapidly evolving field of autonomous driving systems (ADSs) is full of promise. However, in order to fulfil these promises, ADSs need to be safe in all circumstances. This paper introduces ISS-Scenario, an autonomous driving testing framework ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Knut och Alice Wallenbergs Stiftelse
Conference
FSE '24
Sponsor:
FSE '24: 32nd ACM International Conference on the Foundations of Software Engineering
July 15 - 19, 2024
Porto de Galinhas, Brazil
Acceptance Rates
Overall Acceptance Rate 112 of 543 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 160Total Downloads
- Downloads (Last 12 months)160
- Downloads (Last 6 weeks)54
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in