short-paper

ADEPT: A Testing Platform for Simulated Autonomous Driving

Authors:

Xiaoxing MaAuthors Info & Claims

ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

Article No.: 150, Pages 1 - 4

https://doi.org/10.1145/3551349.3559528

Published: 05 January 2023 Publication History

Abstract

Effective quality assurance methods for autonomous driving systems ADS have attracted growing interests recently. In this paper, we report a new testing platform ADEPT, aiming to provide practically realistic and comprehensive testing facilities for DNN-based ADS. ADEPT is based on the virtual simulator CARLA and provides numerous testing facilities such as scene construction, ADS importation, test execution and recording, etc. In particular, ADEPT features two distinguished test scenario generation strategies designed for autonomous driving. First, we make use of real-life accident reports from which we leverage natural language processing to fabricate abundant driving scenarios. Second, we synthesize physically-robust adversarial attacks by taking the feedback of ADS into consideration and thus are able to generate closed-loop test scenarios. The experiments confirm the efficacy of the platform.

References

[1]

Stephanie Abrecht, Lydia Gauerhof, Christoph Gladisch, Konrad Groh, Christian Heinzemann, and Matthias Woehrle. 2021. Testing deep learning-based visual perception for automated driving. TCPS 5, 4 (2021), 1–28.

[2]

Matthias Althoff and Sebastian Lutz. 2018. Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles. In 2018 IEEE Intelligent Vehicles Symposium. 1326–1333.

Digital Library

[3]

Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. 2018. Synthesizing Robust Adversarial Examples. In ICML.

[4]

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316(2016).

[5]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.

[6]

Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy. 39–57.

[7]

Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Polo Chau. 2018. Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector. In ECML PKDD. 52–68.

[8]

R. Craig Coulter. 1992. Implementation of the Pure Pursuit Path Tracking Algorithm.

[9]

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In CoRL. 1–16.

[10]

Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In CVPR. 1625–1634.

[11]

Daniel J Fremont, Edward Kim, Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Alberto L Sangiovanni-Vincentelli, and Sanjit A Seshia. 2020. Scenic: A language for scenario specification and data generation. (2020).

[12]

Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019. Generating effective test cases for self-driving cars from police reports. In ESEC/FSE 2019. 257–267.

Digital Library

[13]

Christoph Gladisch, Thomas Heinz, Christian Heinzemann, Jens Oehlerking, Anne von Vietinghoff, and Tim Pfitzer. 2019. Search-based testing in automated driving control applications. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 26–37.

Digital Library

[14]

Quentin Goss, Yara AlRashidi, and Mustafa İlhan Akbaş. 2021. Generation of modular and measurable validation scenarios for autonomous vehicles using accident data. In 2021 IEEE Intelligent Vehicles Symposium. 251–257.

Digital Library

[15]

James C Holland and Arman Sargolzaei. 2020. Verification of autonomous vehicles: Scenario generation based on real world accidents. In 2020 SoutheastCon, Vol. 2. 1–7.

[16]

Moritz Klischat and Matthias Althoff. 2019. Generating critical test scenarios for automated vehicles with evolutionary algorithms. In 2019 IEEE Intelligent Vehicles Symposium. 2352–2358.

Digital Library

[17]

Zelun Kong, Junfeng Guo, Ang Li, and Cong Liu. 2020. Physgan: Generating physical-world-resilient adversarial examples for autonomous driving. In CVPR. 14254–14263.

[18]

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083(2017).

[19]

Rupak Majumdar, Aman Mathur, Marcus Pirron, Laura Stegner, and Damien Zufferey. 2019. Paracosm: A language and tool for testing autonomous driving systems. (2019).

[20]

Vuong Nguyen, Alessio Gambi, Jasim Ahmed, and Gordon Fraser. 2022. CRISCE: Towards Generating Test Cases from Accident Sketches. In 2022 IEEE/ACM 44th ICSE-Companion. 339–340.

Digital Library

[21]

Vuong Nguyen, Stefan Huber, and Alessio Gambi. 2021. SALVO: Automated Generation of Diversified Tests for Self-driving Cars from Existing Maps. In 2021 IEEE International Conference on Artificial Intelligence Testing. 128–135.

[22]

NHTSA. 2016. NHTSA Crash Viewer. http://crashviewer.nhtsa.dot.gov/, Last accessed on 2022-5-24.

[23]

Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, and Vincenzo Riccio. 2021. Sbst tool competition 2021. In 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST). 20–27.

[24]

Naman Patel, Prashanth Krishnamurthy, Siddharth Garg, and Farshad Khorrami. 2019. Adaptive adversarial videos on roadside billboards: Dynamically modifying trajectories of autonomous vehicles. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 5916–5921.

Digital Library

[25]

Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2018. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics. Springer, 621–635.

[26]

Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. 2020. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR. 7153–7162.

[27]

Fitash Ul Haq, Donghwan Shin, and Lionel Briand. 2022. Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization. In ICSE 2022.

[28]

Zhang Xinxin, Li Fei, and Wu Xiangbin. 2020. CSG: Critical scenario generation from real traffic accidents. In 2020 IEEE Intelligent Vehicles Symposium. 1330–1336.

Digital Library

[29]

Xinhai Zhang, Jianbo Tao, Kaige Tan, Martin Torngren, Jose Manuel Gaspar Sanchez, Muhammad Rusyadi Ramli, Xin Tao, Magnus Gyllenhammar, Franz Wotawa, Naveen Mohan, 2022. Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study. IEEE TSE (2022).

[30]

Husheng Zhou, Wei Li, Zelun Kong, Junfeng Guo, Yuqun Zhang, Bei Yu, Lingming Zhang, and Cong Liu. 2020. Deepbillboard: Systematic physical-world testing of autonomous driving systems. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). 347–358.

Digital Library

Cited By

Biagiola MTonella P(2024)Boundary State Generation for Testing and Improvement of Autonomous Driving SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.342081650:8(2040-2053)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3420816
Schafhalter PKalra SXu LGonzalez JStoica I(2023)Leveraging Cloud Computing to Make Autonomous Vehicles Safer2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10341821(5559-5566)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10341821

Index Terms

ADEPT: A Testing Platform for Simulated Autonomous Driving

Index terms have been assigned to the content through auto-classification.

Recommendations

Numerical Analysis of Tractor Accidents using Driving Simulator for Autonomous Driving Tractor
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics Engineering

Autonomous driving of automobiles is a hot research topic in recent years. The autonomous driving tractor also has been studied in the agricultural field as well as an autonomous driving automobile. On the other hand, tractor accidents frequently occur ...
Autonomous Driving: Investigating the Feasibility of Bimodal Take-Over Requests

Autonomous vehicles will need de-escalation strategies to compensate when reaching system limitations. Car-driver handovers can be considered one possible method to deal with system boundaries. The authors suggest a bimodal auditory and visual handover ...
DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems
ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

While Deep Neural Networks (DNNs) have established the fundamentals of image-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To address the safety issues in autonomous driving systems, a recent set of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

October 2022

2006 pages

ISBN:9781450394758

DOI:10.1145/3551349

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 January 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

ASE '22

ASE '22: 37th IEEE/ACM International Conference on Automated Software Engineering

October 10 - 14, 2022

MI, Rochester, USA

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)118
Downloads (Last 6 weeks)7

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Biagiola MTonella P(2024)Boundary State Generation for Testing and Improvement of Autonomous Driving SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.342081650:8(2040-2053)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3420816
Schafhalter PKalra SXu LGonzalez JStoica I(2023)Leveraging Cloud Computing to Make Autonomous Vehicles Safer2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10341821(5559-5566)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10341821

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents