research-article

Data-Driven Parameterized Corner Synthesis for Efficient Validation of Perception Systems for Autonomous Driving

Authors:

Xin LiAuthors Info & Claims

ACM Transactions on Cyber-Physical Systems, Volume 7, Issue 2

Article No.: 9, Pages 1 - 24

https://doi.org/10.1145/3571286

Published: 19 April 2023 Publication History

Abstract

Today's automotive cyber-physical systems for autonomous driving aim to enhance driving safety by replacing the uncertainties posed by human drivers with standard procedures of automated systems. However, the accuracy of in-vehicle perception systems may significantly vary under different operational conditions (e.g., fog density, light condition, etc.) and consequently degrade the reliability of autonomous driving. A perception system for autonomous driving must be carefully validated with an extremely large dataset collected under all possible operational conditions in order to ensure its robustness. The aforementioned dataset required for validation, however, is expensive or even impossible to acquire in practice, since most operational corners rarely occur in a real-world environment. In this paper, we propose to generate synthetic datasets at a variety of operational corners by using a parameterized cycle-consistent generative adversarial network (PCGAN). The proposed PCGAN is able to learn from an image dataset recorded at real-world operational conditions with only a few samples at corners and synthesize a large dataset at a given operational corner. By taking STOP sign detection as an example, our numerical experiments demonstrate that the proposed approach is able to generate high-quality synthetic datasets to facilitate accurate validation.

References

[1]

L. V. Nguyen et al. 2018. Cyber-physical specification mismatches. ACM Trans. on Cyber-Physical Systems 2, 4 (2018), 23:1–23:26.

Digital Library

[2]

S. Chakraborty et al. 2016. Automotive cyber–physical systems: A tutorial introduction. IEEE Design & Test 33, 4 (2016), 92–108.

[3]

Society of Automotive Engineers International. 2018. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. J3016_201806.

[4]

C. Urmson et al. 2008. Autonomous driving in urban environments: BOSS and the urban challenge. Journal of Field Robot 25, 8 (2018), 425–466.

Digital Library

[5]

J. Kim et al. 2013. Parallel scheduling for cyber-physical systems: Analysis and case study on a self-driving car. International Conference on Cyber-Physical Systems. 31–40.

Digital Library

[6]

A. Bhat et al. 2018. Tools and methodologies for autonomous driving systems. Proc. of the IEEE 106, 9 (2018), 1700–1716.

[7]

A. Bhat et al. 2017. Practical task allocation for software fault-tolerance and its implementation in embedded automotive systems. IEEE Real-Time and Embedded Technology and Applications Symposium (2017), 87–98.

[8]

S. Starik and M. Werman. 2003. Simulation of rain in videos. Texture Workshop, ICCV 2 (2003), 406–409.

[9]

K. Garg et al. 2006. Photorealistic rendering of rain streaks. ACM Transactions on Graphics 25, 3 (2006), 996–1002.

Digital Library

[10]

D. Hospach et al. 2016. Simulation of falling rain for robustness testing of video-based surround sensing systems. Design, Automation & Test in Europe Conference & Exhibition (2016), 233–236.

[11]

M. Negru et al. 2015. Exponential contrast restoration in fog conditions for driving assistance. IEEE Trans. on Intelligent Transportation Systems 16, 4 (2015), 2257–2268.

Digital Library

[12]

R. Gallen et al. 2015. Nighttime visibility analysis and estimation method in the presence of dense fog. IEEE Trans. on Intelligent Transportation Systems 16, 1 (2015), 310–320.

Digital Library

[13]

M. Zhang et al. 2018. DeepRoad: GAN-based metamorphic autonomous driving system testing. arXiv preprint. arXiv:1802.02295.

[14]

H. Yu and X. Li. 2018. Intelligent corner synthesis via cycle-consistent generative adversarial networks for efficient validation of autonomous driving systems. Asia and South Pacific Design Automation Conference. 9–15.

Digital Library

[15]

H. Yu et al. 2017. Impact of circuit-level non-idealities on vision-based autonomous driving systems. International Conference on Computer-Aided Design. 976–983.

Digital Library

[16]

W. Li et al. 2019. AADS: Augmented autonomous driving simulation using data-driven algorithms. Science Robotics 4, 28 (2019).

[17]

S. Aoki and R. Rajkumar. 2018. Dynamic intersections and self-driving vehicles. International Conference on Cyber-Physical Systems. 320–330.

Digital Library

[18]

D. Zhang et al. 2017. Heterogeneous model integration for multi-source urban infrastructure data. ACM Trans. on Cyber-Physical Systems 1, 1 (2017), 4:1–4:26.

Digital Library

[19]

D. Zhao and H. Peng. 2020. From the lab to the street: Solving the challenge of accelerating automated vehicle testing. arXiv preprint. arXiv:1707.04792.

[20]

E. A. Lee. 2017. Fundamental limits of cyber-physical systems modeling. ACM Trans. on Cyber-Physical Systems 1, 1 (2017), 3:1-3:26.

Digital Library

[21]

M. Nentwig and M. Stamminger. 2011. Hardware-in-the-loop testing of computer vision based driver assistance systems. Intelligent Vehicles Symposium (2011), 339–344.

[22]

M. Nentwig et al. 2012. Concerning the applicability of computer graphics for the evaluation of image processing algorithms. International Conference on Vehicular Electronics and Safety. 205–210.

[23]

S. Tokunaga and T. Azumi. 2017. Demo abstract: Co-simulation framework for autonomous driving systems with MATLAB/Simulink. IEEE Real-Time and Embedded Technology and Applications Symposium (2017), 153–154.

[24]

I. Goodfellow et al. 2014. Generative adversarial nets. International Conference on Neural Information Processing Systems. 2672–2680.

Digital Library

[25]

L. Yang et al. 2018. Real-to-virtual domain unification for end-to-end autonomous driving. arXiv preprint. arXiv:1801.03458.

[26]

J. Zhu et al.. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. International Conference on Computer Vision. 2242–2251.

[27]

A. Radford et al. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint. arXiv:1511.06434.

[28]

W. Qiu and A. Yuille. 2016. UnrealCV: Connecting computer vision to unreal engine. arXiv preprint. arXiv:1609.01326.

[29]

A. Dosovitskiy et al. 2017. CARLA: An open urban driving simulator. arXiv preprint. arXiv:1711.03938.

[30]

S. R. Richter et al. 2016. Playing for data: Ground truth from computer games. arXiv preprint. arXiv:1608.02192.

[31]

E. L. Denton et al. 2015. Deep generative image models using a Laplacian pyramid of adversarial networks. Advances in Neural Information Processing Systems (2015), 1486–1494.

[32]

P. Isola et al. 2017. Image-to-image translation with conditional adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition (2017).

[33]

M. Uricar et al. 2019. Yes, we GAN: Applying adversarial techniques for autonomous driving. Electronic Imaging 15, (2019), 48:1–48:17.

[34]

Vaisala. Visibility sensors PWD10, PWD20 and PWD20W. https://www.vaisala.com/en/products/instruments-sensors-and-other-measurement-devices/weather-stations-and-sensors/pwd10-20w.

[35]

A. Kendall et al. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. arXiv preprint. arXiv:1705.07115.

[36]

E. T. Jaynes. 1957. Information theory and statistical mechanics. II. Physical Review 108, 2 (1957), 171–190.

[37]

C. Bishop. 2007. Pattern Recognition and Machine Learning. Prentice Hall.

[38]

B. W. Silverman. 1986. Density Estimation for Statistics and Data Analysis. CRC Press.

[39]

S. L. Smith et al. 2017. Don't decay the learning rate, increase the batch size. arXiv preprint. arXiv: 1711.00498.

[40]

D. Kingma and J. B. Adam. 2014. A method for stochastic optimization. arXiv preprint. arXiv:1412.6980.

[41]

J. Towns et al. 2014. XSEDE: Accelerating scientific discovery. Computing in Science & Engineering 16, 5 (2014), 62–74.

[42]

W. Shi et al. 2017. An FPGA-based hardware accelerator for traffic sign detection. IEEE Trans. on Very Large Scale Integration Systems 25, 4 (2017), 1362–1372.

Digital Library

[43]

P. Dollar. 2016. Piotr's computer vision MATLAB toolbox. https://pdollar.github.io/toolbox/.

[44]

R. Timof et al. 2013. Traffic sign recognition — how far are we from the solution? International Joint Conference on Neural Networks. 1–8.

[45]

J. Stallkamp et al. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks 32 (2012), 323–332.

Digital Library

[46]

S. Houben et al. 2014. Detection of traffic signs in real-world images: The German traffic sign detection benchmark. International Joint Conference on Neural Networks. 1–8.

[47]

L. Li et al. 2007. OPTIMOL: Automatic object picture collection via incremental model learning. Computer Vision and Pattern Recognition (2007), 147–168.

[48]

H. Koschmieder. 1924. Theorie der horizontalen sichtweite. Beitrage zur physik der freien atmosphäre. Meteorol. 12 (1924), 3353.

[49]

I. Sousa et al. 2017. An efficient visibility prediction framework for free-space optical systems. Wireless Personal Communications 96, 3 (2017), 3483–3498.

Digital Library

[50]

National Centers for Environmental Information of National Oceanic and Atmospheric Administration. Climate data online (CDO). https://www.ncdc.noaa.gov/cdo-web/.

[51]

G. C. Holst. 1998. CCD Arrays, Cameras, and Displays. JCD Pub.

[52]

International Organization for Standardization. 1974. Photography — general purpose photographic exposure meters (photoelectric type) — Guide to product specification. ISO 2720:1974.

[53]

Lawrence Berkeley National Laboratory in California. Radiance synthetic imaging system. https://floyd.lbl.gov/radiance/HOME.html.

[54]

M. Woehrle et al. 2019. Open questions in testing of learned computer vision functions for automated driving. Computer Safety, Reliability, and Security 11699 (2019), 333–345.

Digital Library

Cited By

Rawlley OGupta SKathera HKatyal SBatwara Y(2024)Employing cross-domain modelings for robust object detection in dynamic environment of autonomous vehiclesMultimedia Tools and Applications10.1007/s11042-024-19409-zOnline publication date: 31-May-2024
https://doi.org/10.1007/s11042-024-19409-z
Yu JHo DWierman A(2023)Online Adversarial Stabilization of Unknown Networked SystemsACM SIGMETRICS Performance Evaluation Review10.1145/3606376.359355751:1(73-74)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3606376.3593557

Index Terms

Data-Driven Parameterized Corner Synthesis for Efficient Validation of Perception Systems for Autonomous Driving
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
      1. Robotic autonomy
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object detection

Recommendations

Numerical Analysis of Tractor Accidents using Driving Simulator for Autonomous Driving Tractor
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics Engineering

Autonomous driving of automobiles is a hot research topic in recent years. The autonomous driving tractor also has been studied in the agricultural field as well as an autonomous driving automobile. On the other hand, tractor accidents frequently occur ...
Read More
Autonomous Driving: Investigating the Feasibility of Bimodal Take-Over Requests

Autonomous vehicles will need de-escalation strategies to compensate when reaching system limitations. Car-driver handovers can be considered one possible method to deal with system boundaries. The authors suggest a bimodal auditory and visual handover ...
Read More
The Ceprei-Scape Dataset for the Validation of Autonomous Driving
CSAI '18: Proceedings of the 2018 2nd International Conference on Computer Science and Artificial Intelligence

Validation and test is an enabling factor of the autonomous driving for a wide range of applications, especially for vision-based applications. Deep learning methods has been widely deployed in vision-based applications of autonomous driving and has ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Cyber-Physical Systems

ACM Transactions on Cyber-Physical Systems Volume 7, Issue 2

April 2023

187 pages

ISSN:2378-962X

EISSN:2378-9638

DOI:10.1145/3592783

Editor:
Chenyang Lu
Washington University in St. Louis, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 19 April 2023

Online AM: 20 January 2023

Accepted: 28 October 2022

Revised: 10 January 2021

Received: 20 February 2020

Published in TCPS Volume 7, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation
National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
235
Total Downloads

Downloads (Last 12 months)103
Downloads (Last 6 weeks)9

Other Metrics

View Author Metrics

Citations

Cited By

Rawlley OGupta SKathera HKatyal SBatwara Y(2024)Employing cross-domain modelings for robust object detection in dynamic environment of autonomous vehiclesMultimedia Tools and Applications10.1007/s11042-024-19409-zOnline publication date: 31-May-2024
https://doi.org/10.1007/s11042-024-19409-z
Yu JHo DWierman A(2023)Online Adversarial Stabilization of Unknown Networked SystemsACM SIGMETRICS Performance Evaluation Review10.1145/3606376.359355751:1(73-74)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3606376.3593557

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents