research-article

Utilizing S-TaLiRo as an automatic test generation framework for autonomous vehicles

Authors:

Cumhur Erkan Tuncali,

Theodore P. Pavlic,

Georgios FainekosAuthors Info & Claims

2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC)

Pages 1470 - 1475

https://doi.org/10.1109/ITSC.2016.7795751

Published: 01 November 2016 Publication History

Abstract

This paper proposes an approach to automatically generating test cases for testing motion controllers of autonomous vehicular systems. Test scenarios may consist of single or multiple vehicles under test at the same time. Tests are performed in simulation environments. The approach is based on using a robustness metric for evaluating simulation outcomes as a cost function. Initial states and inputs are updated by stochastic optimization methods between the tests for achieving smaller robustness values. The test generation framework has been implemented in the toolbox S-TaLiRo. The proposed framework's ability to generate interesting test cases is demonstrated by a case study.

References

[1]

K. Bengler, K. Dietmayer, B. Farber, M. Maurer, C. Stiller, and H. Winner, “Three decades of driver assistance systems: Review and future perspectives,” Intelligent Transportation Systems Magazine, IEEE, vol. 6, no. 4, pp. 6–22, 2014.

[2]

M. Maurer and H. Winner, Automotive systems engineering. Springer 2013.

Digital Library

[3]

K.-W. Han and C.-H. Chang, “Gain margins and phase margins for control systems with adjustable parameters,” Journal of guidance, control, and dynamics, vol. 13, no. 3, pp. 404–408, 1990.

[4]

H. Winner, S. Hakuli, F. Lotz, and C. Singer, Handbook of Driver Assistance Systems: Basic Information, Components and Systems for Active Safety and Comfort. Springer 2015.

[5]

J. E. Stellet, M. R. Zofka, J. Schumacher, T. Schamm, F. Niewels, and J. M. Zollner, “Testing of advanced driver assistance towards automated driving: A survey and taxonomy on existing approaches and open questions,” in Intelligent Transportation Systems (ITSC), 2015 IEEE 18th InternationalConference on. IEEE, 2015, pp. 1455–1462.

[6]

T. Bock, M. Maurer, and G. Farber, “Validation of the vehicle in the loop (VIL); a milestone for the simulation of driver assistance systems,” in Intelligent Vehicles Symposium, 2007 IEEE. IEEE, 2007, pp. 612–617.

[7]

M. Althoff and J. M. Dolan, “Online verification of automated road vehicles using reachability analysis,” Robotics, IEEE Transactions on, vol. 30, no. 4, pp. 903–918, 2014.

[8]

H. Abbas, G. Fainekos, S. Sankaranarayanan, F. Ivančić, and A. Gupta, “Probabilistic temporal logic falsification of cyber-physical systems,” ACM Transactions on Embedded Computing Systems (TECS), vol. 12, no. 2s, p. 95, 2013.

[9]

M. Egerstedt and C. Martin, Control Theoretic Splines: Optimal Control, Statistics, and Path Planning. Princeton University Press, 2009.

[10]

Y. Annpureddy, C. Liu, G. Fainekos, and S. Sankaranarayanan, “S-taliro: A tool for temporal logic falsification for hybrid systems,” in International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 2011, pp. 254–257.

[11]

MATLAB, version 9.0.0 (R2016a). Natick, Massachusetts:The MathWorks Inc., 2016.

[12]

O. Michel, “Webots™: Professional mobile robot simulation,” arXiv preprint cs/0412052, 2004.

[13]

Mechanical Simulation, “CarSim,” 2016. [Online]. Available: http://www.carsim.com/

[14]

G. E. Fainekos and G. J. Pappas, “Robustness of temporal logic specifications for continuous-time signals,” Theoretical Computer Science, vol. 410, no. 42, pp. 4262–4291, 2009.

Digital Library

[15]

J. C. Hayward, “Near-miss determination through use of a scale of danger,” Highway ResearchRecord, no. 384, 1972.

[16]

J. Ward, G. Agamennoni, S. Worrall, and E. Nebot, “Vehicle collision probability calculation for general traffic scenarios under uncertainty,” in Intelligent Vehicles Symposium Proceedings, 2014 IEEE., 2014, pp. 986–992.

[17]

H. Abbas and G. Fainekos, “Convergence proofs for simulated annealing falsification of safety properties,” in Communication, Control, and Computing (Allerton), 2012 50th Annual AllertonConference on. IEEE, 2012, pp. 1594–1601.

[18]

K. Zhang, J. Sprinkle, and R. G. Sanfelice, “A hybrid model predictive controller for path planning and path following,” in Proceedings of the ACM/IEEE Sixth InternationalConference on Cyber-Physical Systems.ACM, 2015, pp. 139–148.

[19]

E. Narby, “Modeling and estimation of dynamic tire properties,” Master's thesis, Linkopings Universitet, Linkoping, 2006.

[20]

G. Walsh, D. Tilbury, S. Sastry, R. Murray, and J.-P. Laumond, “Stabilization of trajectories for systems with nonholonomic constraints,” Automatic Control, IEEE Transactions on, vol. 39, no. 1, pp. 216–222, 1994.

[21]

G. M. Hoffmann, C. J. Tomlin, M. Montemerlo, and S. Thrun, “Autonomous automobile trajectory tracking for off-road driving: Controller design, experimental validation and racing,” in American Control Conference, 2007, pp. 2296–2301.

[22]

P. Liu and U. Ozguner, “Predictive control of a vehicle convoy considering lane change behavior of the preceding vehicle,” in American Control Conference (ACC), 2015. IEEE, 2015, pp. 4374–4379.

[23]

F. N. Fritsch and R. E. Carlson, “Monotone piecewise cubic interpolation,” SIAM Journal on Numerical Analysis, vol. 17, no. 2, pp. 238–246, 1980.

Digital Library

[24]

J. Kapinski, J. Deshmukh, X. Jin, H. Ito, and K. Butts, “Simulation-guided approaches for verification of automotive powertrain control systems,” in 2015 American Control Conference (ACC). IEEE, 2015, pp. 4086–4095.

[25]

B. Nagy and A. Kelly, “Trajectory generation for car-like robots using cubic curvature polynomials,” Field and Service Robots, vol. 11, 2001.

Cited By

Sorokin LMunaro TSafin DLiao BMolin ARoychoudhury APaiva AAbreu RStorey M(2024)OpenSBT: A Modular Framework for Search-based Testing of Automated Driving SystemsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3640027(94-98)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3640027
Song ROzmen MKim HMuller RCelik ZBianchi ACalandrino JTroncoso C(2023)Discovering adversarial driving maneuvers against autonomous vehiclesProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620403(2957-2974)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620403
Formica FPetrunti NBruck LPantelic VLawford MMenghi CChandra SBlincoe KTonella P(2023)Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial SimulatorProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613894(1949-1960)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613894
Show More Cited By

Recommendations

Automatic test data generation
Constraint-Based Automatic Test Data Generation

A novel technique for automatically generating test data is presented. The technique is based on mutation analysis and creates test data that approximate relative adequacy. It is a fault-based technique that uses algebraic constraints to describe test ...
Automatic Test Generation using Checkpoint Encoding and Antirandom Testing
ISSRE '97: Proceedings of the Eighth International Symposium on Software Reliability Engineering

The implementation of an efficient automatic test generation scheme for black-box testing is discussed. It uses checkpoint encoding and antirandom testing schemes. Checkpoint encoding converts test generation to a binary problem. The checkpoints are ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC)

2678 pages

Copyright © 2016.

Publisher

IEEE Press

Publication History

Published: 01 November 2016

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sorokin LMunaro TSafin DLiao BMolin ARoychoudhury APaiva AAbreu RStorey M(2024)OpenSBT: A Modular Framework for Search-based Testing of Automated Driving SystemsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3640027(94-98)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3640027
Song ROzmen MKim HMuller RCelik ZBianchi ACalandrino JTroncoso C(2023)Discovering adversarial driving maneuvers against autonomous vehiclesProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620403(2957-2974)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620403
Formica FPetrunti NBruck LPantelic VLawford MMenghi CChandra SBlincoe KTonella P(2023)Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial SimulatorProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613894(1949-1960)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613894
Tang SZhang ZZhang YZhou JGuo YLiu SGuo SLi YMa LXue YLiu Y(2023)A Survey on Automated Driving System Testing: Landscapes and TrendsACM Transactions on Software Engineering and Methodology10.1145/357964232:5(1-62)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3579642
Horel JLaugier CMarsso LMateescu RMuller LPaigwar ARenzaglia ASerwe WBolchini CO'Connor IVerbauwhede IWille R(2022)Using formal conformance testing to generate scenarios for autonomous vehiclesProceedings of the 2022 Conference & Exhibition on Design, Automation & Test in Europe10.5555/3539845.3539969(532-537)Online publication date: 14-Mar-2022
https://dl.acm.org/doi/10.5555/3539845.3539969
Tian HWu GYan JJiang YWei JChen WLi SYe D(2022)Generating Critical Test Scenarios for Autonomous Driving Systems via Influential Behavior PatternsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3560430(1-12)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3560430
Tian HJiang YWu GYan JWei JChen WLi SYe DRoychoudhury ACadar CKim M(2022)MOSAT: finding safety violations of autonomous driving systems using multi-objective genetic algorithmProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549100(94-106)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549100
Birkemeyer LPett TVogelsang ASeidl CSchaefer I(2022)Feature-Interaction Sampling for Scenario-based Testing of Advanced Driver Assistance Systems✱Proceedings of the 16th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3510466.3510474(1-10)Online publication date: 23-Feb-2022
https://dl.acm.org/doi/10.1145/3510466.3510474
Hildebrandt CElbaum SBezzo NDwyer MKhurshid SPăsăreanu C(2020)Feasible and stressful trajectory generation for mobile robotsProceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3395363.3397387(349-362)Online publication date: 18-Jul-2020
https://dl.acm.org/doi/10.1145/3395363.3397387
Gambi AHuynh TFraser GDumas MPfahl DApel SRusso A(2019)Generating effective test cases for self-driving cars from police reportsProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338942(257-267)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338942

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents