research-article

Requirements driven falsification with coverage metrics

Authors:

Adel Dokhanchi,

Rahul T. Sriniva,

Sriram Sankaranarayanan,

Georgios FainekosAuthors Info & Claims

EMSOFT '15: Proceedings of the 12th International Conference on Embedded Software

Pages 31 - 40

Published: 04 October 2015 Publication History

Abstract

Specification guided falsification methods for hybrid systems have recently demonstrated their value in detecting design errors in models of safety critical systems. In specification guided falsification, the correctness problem, i.e., does the system satisfy the specification, is converted into an optimization problem where local negative minima indicate design errors. Due to the complexity of the resulting optimization problem, the problem is solved iteratively by performing a number of simulations on the system. Even though it is theoretically guaranteed that falsification methods will eventually find the bugs in the system, in practice, the performance of these methods, i.e., how many tests/simulations are executed before a bug is detected, depends on the specification, on the system and on the optimization method. In this paper, we define and utilize coverage metrics on the state space of hybrid systems in order to improve the performance of the falsification methods.

References

[1]

TaLiRo Tools. https://sites.google.com/a/asu.edu/s-taliro/.

[2]

H. Abbas and G. Fainekos. Linear hybrid system falsification through local search. In Automated Technology for Verification and Analysis, volume 6996 of LNCS, pages 503--510. Springer, 2011.

Digital Library

[3]

H. Abbas and G. Fainekos. Convergence proofs for simulated annealing falsification of safety properties. In Proc. of 50th Annual Allerton Conference on Communication, Control, and Computing. IEEE Press, 2012.

[4]

H. Abbas and G. Fainekos. Computing descent direction of mtl robustness for non-linear systems. In American Control Conference, 2013. {Under review}.

[5]

H. Abbas, G. E. Fainekos, S. Sankaranarayanan, F. Ivancic, and A. Gupta. Probabilistic temporal logic falsification of cyber-physical systems. ACM Transactions on Embedded Computing Systems, 12(s2), May 2013.

Digital Library

[6]

H. Abbas, B. Hoxha, G. Fainekos, J. V. Deshmukh, J. Kapinski, and K. Ueda. Conformance testing as falsification for cyber-physical systems. Technical Report 1401.5200, arXiv, 2014.

[7]

H. Abbas, B. Hoxha, G. Fainekos, and K. Ueda. Robustness-guided temporal logic testing and verification for stochastic cyber-physical systems. In Proc. of IEEE International Conference on CYBER Technology in Automation, Control, and Intelligent Systems, 2014.

[8]

H. Abbas, H. Mittelmann, and G. Fainekos. Formal property verification in a conformance testing framework. In 12th ACM-IEEE International Conference on Formal Methods and Models for System Design, 2014.

Digital Library

[9]

H. Abbas, A. Winn, G. Fainekos, and A. A. Julius. Functional gradient descent method for metric temporal logic specifications. In American Control Conference, 2014. {Under review}.

[10]

T. Akazaki and I. Hasuo. Time robustness in mtl and expressivity in hybrid system falsification. In Computer Aided Verification, volume 9207 of LNCS, pages 356--374. Springer, 2015.

[11]

R. Alur. Principles of Cyber-Physical Systems. MIT Press, 2015.

Digital Library

[12]

P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press, 2008.

Digital Library

[13]

Y. S. R. Annapureddy and G. E. Fainekos. Ant colonies for temporal logic falsification of hybrid systems. In Proceedings of the 36th Annual Conference of IEEE Industrial Electronics, pages 91--96, 2010.

[14]

Y. S. R. Annapureddy, C. Liu, G. E. Fainekos, and S. Sankaranarayanan. S-TaLiRo: A tool for temporal logic falsification for hybrid systems. In Tools and algorithms for the construction and analysis of systems, volume 6605 of LNCS, pages 254--257. Springer, 2011.

Digital Library

[15]

M. Branicky, M. Curtiss, J. Levine, and S. Morgan. Sampling-based planning, control and verification of hybrid systems. IEE Proc.-Control Theory Appl., 153(5):575--590, 2006.

[16]

X. Chen, E. Abraham, and S. Sankaranarayanan. Flow*: An analyzer for non-linear hybrid systems. In Computer-Aided Verification, 2013.

Digital Library

[17]

T. Dang, A. Donze, O. Maler, and N. Shalev. Sensitive state-space exploration. In Proc. of the 47th IEEE Conference on Decision and Control, pages 4049--4054, Dec. 2008.

[18]

T. Dang and N. Shalev. State estimation and property-guided exploration for hybrid systems testing. In International Conference Testing Software and Systems, volume 7641 of LNCS, pages 152--167. Springer, 2012.

[19]

Y. Deng, A. Rajhans, and A. A. Julius. Strong: A trajectory-based verification toolbox for hybrid systems. In Quantitative Evaluation of Systems, volume 8054 of LNCS, pages 165--168. Springer, 2013.

Digital Library

[20]

A. Donze. Breach, a toolbox for verification and parameter synthesis of hybrid systems. In Computer Aided Verification, volume 6174 of LNCS, pages 167--170. Springer, 2010.

Digital Library

[21]

A. Donze and O. Maler. Systematic simulation using sensitivity analysis. In Hybrid Systems: Computation and Control, volume 4416 of LNCS, pages 174--189. Springer, 2007.

Digital Library

[22]

A. Donze and O. Maler. Robust satisfaction of temporal logic over real-valued signals. In Formal Modelling and Analysis of Timed Systems, volume 6246 of LNCS. Springer, 2010.

Digital Library

[23]

J. Eker, J. Janneck, E. A. Lee, J. Liu, X. Liu, J. Ludvig, S. Sachs, and Y. Xiong. Taming heterogeneity - the ptolemy approach. Proceedings of the IEEE, 91(1):127--144, Jan. 2003.

[24]

G. Fainekos and G. J. Pappas. Robustness of temporal logic specifications. In Formal Approaches to Testing and Runtime Verification, volume 4262 of LNCS, pages 178--192. Springer, 2006.

Digital Library

[25]

G. Fainekos and G. J. Pappas. Robustness of temporal logic specifications for continuous-time signals. Theoretical Computer Science, 410(42):4262--4291, 2009.

Digital Library

[26]

G. Fainekos, S. Sankaranarayanan, K. Ueda, and H. Yazarel. Verification of automotive control applications using s-taliro. In Proceedings of the American Control Conference, 2012.

[27]

G. E. Fainekos, A. Girard, and G. J. Pappas. Temporal logic verification using simulation. In E. Asarin and P. Bouyer, editors, Proceedings of the 4th International Conference on Formal Modelling and Analysis of Timed Systems, volume 4202 of LNCS, pages 171--186. Springer, 2006.

Digital Library

[28]

G. Frehse, C. L. Guernic, A. Donzé, S. Cotton, R. Ray, O. Lebeltel, R. Ripado, A. Girard, T. Dang, and O. Maler. Spaceex: Scalable verification of hybrid systems. In Proceedings of the 23d CAV, 2011.

Digital Library

[29]

A. Girard and G. J. Pappas. Verification using simulation. In Hybrid Systems: Computation and Control (HSCC), volume 3927 of LNCS, pages 272--286. Springer, 2006.

Digital Library

[30]

T. A. Henzinger, P. W. Kopke, A. Puri, and P. Varaiya. What's decidable about hybrid automata? J. Comput. Syst. Sci., 57(1):94--124, 1998.

Digital Library

[31]

B. Hoxha, H. Abbas, and G. Fainekos. Using s-taliro on industrial size automotive models. In Proc. of Applied Verification for Continuous and Hybrid Systems, 2014.

[32]

Z. Huang and S. Mitra. Computing bounded reach sets from sampled simulation traces. In The 15th International Conference on Hybrid Systems: Computation and Control (HSCC 2012), Beijing, China., 2012.

Digital Library

[33]

Z. Huang and S. Mitra. Proofs from simulations and modular annotations. In Proceedings of the 17th International Conference on Hybrid Systems: Computation and Control, pages 183--192, 2014.

Digital Library

[34]

J.-B. Jeannin, K. Ghorbal, Y. Kouskoulas, R. Gardner, A. Schmidt, and E. Z. A. Platzer. A formally verified hybrid system for the next-generation airborne collision avoidance system. In TACAS, volume 9035 of LNCS, pages 21--36. Springer, 2015.

[35]

X. Jin, A. Donze, J. Deshmukh, and S. Seshia. Mining requirements from closed-loop control models. In Hybrid Systems: Computation and Control. ACM Press, 2013.

Digital Library

[36]

A. A. Julius, G. E. Fainekos, M. Anand, I. Lee, and G. J. Pappas. Robust test generation and coverage for hybrid systems. In Hybrid Systems: Computation and Control, volume 4416 of LNCS, pages 329--342. Springer, 2007.

Digital Library

[37]

J. Kapinski, J. Deshmukh, X. Jin, H. Ito, and K. R. Butts. Simulation-guided approaches for verification of automotive powertrain control systems. In American Control Conference, 2015.

[38]

Z. Kong, A. Jones, A. M. Ayala, E. A. Gol, and C. Belta. Temporal logic inference for classification and prediction from data. In Proceedings of the 17th International Conference on Hybrid Systems: Computation and Control, 2014.

Digital Library

[39]

R. Koymans. Specifying real-time properties with metric temporal logic. Real-Time Systems, 2(4):255--299, 1990.

Digital Library

[40]

J. Lygeros, K. H. Johansson, S. N. Simic, J. Zhang, and S. Sastry. Dynamical properties of hybrid automata. IEEE Transactions on Automatic Control, 48:2--17, 2003.

[41]

O. Maler and D. Nickovic. Monitoring temporal properties of continuous signals. In Proceedings of FORMATS-FTRTFT, volume 3253 of LNCS, pages 152--166, 2004.

[42]

T. Nghiem, S. Sankaranarayanan, G. E. Fainekos, F. Ivancic, A. Gupta, and G. J. Pappas. Monte-carlo techniques for falsification of temporal properties of non-linear hybrid systems. In Proceedings of the 13th ACM International Conference on Hybrid Systems: Computation and Control, pages 211--220. ACM Press, 2010.

Digital Library

[43]

E. Plaku, L. E. Kavraki, and M. Y. Vardi. Falsification of ltl safety properties in hybrid systems. In Proc. of the Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), volume 5505 of LNCS, pages 368--382. Springer, 2009.

Digital Library

[44]

A. Platzer. Logical Analysis of Hybrid Systems: Proving Theorems for Complex Dynamics. Springer, Heidelberg, 2010.

Digital Library

[45]

S. Sankaranarayanan and G. Fainekos. Falsification of temporal properties of hybrid systems using the cross-entropy method. In ACM International Conference on Hybrid Systems: Computation and Control, 2012.

Digital Library

[46]

A. K. Seda and P. Hitzler. Generalized distance functions in the theory of computation. The Computer Journal, 53(4):bxm108443--464, 2008.

Digital Library

[47]

P. Tabuada. Verification and Control of Hybrid Systems: A Symbolic Approach. Springer, 2009.

Digital Library

[48]

S. Tripakis and T. Dang. Model-Based Design for Embedded Systems, chapter Modeling, Verification and Testing using Timed and Hybrid Automata, pages 383--436. CRC Press, 2009.

[49]

H. Yang, B. Hoxha, and G. Fainekos. Querying parametric temporal logic properties on embedded systems. In Int. Conference on Testing Software and Systems, volume 7641, pages 136--151. Springer, 2012.

[50]

A. Zutshi, J. V. Deshmukh, S. Sankaranarayanan, and J. Kapinski. Multiple shooting, cegar-based falsification for hybrid systems. In Proceedings of the 14th International Conference on Embedded Software, page 5. ACM, 2014.

Digital Library

[51]

A. Zutshi, S. Sankaranarayanan, J. V. Deshmukh, and J. Kapinski. A trajectory splicing approach to concretizing counterexamples for hybrid systems. In Decision and Control (CDC), 2013 IEEE 52nd Annual Conference on, pages 3918--3925. IEEE, 2013.

[52]

A. Zutshi, S. Sankaranarayanan, J. V. Deshmukh, J. Kapinski, and X. Jin. Falsification of safety properties for closed loop control systems. In Proceedings of the 18th International Conference on Hybrid Systems: Computation and Control, pages 299--300. ACM, 2015.

Digital Library

Cited By

Mathesen LPedrielli GFainekos G(2021)Efficient Optimization-Based Falsification of Cyber-Physical Systems with Multiple Conjunctive Requirements2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)10.1109/CASE49439.2021.9551474(732-737)Online publication date: 23-Aug-2021
https://dl.acm.org/doi/10.1109/CASE49439.2021.9551474
Mancini TMari FMassini AMelatti ISalvo ITronci E(2017)On minimising the maximum expected verification timeInformation Processing Letters10.1016/j.ipl.2017.02.001122:C(8-16)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1016/j.ipl.2017.02.001

Index Terms

Requirements driven falsification with coverage metrics

Recommendations

Coverage metrics for formal verification

In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is and whether it really covers all the behaviors of ...
Abstraction for falsification
CAV'05: Proceedings of the 17th international conference on Computer Aided Verification

Abstraction is traditionally used in the process of verification. There, an abstraction of a concrete system is sound if properties of the abstract system also hold in the concrete system. Specifically, if an abstract state a satisfies a property ψ then ...
How verified (or tested) is my code? Falsification-driven verification and testing

Formal verification has advanced to the point that developers can verify the correctness of small, critical modules. Unfortunately, despite considerable efforts, determining if a "verification" verifies what the author intends is still difficult. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EMSOFT '15: Proceedings of the 12th International Conference on Embedded Software

October 2015

276 pages

ISBN:9781467380799

Program Chairs:
Alain Girault
INRIA, Grenoble, France
,
Nan Guan
Northeastern University, China

Sponsors

SIGBED: ACM Special Interest Group on Embedded Systems
SIGDA: ACM Special Interest Group on Design Automation
IEEE: Institute of Electrical and Electronics Engineers
SIGMICRO: ACM Special Interest Group on Microarchitectural Research and Processing

Publisher

IEEE Press

Publication History

Published: 04 October 2015

Check for updates

Qualifiers

Research-article

Conference

ESWEEK'15

Sponsor:

ESWEEK'15: ELEVENTH EMBEDDED SYSTEM WEEK

October 4 - 9, 2015

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 60 of 203 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
85
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mathesen LPedrielli GFainekos G(2021)Efficient Optimization-Based Falsification of Cyber-Physical Systems with Multiple Conjunctive Requirements2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)10.1109/CASE49439.2021.9551474(732-737)Online publication date: 23-Aug-2021
https://dl.acm.org/doi/10.1109/CASE49439.2021.9551474
Mancini TMari FMassini AMelatti ISalvo ITronci E(2017)On minimising the maximum expected verification timeInformation Processing Letters10.1016/j.ipl.2017.02.001122:C(8-16)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1016/j.ipl.2017.02.001

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents