research-article

Witness validation and stepwise testification across software verifiers

Authors:

Matthias Dangl,

Daniel Dietsch,

Matthias Heizmann,

Andreas StahlbauerAuthors Info & Claims

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Pages 721 - 733

https://doi.org/10.1145/2786805.2786867

Published: 30 August 2015 Publication History

Abstract

It is commonly understood that a verification tool should provide a counterexample to witness a specification violation. Until recently, software verifiers dumped error witnesses in proprietary formats, which are often neither human- nor machine-readable, and an exchange of witnesses between different verifiers was impossible. To close this gap in software-verification technology, we have defined an exchange format for error witnesses that is easy to write and read by verification tools (for further processing, e.g., witness validation) and that is easy to convert into visualizations that conveniently let developers inspect an error path. To eliminate manual inspection of false alarms, we develop the notion of stepwise testification: in a first step, a verifier finds a problematic program path and, in addition to the verification result FALSE, constructs a witness for this path; in the next step, another verifier re-verifies that the witness indeed violates the specification. This process can have more than two steps, each reducing the state space around the error path, making it easier to validate the witness in a later step. An obvious application for testification is the setting where we have two verifiers: one that is efficient but imprecise and another one that is precise but expensive. We have implemented the technique of error-witness-driven program analysis in two state-of-the-art verification tools, CPAchecker and Ultimate Automizer, and show by experimental evaluation that the approach is applicable to a large set of verification tasks.

References

[1]

L. Alawneh and A. Hamou-Lhadj. MTF: A scalable exchange format for traces of high performance computing systems. In Proc. ICPC, pages 181–184. IEEE, 2011.

Digital Library

[2]

T. Ball and S. K. Rajamani. The Slam project: Debugging system software via static analysis. In Proc. POPL, pages 1–3. ACM, 2002.

Digital Library

[3]

T. Ball and S. K. Rajamani. SLIC: A specification language for interface checking (of C). Technical Report MSR-TR-2001-21, Microsoft Research, 2002.

[4]

D. Beyer. Status report on software verification. In Proc. TACAS, LNCS 8413, pages 373–388. Springer, 2014.

[5]

D. Beyer. Software verification and verifiable witnesses (Report on SV-COMP 2015). In Proc. TACAS, LNCS 9035, pages 401–416. Springer, 2015.

[6]

D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating tests from counterexamples. In Proc. ICSE, pages 326–335. IEEE, 2004.

Digital Library

[7]

D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. The Blast query language for software verification. In Proc. SAS, LNCS 3148, pages 2–18. Springer, 2004.

[8]

D. Beyer, A. Cimatti, A. Griggio, M. E. Keremoglu, and R. Sebastiani. Software model checking via large-block encoding. In Proc. FMCAD, pages 25–32. IEEE, 2009.

[9]

D. Beyer, T. A. Henzinger, R. Jhala, and R. Majumdar. The software model checker Blast. Int. J. Softw. Tools Technol. Transfer, 9(5-6):505–525, 2007.

Digital Library

[10]

D. Beyer, T. A. Henzinger, M. E. Keremoglu, and P. Wendler. Conditional model checking: A technique to pass information between verifiers. In Proc. FSE. ACM, 2012.

Digital Library

[11]

D. Beyer, T. A. Henzinger, R. Majumdar, and A. Rybalchenko. Path invariants. In Proc. PLDI, pages 300–309. ACM, 2007.

Digital Library

[12]

D. Beyer, T. A. Henzinger, and G. Théoduloz. Configurable software verification: Concretizing the convergence of model checking and program analysis. In Proc. CAV, LNCS 4590, pages 504–518. Springer, 2007.

Digital Library

[13]

D. Beyer, A. Holzer, M. Tautschnig, and H. Veith. Information reuse for multi-goal reachability analyses. In Proc. ESOP, LNCS 7792, pages 472–491. Springer, 2013.

Digital Library

[14]

D. Beyer and M. E. Keremoglu. CPAchecker: A tool for configurable software verification. In Proc. CAV, LNCS 6806, pages 184–190. Springer, 2011.

Digital Library

[15]

D. Beyer, M. E. Keremoglu, and P. Wendler. Predicate abstraction with adjustable-block encoding. In Proc. FMCAD, pages 189–197. FMCAD, 2010.

Digital Library

[16]

D. Beyer and S. Löwe. Explicit-state software model checking based on CEGAR and interpolation. In Proc. FASE, LNCS 7793, pages 146–162. Springer, 2013.

Digital Library

[17]

D. Beyer, S. Löwe, and P. Wendler. Benchmarking and resource measurement. In Proc. SPIN, LNCS 9232. Springer, 2015.

Digital Library

[18]

D. Beyer and P. Wendler. Reuse of verification results: Conditional model checking, precision reuse, and verification witnesses. In Proc. SPIN, LNCS 7976, pages 1–17. Springer, 2013.

[19]

U. Brandes, M. Eiglsperger, I. Herman, M. Himsolt, and M. S. Marshall. GraphML progress report. In Graph Drawing, LNCS 2265, pages 501–512. Springer, 2001.

[20]

S. Chaki, E. M. Clarke, A. Groce, S. Jha, and H. Veith. Modular verification of software components in C. IEEE Trans. Softw. Eng., 30(6):388–402, 2004.

Digital Library

[21]

E. M. Clarke, O. Grumberg, S. Jha, Y. Lu, and H. Veith. Counterexample-guided abstraction refinement for symbolic model checking. J. ACM, 50(5):752–794, 2003.

Digital Library

[22]

E. M. Clarke, T. A. Henzinger, and H. Veith. Handbook of Model Checking. Springer.

[23]

E. M. Clarke, M. Talupur, H. Veith, and D. Wang. SAT-based predicate abstraction for hardware verification. In Proc. SAT, LNCS 2919, pages 78–92. Springer, 2003.

[24]

E. M. Clarke and H. Veith. Counterexamples revisited: Principles, algorithms, applications. In Verification: Theory and Practice, Essays Dedicated to Zohar Manna on the Occasion of His 64th Birthday, LNCS 2772, pages 208–224. Springer, 2003.

[25]

H. Cleve and A. Zeller. Locating causes of program failures. In Proc. ICSE, pages 342–351. ACM, 2005.

Digital Library

[26]

L. Correnson and J. Signoles. Combining analyses for C program verification. In Proc. FMICS, LNCS 7437, pages 108–130. Springer, 2012.

[27]

P. Cousot and R. Cousot. Static determination of dynamic properties of programs. In Proc. Int. Symp. on Programming, pages 106–130. Dunod, 1976.

[28]

P. Cousot and R. Cousot. Systematic design of program-analysis frameworks. In Proc. POPL, pages 269–282. ACM, 1979.

Digital Library

[29]

D. Dams and K. S. Namjoshi. Orion: High-precision methods for static error analysis of C and C++ programs. In Proc. FMCO, LNCS 4111, pages 138–160. Springer, 2005.

Digital Library

[30]

M. Dangl, S. Löwe, and P. Wendler. CPAchecker with support for recursive programs and floating-point arithmetic. In Proc. TACAS, LNCS 9035, pages 423–425. Springer, 2015.

[31]

S. Graf and H. Sa¨ıdi. Construction of abstract state graphs with Pvs. In Proc. CAV, LNCS 1254, pages 72–83. Springer, 1997.

Digital Library

[32]

A. Groce, S. Chaki, D. Kröning, and O. Strichman. Error explanation with distance metrics. STTT, 8(3):229–247, 2006.

Digital Library

[33]

A. Groce and W. Visser. What went wrong: Explaining counterexamples. In Proc. SPIN, LNCS 2648, pages 121–135. Springer, 2003.

Digital Library

[34]

M. Heizmann, D. Dietsch, J. Leike, B. Musa, and A. Podelski. Ultimate Automizer with array interpolation. In Proc. TACAS, LNCS 9035, pages 455–457. Springer, 2015.

[35]

M. Heizmann, J. Hoenicke, and A. Podelski. Software model checking for people who love automata. In Proc. CAV, LNCS 8044, pages 36–52. Springer, 2013.

Digital Library

[36]

T. A. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Lazy abstraction. In Proc. POPL, pages 58–70. ACM, 2002.

Digital Library

[37]

A. Knüpfer, R. Brendel, H. Brunst, H. Mix, and W. E. Nagel. Introducing the open trace format (OTF). In Proc. ICCS, LNCS 3992, pages 526–533. Springer, 2006.

Digital Library

[38]

D. Kröning and N. Sharygina. Formal verification of SystemC by automatic hardware/software partitioning. In Proc. MEMOCODE, pages 101–110. IEEE, 2005.

Digital Library

[39]

A. Leitner, M. Oriol, A. Zeller, I. Ciupa, and B. Meyer. Efficient unit test-case minimization. In Proc. ASE, pages 417–420. ACM, 2007.

Digital Library

[40]

K. S. Namjoshi. Certifying model checkers. In Proc. CAV, LNCS 2102, pages 2–13. Springer, 2001.

Digital Library

[41]

H. Rocha, R. S. Barreto, L. Cordeiro, and A. D. Neto. Understanding programming bugs in ANSI-C software using bounded model checking counter-examples. In Proc. IFM, LNCS 7321, pages 128–142. Springer, 2012.

Digital Library

[42]

F. B. Schneider. Enforceable security policies. ACM Trans. Inf. Syst. Secur., 3(1):30–50, 2000.

Digital Library

[43]

O. ˇ Ser´ y. Enhanced property specification and verification in Blast. In Proc. FASE, LNCS 5503, pages 456–469. Springer, 2009.

Digital Library

[44]

A. Zeller. Isolating cause-effect chains from computer programs. In Proc. FSE, pages 1–10. ACM, 2002.

Digital Library

Cited By

Haltermann JJakobs MRichter CWehrheim H(2024)Parallel Program Analysis on Path RangesScience of Computer Programming10.1016/j.scico.2024.103154(103154)Online publication date: May-2024
https://doi.org/10.1016/j.scico.2024.103154
Haltermann JWehrheim H(2024)Exchanging information in cooperative software validationSoftware and Systems Modeling (SoSyM)10.1007/s10270-024-01155-323:3(695-719)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10270-024-01155-3
Ádám ZBeyer DChien PLee NSirrenberg N(2024)Btor2-Cert: A Certifying Hardware-Verification Framework Using Software AnalyzersTools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-031-57256-2_7(129-149)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57256-2_7
Show More Cited By

Index Terms

Witness validation and stepwise testification across software verifiers

Recommendations

Verification Witnesses
Over the last years, witness-based validation of verification results has become an established practice in software verification: An independent validator re-establishes verification results of a software verifier using verification witnesses, which are ...
Correctness witnesses: exchanging verification results between verifiers
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Standard verification tools provide a counterexample to witness a specification violation, and, since a few years, such a witness can be validated by an independent validator using an exchangeable witness format. This way, information about the ...
MetaVal: Witness Validation via Verification
Computer Aided Verification
Abstract
Witness validation is an important technique to increase trust in verification results, by making descriptions of error paths (violation witnesses) and important parts of the correctness proof (correctness witnesses) available in an exchangeable ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

August 2015

1068 pages

ISBN:9781450336758

DOI:10.1145/2786805

General Chair:
Elisabetta Di Nitto
Politecnico di Milano, Italy
,
Program Chairs:
Mark Harman
University College London, UK
,
Patrick Heymans
University of Namur, Belgium

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE'15

Sponsor:

SIGSOFT

ESEC/FSE'15: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering

August 30 - September 4, 2015

Bergamo, Italy

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

72
Total Citations
View Citations
406
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)18

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Haltermann JJakobs MRichter CWehrheim H(2024)Parallel Program Analysis on Path RangesScience of Computer Programming10.1016/j.scico.2024.103154(103154)Online publication date: May-2024
https://doi.org/10.1016/j.scico.2024.103154
Haltermann JWehrheim H(2024)Exchanging information in cooperative software validationSoftware and Systems Modeling (SoSyM)10.1007/s10270-024-01155-323:3(695-719)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10270-024-01155-3
Ádám ZBeyer DChien PLee NSirrenberg N(2024)Btor2-Cert: A Certifying Hardware-Verification Framework Using Software AnalyzersTools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-031-57256-2_7(129-149)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57256-2_7
Baier DBeyer DChien PJankola MKettl MLee NLemberger TLingsch-Rosenfeld MSpiessl MWachowitz HWendler P(2024)CPAchecker 2.3 with Strategy SelectionTools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-031-57256-2_21(359-364)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57256-2_21
Beyer D(2024)State of the Art in Software Verification and Witness Validation: SV-COMP 2024Tools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-031-57256-2_15(299-329)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57256-2_15
Saan SSchwarz MErhard JSeidl HTilscher SVojdani V(2024)Correctness Witness Validation by Abstract InterpretationVerification, Model Checking, and Abstract Interpretation10.1007/978-3-031-50524-9_4(74-97)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-50524-9_4
Silvestrim RTrigo FRocha WVieira MJunior JMendes OMenezes RCordeiro L(2023)Towards Integrity and Reliability in Embedded Systems: The Synergy of ESBMC and Arduino Integration2023 XIII Brazilian Symposium on Computing Systems Engineering (SBESC)10.1109/SBESC60926.2023.10324098(1-6)Online publication date: 21-Nov-2023
https://doi.org/10.1109/SBESC60926.2023.10324098
Beyer DLingsch-Rosenfeld MSpiessl M(2023)CEGAR-PT: A Tool for Abstraction by Program Transformation2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00215(2078-2081)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00215
Milanese MMiné A(2023)Generation of Violation Witnesses by Under-Approximating Abstract InterpretationVerification, Model Checking, and Abstract Interpretation10.1007/978-3-031-50524-9_3(50-73)Online publication date: 30-Dec-2023
https://doi.org/10.1007/978-3-031-50524-9_3
Heizmann MBarth MDietsch DFichtner LHoenicke JKlumpp DNaouar MSchindler TSchüssele FPodelski A(2023)Ultimate Automizer and the CommuHash Normal FormTools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-031-30820-8_39(577-581)Online publication date: 22-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30820-8_39
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents