-
Probabilistic Verification of Neural Networks using Branch and Bound
Authors:
David Boetius,
Stefan Leue,
Tobias Sutter
Abstract:
Probabilistic verification of neural networks is concerned with formally analysing the output distribution of a neural network under a probability distribution of the inputs. Examples of probabilistic verification include verifying the demographic parity fairness notion or quantifying the safety of a neural network. We present a new algorithm for the probabilistic verification of neural networks b…
▽ More
Probabilistic verification of neural networks is concerned with formally analysing the output distribution of a neural network under a probability distribution of the inputs. Examples of probabilistic verification include verifying the demographic parity fairness notion or quantifying the safety of a neural network. We present a new algorithm for the probabilistic verification of neural networks based on an algorithm for computing and iteratively refining lower and upper bounds on probabilities over the outputs of a neural network. By applying state-of-the-art bound propagation and branch and bound techniques from non-probabilistic neural network verification, our algorithm significantly outpaces existing probabilistic verification algorithms, reducing solving times for various benchmarks from the literature from tens of minutes to tens of seconds. Furthermore, our algorithm compares favourably even to dedicated algorithms for restricted subsets of probabilistic verification. We complement our empirical evaluation with a theoretical analysis, proving that our algorithm is sound and, under mildly restrictive conditions, also complete when using a suitable set of heuristics.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Counterexample-Guided Repair of Reinforcement Learning Systems Using Safety Critics
Authors:
David Boetius,
Stefan Leue
Abstract:
Naively trained Deep Reinforcement Learning agents may fail to satisfy vital safety constraints. To avoid costly retraining, we may desire to repair a previously trained reinforcement learning agent to obviate unsafe behaviour. We devise a counterexample-guided repair algorithm for repairing reinforcement learning systems leveraging safety critics. The algorithm jointly repairs a reinforcement lea…
▽ More
Naively trained Deep Reinforcement Learning agents may fail to satisfy vital safety constraints. To avoid costly retraining, we may desire to repair a previously trained reinforcement learning agent to obviate unsafe behaviour. We devise a counterexample-guided repair algorithm for repairing reinforcement learning systems leveraging safety critics. The algorithm jointly repairs a reinforcement learning agent and a safety critic using gradient-based constrained optimisation.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
Verifying Global Neural Network Specifications using Hyperproperties
Authors:
David Boetius,
Stefan Leue
Abstract:
Current approaches to neural network verification focus on specifications that target small regions around known input data points, such as local robustness. Thus, using these approaches, we can not obtain guarantees for inputs that are not close to known inputs. Yet, it is highly likely that a neural network will encounter such truly unseen inputs during its application. We study global specifica…
▽ More
Current approaches to neural network verification focus on specifications that target small regions around known input data points, such as local robustness. Thus, using these approaches, we can not obtain guarantees for inputs that are not close to known inputs. Yet, it is highly likely that a neural network will encounter such truly unseen inputs during its application. We study global specifications that - when satisfied - provide guarantees for all potential inputs. We introduce a hyperproperty formalism that allows for expressing global specifications such as monotonicity, Lipschitz continuity, global robustness, and dependency fairness. Our formalism enables verifying global specifications using existing neural network verification approaches by leveraging capabilities for verifying general computational graphs. Thereby, we extend the scope of guarantees that can be provided using existing methods. Recent success in verifying specific global specifications shows that attaining strong guarantees for all potential data points is feasible.
△ Less
Submitted 21 June, 2023;
originally announced June 2023.
-
A Robust Optimisation Perspective on Counterexample-Guided Repair of Neural Networks
Authors:
David Boetius,
Stefan Leue,
Tobias Sutter
Abstract:
Counterexample-guided repair aims at creating neural networks with mathematical safety guarantees, facilitating the application of neural networks in safety-critical domains. However, whether counterexample-guided repair is guaranteed to terminate remains an open question. We approach this question by showing that counterexample-guided repair can be viewed as a robust optimisation algorithm. While…
▽ More
Counterexample-guided repair aims at creating neural networks with mathematical safety guarantees, facilitating the application of neural networks in safety-critical domains. However, whether counterexample-guided repair is guaranteed to terminate remains an open question. We approach this question by showing that counterexample-guided repair can be viewed as a robust optimisation algorithm. While termination guarantees for neural network repair itself remain beyond our reach, we prove termination for more restrained machine learning models and disprove termination in a general setting. We empirically study the practical implications of our theoretical results, demonstrating the suitability of common verifiers and falsifiers for repair despite a disadvantageous theoretical result. Additionally, we use our theoretical insights to devise a novel algorithm for repairing linear regression models based on quadratic programming, surpassing existing approaches.
△ Less
Submitted 5 June, 2023; v1 submitted 26 January, 2023;
originally announced January 2023.
-
Formal Modeling and Analysis of Legal Contracts using ContractCheck
Authors:
Alan Khoja,
Martin Kölbl,
Stefan Leue,
Rüdiger Wilhelmi
Abstract:
We describe a method and tool called \textit{ContractCheck} that allows for the consistency analysis of legal contracts, in particular Sales Purchase Agreements (SPAs). The analysis relies on an encoding of the premises for the execution of the clauses of an SPA as well as the proposed consistency constraints using decidable fragments of first-order logic. Textual SPAs are first encoded in a struc…
▽ More
We describe a method and tool called \textit{ContractCheck} that allows for the consistency analysis of legal contracts, in particular Sales Purchase Agreements (SPAs). The analysis relies on an encoding of the premises for the execution of the clauses of an SPA as well as the proposed consistency constraints using decidable fragments of first-order logic. Textual SPAs are first encoded in a structured natural language format, called blocks. \textit{ContractCheck} interprets these blocks and constraints and translates them in first-oder logic assertions. It then invokes a Satisfiability Modulo Theories (SMT) solver in order to establish the executability of a considered contract by either providing a satisfying model, or by providing evidence of contradictory clauses that impede the execution of the contract. We illustrate the application of \textit{ContractCheck} and conclude by proposing directions for future research.
△ Less
Submitted 6 December, 2022;
originally announced December 2022.
-
symQV: Automated Symbolic Verification of Quantum Programs
Authors:
Fabian Bauer-Marquart,
Stefan Leue,
Christian Schilling
Abstract:
We present symQV, a symbolic execution framework for writing and verifying quantum computations in the quantum circuit model. symQV can automatically verify that a quantum program complies with a first-order specification. We formally introduce a symbolic quantum program model. This allows to encode the verification problem in an SMT formula, which can then be checked with a delta-complete decisio…
▽ More
We present symQV, a symbolic execution framework for writing and verifying quantum computations in the quantum circuit model. symQV can automatically verify that a quantum program complies with a first-order specification. We formally introduce a symbolic quantum program model. This allows to encode the verification problem in an SMT formula, which can then be checked with a delta-complete decision procedure. We also propose an abstraction technique to speed up the verification process. Experimental results show that the abstraction improves symQV's scalability by an order of magnitude to quantum programs with 24 qubits (a 2^24-dimensional state space).
△ Less
Submitted 5 December, 2022;
originally announced December 2022.
-
SpecRepair: Counter-Example Guided Safety Repair of Deep Neural Networks
Authors:
Fabian Bauer-Marquart,
David Boetius,
Stefan Leue,
Christian Schilling
Abstract:
Deep neural networks (DNNs) are increasingly applied in safety-critical domains, such as self-driving cars, unmanned aircraft, and medical diagnosis. It is of fundamental importance to certify the safety of these DNNs, i.e. that they comply with a formal safety specification. While safety certification tools exactly answer this question, they are of no help in debugging unsafe DNNs, requiring the…
▽ More
Deep neural networks (DNNs) are increasingly applied in safety-critical domains, such as self-driving cars, unmanned aircraft, and medical diagnosis. It is of fundamental importance to certify the safety of these DNNs, i.e. that they comply with a formal safety specification. While safety certification tools exactly answer this question, they are of no help in debugging unsafe DNNs, requiring the developer to iteratively verify and modify the DNN until safety is eventually achieved. Hence, a repair technique needs to be developed that can produce a safe DNN automatically. To address this need, we present SpecRepair, a tool that efficiently eliminates counter-examples from a DNN and produces a provably safe DNN without harming its classification accuracy. SpecRepair combines specification-based counter-example search and resumes training of the DNN, penalizing counter-examples and certifying the resulting DNN. We evaluate SpecRepair's effectiveness on the ACAS Xu benchmark, a DNN-based controller for unmanned aircraft, and two image classification benchmarks. The results show that SpecRepair is more successful in producing safe DNNs than comparable methods, has a shorter runtime, and produces safe DNNs while preserving their classification accuracy.
△ Less
Submitted 12 May, 2022; v1 submitted 3 June, 2021;
originally announced June 2021.
-
TarTar: A Timed Automata Repair Tool
Authors:
Martin Koelbl,
Stefan Leue,
Thomas Wies
Abstract:
We present TarTar, an automatic repair analysis tool that, given a timed diagnostic trace (TDT) obtained during the model checking of a timed automaton model, suggests possible syntactic repairs of the analyzed model. The suggested repairs include modified values for clock bounds in location invariants and transition guards, adding or removing clock resets, etc. The proposed repairs are guaranteed…
▽ More
We present TarTar, an automatic repair analysis tool that, given a timed diagnostic trace (TDT) obtained during the model checking of a timed automaton model, suggests possible syntactic repairs of the analyzed model. The suggested repairs include modified values for clock bounds in location invariants and transition guards, adding or removing clock resets, etc. The proposed repairs are guaranteed to eliminate executability of the given TDT, while preserving the overall functional behavior of the system. We give insights into the design and architecture of TarTar, and show that it can successfully repair 69% of the seeded errors in system models taken from a diverse suite of case studies.
△ Less
Submitted 12 May, 2020; v1 submitted 29 January, 2020;
originally announced February 2020.
-
Causality for General LTL-definable Properties
Authors:
Georgiana Caltais,
Sophie Linnea Guetlein,
Stefan Leue
Abstract:
In this paper we provide a notion of causality for the violation of general Linear Temporal Logic (LTL) properties. The current work is a natural extension of the previously proposed approach handling causality in the context of LTL-definable safety properties. The major difference is that now, counterexamples of general LTL properties are not merely finite traces, but infinite lasso-shaped traces…
▽ More
In this paper we provide a notion of causality for the violation of general Linear Temporal Logic (LTL) properties. The current work is a natural extension of the previously proposed approach handling causality in the context of LTL-definable safety properties. The major difference is that now, counterexamples of general LTL properties are not merely finite traces, but infinite lasso-shaped traces. We analyze such infinite counterexamples and identify the relevant ordered occurrences of causal events, obtained by unfolding the looping part of the lasso shaped counterexample sufficiently many times. The focus is on LTL properties from practical considerations: the current results are to be implemented in QuantUM, a tool for causality checking, that exploits explicit state LTL model checking.
△ Less
Submitted 2 January, 2019;
originally announced January 2019.
-
Verlässliche Software im 21. Jahrhundert
Authors:
Stefan Wagner,
Matthias Tichy,
Michael Felderer,
Stefan Leue
Abstract:
Software is the main innovation driver in many different areas, like cloud services, autonomous driving, connected medical devices, and high-frequency trading. All these areas have in common that they require high dependability. In this paper, we discuss challenges and research directions imposed by these new areas on guaranteeing the dependability. On the one hand challenges include characteristi…
▽ More
Software is the main innovation driver in many different areas, like cloud services, autonomous driving, connected medical devices, and high-frequency trading. All these areas have in common that they require high dependability. In this paper, we discuss challenges and research directions imposed by these new areas on guaranteeing the dependability. On the one hand challenges include characteristics of the systems themselves, e. g., open systems and ad-hoc structures. On the other hand, we see new aspects of dependability like behavioral traceability.
△ Less
Submitted 4 December, 2018;
originally announced December 2018.
-
Proceedings 2nd International Workshop on Causal Reasoning for Embedded and safety-critical Systems Technologies
Authors:
Alex Groce,
Stefan Leue
Abstract:
The second international CREST workshop continued the focus of the first CREST workshop: addressing approaches to causal reasoning in engineering complex embedded and safety-critical systems. Relevant approaches to causal reasoning have been (usually independently) proposed by a variety of communities: AI, concurrency, model-based diagnosis, software engineering, security engineering, and forma…
▽ More
The second international CREST workshop continued the focus of the first CREST workshop: addressing approaches to causal reasoning in engineering complex embedded and safety-critical systems. Relevant approaches to causal reasoning have been (usually independently) proposed by a variety of communities: AI, concurrency, model-based diagnosis, software engineering, security engineering, and formal methods. The goal of CREST is to bring together researchers and practitioners from these communities to exchange ideas, especially between communities, in order to advance the science of determining root cause(s) for failures of critical systems. The growing complexity of failures such as power grid blackouts, airplane crashes, security and privacy violations, and malfunctioning medical devices or automotive systems makes the goals of CREST more relevant than ever before.
△ Less
Submitted 7 October, 2017;
originally announced October 2017.
-
(De-)Composing Causality in Labeled Transition Systems
Authors:
Georgiana Caltais,
Stefan Leue,
Mohammad Reza Mousavi
Abstract:
In this paper we introduce a notion of counterfactual causality in the Halpern and Pearl sense that is compositional with respect to the interleaving of transition systems. The formal framework for reasoning on what caused the violation of a safety property is established in the context of labeled transition systems and Hennessy Milner logic. The compositionality results are devised for non-commun…
▽ More
In this paper we introduce a notion of counterfactual causality in the Halpern and Pearl sense that is compositional with respect to the interleaving of transition systems. The formal framework for reasoning on what caused the violation of a safety property is established in the context of labeled transition systems and Hennessy Milner logic. The compositionality results are devised for non-communicating systems.
△ Less
Submitted 28 August, 2016;
originally announced August 2016.
-
QuantUM: Quantitative Safety Analysis of UML Models
Authors:
Florian Leitner-Fischer,
Stefan Leue
Abstract:
When developing a safety-critical system it is essential to obtain an assessment of different design alternatives. In particular, an early safety assessment of the architectural design of a system is desirable. In spite of the plethora of available formal quantitative analysis methods it is still difficult for software and system architects to integrate these techniques into their every day work.…
▽ More
When developing a safety-critical system it is essential to obtain an assessment of different design alternatives. In particular, an early safety assessment of the architectural design of a system is desirable. In spite of the plethora of available formal quantitative analysis methods it is still difficult for software and system architects to integrate these techniques into their every day work. This is mainly due to the lack of methods that can be directly applied to architecture level models, for instance given as UML diagrams. Also, it is necessary that the description methods used do not require a profound knowledge of formal methods. Our approach bridges this gap and improves the integration of quantitative safety analysis methods into the development process. All inputs of the analysis are specified at the level of a UML model. This model is then automatically translated into the analysis model, and the results of the analysis are consequently represented on the level of the UML model. Thus the analysis model and the formal methods used during the analysis are hidden from the user. We illustrate the usefulness of our approach using an industrial strength case study.
△ Less
Submitted 6 July, 2011;
originally announced July 2011.
-
Automated Predicate Abstraction for Real-Time Models
Authors:
Bahareh Badban,
Stefan Leue,
Jan-Georg Smaus
Abstract:
We present a technique designed to automatically compute predicate abstractions for dense real-timed models represented as networks of timed automata.
We use the CIPM algorithm in our previous work which computes new invariants for timed automata control locations and prunes the model, to compute a predicate abstraction of the model. We do so by taking information regarding control locations a…
▽ More
We present a technique designed to automatically compute predicate abstractions for dense real-timed models represented as networks of timed automata.
We use the CIPM algorithm in our previous work which computes new invariants for timed automata control locations and prunes the model, to compute a predicate abstraction of the model. We do so by taking information regarding control locations and their newly computed invariants into account.
△ Less
Submitted 17 November, 2009;
originally announced November 2009.