veriFIRE: Verifying an Industrial, Learning-Based Wildfire Detection System

Abstract

In this short paper, we present our ongoing work on the veriFIRE project—a collaboration between industry and academia, aimed at using verification for increasing the reliability of a real-world, safety-critical system. The system we target is an airborne platform for wildfire detection, which incorporates two deep neural networks. We describe the system and its properties of interest, and discuss our attempts to verify the system’s consistency, i.e., its ability to continue and correctly classify a given input, even if the wildfire it describes increases in intensity. We regard this work as a step towards the incorporation of academic-oriented verification tools into real-world systems of interest.

All authors contributed equally.

Appendices

Appendices

A Background: DNNs and Their Verification

Deep Neural Networks. A deep neural network (DNN) [16] is a computational, directed graph, comprised of layers. The network computes a value, by receiving inputs and propagating them through its layers until reaching the final (output) layer. These output values can be interpreted as a classification label or as a regression value, depending on the kind of network in question. The actual computation depends on each layer’s type. For example, a node y in a rectified linear unit (ReLU) layer calculates the value $y=\text {ReLU} {}(x)=\max (0,x)$, for the value x of one of the nodes in its preceding layer. Additional layer types include weighted sum layers, as well as layers with various non-linear activations. Here, we focus on feed-forward neural networks, i.e., DNNs in which each layer is connected only to its following layer.

Fig. 2.

A toy DNN.

Figure 2 depicts a toy DNN. For input $V_1=[1, 3]^T$, the second layer computes the values $V_2=[13,-6]^T$. In the third layer, the ReLU functions are applied, producing $V_3=[13,0]^T$. Finally, the network’s single output value is $V_4=[65]$.

DNN Verification. A DNN verification engine [15, 19, 24, 36, 47] receives a DNN N, a precondition P that defines a subspace of the network’s inputs, and a postcondition Q that limits the network’s output values. The verification engine then searches for an input $x_0$ that satisfies $P(x_0) \wedge Q(N(x_0))$. If such an input exists, the engine returns SAT and a concrete input that satisfies the constraints; otherwise, it returns UNSAT, indicating that no such input exists. The postcondition Q usually encodes the negation of the desired property, and hence a SAT answer indicates that the property is violated, and that the returned $x_0$ triggers a bug. However, an UNSAT result indicates that the property holds.

For example, suppose we wish to verify that the simple DNN depicted in Fig. 2 always outputs a value strictly larger than 25; i.e., for any input $x=\langle v_1^1,v_1^2\rangle$, it holds that $N(x)=v_4^1 > 25$. This property is encoded as a verification query by choosing a precondition that does not restrict the input, i.e., $P=(true)$, and by setting a postcondition $Q=(v_4^1\le 25)$. For this verification query, a sound verification engine will return SAT, alongside a feasible counterexample such as $x=\langle 1, 0\rangle$, which produces $v_4^1=20 \le 25$, proving that the property does not hold for this DNN.

In our work, we used Marabou  [26]—a sound and complete DNN-verification engine, which has recently been used in a variety of applications [1, 2, 4, 6, 9, 13, 14, 20, 40, 41].