In the second step of SEDE, for each RCC, we aim to automatically generate images that (goal 1) exhibit the same hazard-triggering events observed in the RCC and (goal 2) enable a learning algorithm to extract accurate rules predicting unsafe images. These rules shall ideally characterize parameter values that are observed only with unsafe images belonging to a specific RCC.
To achieve goal 1, it is sufficient to explore the input space by generating diverse images that are similar to the ones belonging to the RCC.
To achieve goal 2, instead, we need to generate both safe images and unsafe images. To be accurate, the rules derived by SEDE shall provide ranges for parameter values that are observed only or mostly with unsafe images and are as wide as possible. To this end, the unsafe images used to generate the rules must be as diverse as possible and for each unsafe image, we should identify a very similar safe image so that the algorithm learns to precisely distinguish unsafe images from safe ones, i.e., precisely learn the boundary between them.
First (Step 2.1), we generate diverse images belonging to the RCC, safe or unsafe. The goal is to cover the cluster with representative images.
Second (Step 2.2), we rely on the generated representative images to generate an additional set of unsafe and diverse images belonging to the cluster; this is achieved by generating, for each representative image, one unsafe image that is close to it and belongs to the RCC.
Third (Step 2.3), for each unsafe image derived by the second evolutionary search, we generate one safe image that is as close to it as possible.
In summary, we aim to generate a sufficient number of diverse safe and unsafe images belonging to each RCC to enable effective learning.
3.1.1 Step 2.1: Generate Representative Images.
We have two objectives: (1) generate a set of images that belong to a cluster and (2) minimize their similarity (i.e., maximize their diversity); however, since RCCs tend to contain images that are similar, the two objectives are unlikely to compete. Indeed, two dissimilar images are unlikely to belong to the same cluster. Further, for our purpose, it is useless to generate a set of diverse images if they do not belong to the RCC. Consequently, it is not desirable to rely on a multi-objective search algorithm to address them. To drive the search process, we thus need a fitness function that enables an evolutionary search algorithm to first generate images that belong to a cluster and then decrease their similarity.
In Section
2.4 we clarified why state-of-the-art approaches are inapplicable or ineffective when applied to achieve our objectives; in this section, we propose a dedicated fitness function and algorithm.
In the presence of a function that measures the distance of an individual
i from the center of the RCC
C (hereafter,
\(\mathit {RCC}_{\mathit {distance}}(C,i)\) ) and a function that measures how an individual contributes to minimizing the similarity across cluster members (hereafter,
\(F_\mathit {similarity(i)}\) ), to obtain a mathematically adequate behavior, our fitness function
\(F_1^C(i)\) can be defined as follows:
Our goal is to minimize
\(F_1^C(i)\) and given that (1)
\(F_\mathit {similarity(i)} \le 1\) (see Equation (
7) below) and (2)
\(\mathit {RCC}_{\mathit {distance}} \le 1\) is only true when the image belongs to the RCC, we obtain the targeted property: Fitness is always lower in cases where an individual
i belongs to the RCC. Below, we describe the functions
\(F_\mathit {similarity}\) and
\(\mathit {RCC}_{\mathit {distance}}(C,i)\) .
To determine if an individual belongs to a RCC, we can measure its distance from the RCC centroid and verify if it is shorter than the RCC radius (i.e., the max distance between the centroid and the farthest image). For the distance metric, as in HUDD, we can use the Euclidean distance between two heatmaps (hereafter,
\(\mathit {HeatmapDistance}\) ); however, to generate a heatmap we need a concrete image and, therefore, instead of relying on the cluster centroid, we identify its medoid. The formula for the
\(\mathit {HeatmapDistance}\) function is as follows:
where A and B are two heatmap matrices generated with LRP, and
\({m,n}\) indicate the row and column of a cell in a matrix (
M and
N indicate the total number of rows and columns, respectively). Since LRP generates one heatmap matrix for every neuron layer of the DNN, we rely on the heatmap generated for layer L as selected by HUDD to generate the RCC.
Similarly to related work [
25,
26], within our evolutionary algorithms, we represent an individual using a vector of simulator parameter values. For this reason, to compute the heatmap distance, our search algorithm, for every offspring individual, first generates one image using the simulator, then processes the generated image using the DNN under test, and finally generates its heatmap using the LRP algorithm.
The medoid of a RCC
C is the image that minimizes the average pairwise distance from the other images of the RCC; it is computed as follows:
The radius of a RCC
C is the maximum distance between its medoid and any other image in
C:
Since clusters may have different radii, we compute
\(\mathit {RCC}_{\mathit {distance}}(C,i)\) for a RCC
C as the heatmap distance between individual
i and
C’s medoid
\(\mathit {Medoid}(C)\) , normalized by
C’s radius
\(\mathit {Radius}(C)\) :
To compute
\(F_{\mathit {similarity}}(i)\) , we follow related work [
58] and measure how much an individual contributes to the diversity of the population by measuring its distance from the closest individual in the population.
We can measure the distance between two individuals
i and
j based on their chromosome vectors
\(v_i\) and
\(v_j\) (containing simulator parameter values), as follows:
where
\(\cos (i,j)\) is the cosine similarity between
\(v_i\) and
\(v_j\) , whose range is [0, 1];
\(v_i[p]\) and
\(v_i[p]\) indicate the value of the
pth component in the vectors
\(v_i\) and
\(v_j\) , respectively.
\(F_1^C\) can be used to drive the search algorithm, described below.
The SEDE search algorithm: PaiR. PaiR is our algorithm to generate representative RCC images; it is presented in Figure
8 and described below.
PaiR aims to evolve a whole population of individuals to generate a population of individuals that both belong to the RCC under analysis and are diverse. The size of the population is a parameter of the algorithm. PaiR leverages all the images generated by the simulator at every search iteration by replacing one or more images in the parent population with offspring individuals having a better fitness.
PaiR works by looking for individuals that minimize \(F_1^C\) . At every iteration, PaiR creates a new population \(P^{\prime }\) obtained by replacing an individual \(i_p\) with an individual \(i_o\) having a better fitness (i.e., \(F_1^C(i_o) \lt F_1^C(i_p)\) ), as described in the following.
The PaiR algorithm receives as input the RCC under analysis (hereafter, \(RCC_C\) ), the configuration parameters s (population size) and r (number of random populations to generate initially), and the search budget b indicating the number of iterations to perform.
To maximize the chances of finding an optimal solution, as in related work [
63], the PaiR algorithm starts by generating several random populations (Line 1 to 3) and selecting the population including the individual with the lowest fitness value (
P, Line 5).
In its main loop (Lines 6 to 21), PaiR computes the fitness values of individuals in the parent population P and, from P, generates an offspring population O by relying on the same strategy commonly adopted with NSGA-II in similar contexts (i.e., tournament selection, simulated binary crossover, crossover probability \(p_c\) , and mutation probability \(p_m\) ).
The offspring individuals are sorted to process first the ones with better fitness (Line 11). PaiR considers each individual in O ( \(i_o\) ) as a replacement of an individual in P ( \(i_p\) ), if the former has a better fitness. PaiR distinguishes between the cases in which P only contains individuals belonging to the RCC (Line 15) and when at least one individual does not (Line 13). Indeed, if we do not have enough individuals belonging to the RCC, then it is not important to maximize their diversity.
If P contains at least one individual not belonging to the RCC, PaiR simply looks for the individual in P with the highest fitness (i.e., the one that is furthest from the cluster medoid). Such individual will be replaced by \(i_o\) if the latter has a lower fitness (Line 17), which is always the case if \(i_o\) belongs to the RCC; otherwise, it depends on the value of \(RCC_{distance}\) : \(i_o\) replaces \(i_p\) if \(i_o\) is closer to the RCC medoid.
If all the individuals in
P belong to the RCC, PaiR selects the individual in
P that is closer to
\(i_o\) (i.e.,
\(i_p\) , Line 16). The offspring individual
\(i_o\) replaces
\(i_p\) if
\(i_o\) has a lower fitness than
\(i_p\) (Line 17). Since
\(i_p\) belongs to the RCC, according to Equation (
1),
\(i_o\) has a lower fitness than
\(i_p\) when
\(i_o\) belongs to the RCC and has a lower
\(\mathit {F}_{\mathit {similarity}}\) .
\(F_{\mathit {similarity}}(i)\) is computed differently for individuals in the parent and in the offspring population; the reason is that we must decide if an individual
\(i_o\) in the offspring population
O would lead to a population
\(P^{\prime }\) with a larger diversity than
P.
\(P^{\prime }\) is obtained by replacing an individual
\(i_p\) in the parent population
P with
\(i_o\) ; therefore, we obtain a different
\(P^{\prime }\) from a same
P, depending on the selected individuals
\(i_p\) and
\(i_o\) . To help identify the most diverse
\(P^{\prime }\) and avoid computing the similarity for all possible
\(P^{\prime }\) obtained from every individual in
O, inspired by related work [
58], we compare (a) the distance between
\(i_p\) and its closest neighbour in
P (i.e., the closest individual in
P excluding
\(i_p\) itself) with (b) the distance between
\(i_o\) and the closest individual in
\(P^{\prime }\) . Note that
\(P^{\prime }\) includes
\(i_o\) and
\(P \setminus \lbrace i_p\rbrace\) . Therefore, for an individual
\(i_p\) in the parent population P, we compute one minus its distance from
\(\mathit {closest}_{i_p}\) , the closest neighbour to
\(i_p\) ,
For an individual
\(i_o\) in the offspring population
O, we compute one minus its distance from
\(\mathit {closest}_{i_o}\) , the individual in
\(P \setminus \lbrace i_p\rbrace\) that is closest to
\(i_o\) ,
Based on the above, given two individuals
\(i_o\) and
\(i_p\) in the offspring and parent populations, respectively, the individual
\(i_o\) has a better fitness than
\(i_p\) if they both belong to the RCC and the similarity between
\(i_o\) and the closest individual in
\(P \setminus \lbrace i_p\rbrace\) is lower than the similarity between
\(i_p\) and its closest individual in
P. Since the distance from the closest individual should increase (lower similarity) when replacing
\(i_p\) with
\(i_o\) in the population, we can assume that a population
\(P^{\prime }\) will have higher diversity than
P, which we demonstrate in Section
4.2.
The process is repeated until O is empty (Lines 10 to 20); since \(F_1^C\) depends on the distance between individuals being close to each other, given that both O and P vary over iterations, the fitness of the two populations is recomputed after every replacement (Lines 19 and 20).
After b iterations, the algorithm has generated a population \(P_1^C\) of individuals that are diverse and very likely to belong to \(RCC_C\) .
3.1.2 Step 2.2: Generate a Set of Unsafe Images Belonging to the RCC.
For each \(RCC_C\) , we aim to generate a set of diverse, unsafe images, belonging to the cluster. To preserve the diversity of the images generated by PaiR in Step 2.1 ( \(P_1^C\) ), we can identify, for each image in \(P_1^C\) , an image that is close to it and makes the DNN fail. We can thus model our problem as a many-objective optimization problem with q objectives, where q is the number of images in \(P_1^C\) .
We define
q fitness functions, one for each individual in
\(P_1^C\) . An individual is an image generated with the simulator, modelled as described in Section
3.1.1. All fitness functions share the same parameterized formula
\(F_{2}^{C,t}\) , where
\(t \in \lbrace 1, \ldots , q\rbrace\) ,
\(F_{2}^{C,t}\) is a piecewise-defined function implemented through four sub-functions that return a value in the range [0, 1] incremented by a constant (from 0 for the first sub-function to 3 for the fourth sub-function).
The fourth sub-function of the equation drives the search toward generating an individual belonging to \(RCC_C\) ; indeed, when this is not the case, \(F_{2}^{C,t}\) measures how far the individual is from the RCC medoid with \(\mathit {RCC}_{\mathit {distance}}(\mathit {RCC}_C,i)\) . Since not belonging to \(RCC_C\) is the worst case for an individual, \(F_{2}^{C,t}\) must be higher than in other cases. This is achieved by returning \(3 + \mathit {RCC}_{\mathit {distance}}(\mathit {RCC}_C,i)\) , given that the codomain of the first three sub-functions referred to in \(F_{2}^{C,t}\) lay in the range [0, 3].
The third sub-function of the equation ensures that we generate one image for each individual in \(P_1^C\) ; indeed, if the individual i belongs to \(RCC_C\) but the individual in \(P_1^C\) that is closest to i is not the tth individual (i.e., \(CLOSEST(i,P_1^C) \ne P_1^C[t]\) ), then \(F_{2}^{C,t}\) returns the cosine distance between i and the tth individual in \(P_1^C\) , incremented by 2.
The second sub-function helps generating an individual that makes the DNN fail. Indeed, if the individual i belongs to \(RCC_C\) , is the closest to the tth individual targeted by \(F_{2}^{C,t}\) , but does not make the DNN fail, then the fitness function returns a measure of DNN uncertainty increased by one.
To compute the uncertainty component of the fitness, we return one minus the cross-entropy loss function, which is commonly used to measure uncertainty:
Cross-entropy measures how uncertain is the DNN about the provided output where lower cross-entropy values indicate higher certainty; therefore, by using one minus cross-entropy loss, we direct the search (minimization) toward the identification of images for which the DNN is less certain about outputs and, therefore, likely to produce an erroneous output.
The first sub-function of \(F_{2}^{C,t}\) , instead, for individuals that make the DNN fail, helps select the individual that is closest to the individual targeted by \(F_{2}^{C,t}\) ; indeed, it returns the chromosome distance between the individual i and the individual \(P_1^C[t]\) .
Search Algorithm. To address our problem, we rely on a modified version of NSGA-II. NSGA-II is known to underperform in the presence of a large number of objectives (
\(\gt\) 3), mainly because of the exponential growth in the number of non-dominated solutions required for approximating the Pareto front [
33]. However, we do not aim to find, as a solution to our problem, one single individual (i.e., image) that finds the best balance among different objectives but a set of images, each optimizing one independent objective (indeed, each image shall be close to a reference image in
\(P_1^C\) ); therefore, we are unlikely to find a set of nondominated solutions larger than
\(|P_1^C|\) (i.e., we will find one solution for each objective). Also, by definition,
\(F_{2}^{C,t}\) is always different than zero, which renders ineffective algorithms that look for an individual that
covers an objective like MOSA. A solution to enable the adoption of MOSA would be to set a threshold to determine when the objective has been
covered; however, since RCCs differ with respect to their radius, we choose not to introduce a threshold that may lead to varying performance results across RCCs. Since we aim to optimize all the objectives, we rely on an extended version of NSGA-II with a modified crowding-distance function that ensures we select the minimal value for each objective, thus resembling the preference criterion of MOSA. Finally, we do not require an archive, because the population will always include the best individual found for each objective and we do not need to evaluate an individual according to objectives not explicitly modeled (e.g., inputs’ length in MOSA).
We propose a modified version of NSGA-II (hereafter,
\({\it NSGA-II}^{\prime }\) ) that includes a modified crowding-distance-assignment that ensures preserving, for each objective, the individual with the lowest fitness value. Our modified crowding-distance-assignment is shown in Figure
3; different from the original crowding-distance-assignment used by NSGA-II, for each objective, we assign infinite crowding distance only to the individual that minimizes the fitness for the objective. NSGA-II, instead, assigns infinite distance also to the individual that maximizes the fitness for the objective (Line 7 in Figure
3). As in NSGA-II, if more than one individual has the same minimal fitness, then
\({\it NSGA-II}^{\prime }\) assigns infinite distance only to one randomly selected individual. Our choice ensures that, when the Pareto front includes a number of individuals larger than the population size
s, for each objective, we preserve the individual that minimizes the fitness value. When the Pareto front has a number of individuals lower than the population size
s, our crowding-distance assignment ensures the selection of individuals that minimize the fitness and are likely unsafe, which is what we intend to preserve in the final population. However, this situation is unlikely. Indeed, it may occur only when one individual has the same fitness value for several (i.e.,
z, with
\(z \le q\) ) objectives, in one of the following unlikely scenarios:
•
one image has the lowest \(\mathit {RCC}_{\mathit {distance}}\) and no other individual falls in the cases covered by the first three sub-functions in \(F_{2}^{C,t}\) , for the same z objectives;
•
one image has the lowest \(\mathit {ChromosomeDistance}(i, \hspace{1.0pt} P_1^C[t])\) for z reference images and no other individual falls in the cases covered by the first two sub-functions in \(F_{2}^{C,t}\) , for the same z objectives;
•
one image has the lowest \(F_\mathit {uncertainty}(i)\) and no other individual falls in the cases covered by the first sub-function in \(F_{2}^{C,t}\) , for the same z objectives;
•
one image is failure-inducing and has the lowest \(\mathit {ChromosomeDistance}(i,P_1^C[t])\) , for the same z objectives.
We apply \({\it NSGA-II}^{\prime }\) for k iterations. Please note that when \(P_1^C\) already includes unsafe images belonging to the RCC C, \({\it NSGA-II}^{\prime }\) simply retains such images at every iteration; indeed, their fitness is already optimal according to \(F_{2}^{C,t}\) . After the kth iteration of \({\it NSGA-II}^{\prime }\) , SEDE generates a population of individuals that likely lead to a DNN failure, which we refer to as \(P_2^C\) ; at the end of the search, images not leading to DNN failures are removed from \(P_2^C\) .
3.1.3 Step 2.3: Generate One Safe Image for Each Unsafe Image.
We aim to generate a set of images that are similar to the unsafe images in \(P_2^C\) but do not lead to a DNN failure. As for the previous case, we model our problem as a many-objective optimization problem with q objectives, where q is the number of images in \(P_2^C\) ; for each image in \(P_2^C\) , we aim to generate an image that is close to it and makes the DNN pass. We rely on the same \({\it NSGA-II}^{\prime }\) configuration used for Step 2.2 except for the fitness function, which in this step needs to be adapted to drive the generation of safe images; also, it is not necessary for these generated images to belong to \(RCC_C\) (indeed, we may not have any safe image within \(RCC_C\) ). In Step 2.3, \({\it NSGA-II}^{\prime }\) evolves a population \(P_3^C\) that initially matches \(P_2^C\) .
As for Step 2.2, we define
q fitness functions, one for each image in
\(P_2^C\) ; these functions share the same parameterized formula, for
\(t \in \lbrace 1, \ldots , q\rbrace\) :
The third sub-formula in \(F_3^{C,t}\) matches the third formula in \(F_2^{C,t}\) and aims to generate one image close to each each unsafe image in \(P_2\) .
The definition of the second sub-formula in \(F_3^{C,t}\) is motivated by the fact that, since the initial population is \(P_2^C\) , all the images in the population initially cause a DNN failure and, therefore, the fitness should drive the search algorithm to find a close image leading to a correct output. To do so, we should look for images that either increase the confidence of the DNN output (i.e., leading to a lower entropy) or are close to the border of the RCC. Concerning the latter, since a RCC characterizes unsafe images, we expect that images next to its border are similar to the ones in the RCC but are less likely to be failure inducing. In addition, moving further away from the border is expected to lead to images that are increasingly different from the images in the RCC. Such observations are captured by the absolute difference between 1 and \(RCC_{\mathit {distance}}\) (i.e., how close the image is to the RCC border), which is added to \(\mathit {entropy}(i)\) to help generate safe images.
The codomains of the second and the third sub-formula overlap as the former may return a fitness value above 3; this choice reflects the fact that an image being far away from the RCC border (when \(|1 - RCC_\mathit {distance}(C,i)| \gt 1\) ) is unlikely to provide useful information as it is not helpful to derive rules that characterize safe images that are not similar to the ones in the RCC. Therefore, such image would be as useless as an image that is not close to the target image (i.e., \(P_2^C[t]\) ). Also, if the DNN is highly uncertain about an image i (i.e., \(\mathit {entropy}(i)\) is close to 1), then it is unlikely for that image to help driving the algorithm toward a correct output. For the reasons above, the output of the second sub-formula falls into the codomain of the third sub-formula when the sum \(\hspace{1.0pt} \mathit {entropy}(i) \hspace{1.0pt} + |1-\mathit {RCC}_{\mathit {distance}}(\mathit {RCC}_C,i)|\) \(\gt\) 2.
The first sub-formula in \(F_3^{C,t}\) , in the presence of images leading to correct DNN outputs, aims to minimize the distance from the tth image; therefore, it returns the chromosome distance between the individual i and the target image \(P_2^C[t]\) .
The algorithm \({\it NSGA-II}^{\prime }\) terminates after k iterations with a population \(P_3^C\) including images that are likely safe and close to the images in \(P_2^C\) ; though unlikely, images leading to DNN failures are removed from \(P_3^C\) before SEDE’s Step 3.