Test Optimization in DNN Testing: A Survey

5.1.1.1 Output-Based Methods. In 2018, Kim et al. [54] proposed the surprise adequacy (SA) criteria for testing of DNNs systems, called SADL, which is a very early work that introduced test selection for DNNs. SADL first extracts the intermediate outputs from DNNs of test data and training data as features. Then, it measures the SA according to the dissimilarity between these two features. Two measurements have been proposed in the original work, likelihood-based surprise adequacy (LSA) and distance-based surprise adequacy (DSA). LSA uses kernel density estimation to calculate the distance, while DSA uses Euclidean distance directly. Even though the authors did not state that SADL can be used to detect the faults of DNNs, they evaluated the relation between the accuracy and the SA scores of selected test inputs. This evaluation indicates that SADL can reveal the faults of DNNs. In their extension version [55], the authors proposed a variant SA, Mahalanobis Distance based Surprise Adequacy (MDSA), and evaluated SA criteria on complex models trained on a large dataset.

Another previous work DeepGini [26] was proposed by Feng et al. DeepGini is an output probability-based test prioritization technique. After obtaining the output of the test input, the Gini score is calculated by Equation (1) in Table 5. The input is more likely a fault if it has a higher Gini score. The authors demonstrated that DeepGini has a powerful ability to reveal DNN faults.

Wang et al. [106] proposed to leverage the neural coverage to rank the test inputs and find the faults. Concretely, given the unlabeled data pool, it iteratively selects a subset that has the same coverage score as the whole unlabeled set and removes the selected set from the pool. Finally, the remaining data in the pool are the ones that have different activation distributions from the whole set and are regarded as faults. Similar to [106], Yan et al. tried to use activation patterns of classes to distinguish if the output of one input is reliable [116]. Concretely, given all the training data of each class, the authors computed the lower bound and upper bound of activation values of each neuron as the pattern of this class. Then, given the test data and their predicted label, the authors compared their activation information with the activation pattern of the corresponding class. The priority score is finally calculated by the number of neurons that have greater activation values than a threshold plus the number of neurons that have smaller activation values than a threshold, divided by the total number of neurons. A greater priority score indicates that the prediction of this input is more reliable.

Guerriero et al. introduced the adaptive test selection (ATS) method DeepEST [35] for fault detection. It randomly selects one input sample first and then uses two selection methods, simple random sampling (SRS) and weight-based sampling (WBS) to add more test inputs to the selected pool. Thus, the key component of DeepEST is the WBS method. Roughly speaking, WBS assigns a weight to each input based on its distance to the labeled data divided by the total distance between all unlabeled data and labeled data. Here, the distance is defined by auxiliary variables which are the prediction confidence, distance based on SA, and the combination of confidence and SA distance. Another ATS [28] method was proposed by Gao et al. to select diverse faults from the unlabeled dataset. In this work, the output vectors are used to represent the model behaviors and measure the diversity of test inputs. ATS first projects the output vectors (top-3 maximum vectors are considered in ATS) to a space plane and then calculates the coverage of each data on this plane. After that, the difference between the coverage of a single data and the whole candidate set is utilized as an identifier to distinguish the faults and correctly predicted test data. Compared to previous works, ATS can select more diverse faults that range from different classes.

Inspired by the metamorphic testing in SE, Xie et al. proposed a diversity-guided method to detect faults [115]. The key idea is to rank metamorphic test case pairs (MPs) based on their ability to reveal violations of DNNs, Specifically, it chooses an internal layer L in the DNN model and splits this model into two parts, (1) the first part consists of layers from the input layer to L, (2) the second part consists of layers from L layer to the output layer. Then, MPs are prioritized based on the diversity of outputs from the first part. Here, the authors tried different distribution discrepancy methods to compute the output diversity, for example, Kullback-Leibler (KL) divergence. Finally, the sorted MPs are fed to the second part of the model to check the output consistency for fault detection.

Ma et al. [77] proposed to use the prediction difference between the DNN model and its subspecialized models to select fault data. Specifically, a series of models (called subspecialized models) have been trained to classify one single class. Here, each subspecialized model follows the same architecture as the original model. Then, the inputs are selected if they have high output discrimination between the original and subspecialized models. Similar to [77] and inspired by mutation testing in SE, Wei et al. proposed the Efficient Mutation Analysis for Prioritization (EffiMAP) [108] for fault detection. EffiMAP has three components, Generator, Tracer, and Estimator. Generator aims at generating fault-revealing models and input mutants. Specifically, it selects model mutants with higher killing scores and uses an autoencoder to generate input mutants. This autoencoder is iteratively updated with the generated diverse inputs. Besides, Tracer collects trace information for the given inputs. Three trace features have been considered by EffiMAP, feature map, proportion of activated neurons, and entropy of outputs. Finally, EffiMAP uses XGBoost to learn the above two types of features for fault prediction.

To further enhance the uncertainty-based methods, Li et al. proposed the distance-based dynamic random testing with prioritization (D-DRT-P) [67]. D-DRT-P first extracts features of inputs and clusters them into different subdomains. Then, D-DRT-P borrows uncertainty metrics to assign prioritization scores to each data in each subdomain. Finally, the input with the highest prioritization is selected. At the same time, if the selected input is a fault, the selection probability for its subdomain is increased. In this way, the importance of each subdomain dynamically changes and there is more chance to detect faults. Bao et al. introduced their approach Nearest Neighbor Smoothing (NNS) based test case selection method [7] by calculating the uncertainty score on the input and its neighbors. Specifically, NNS first extracted the representation of inputs using the outputs of inner layers from the model. Then, the cosine distance between each pair of inputs is computed using these representations. Inputs that are close to each other are put in the same group and their predictions are used for output probability smoothing by label smoothing technique [100]. Finally, NNS used the smoothed outputs to compute the uncertainty score for prioritization.

Tao et al. proposed TPFL [101], a test prioritization method based on fault location. Firstly, TPFL uses all the training data to collect the activation information of each neuron. Different from the classical way to set the activation threshold, TPFL sets the threshold of each neuron by the sum of its average and standard deviation of outputs. Then, TPFL indicates the suspicious neurons if they have a higher frequency activated by test inputs where the DNN makes incorrect decisions and a lower frequency activated by test inputs where the DNN makes correct decisions. Finally, if the new unlabeled data activate more suspicious neurons, they are more likely faults.

Al-Qadasi et al. proposed DeepAbstraction [4], which leverages runtime monitors to detect faults. Specifically, DeepAbstraction first extracts the features of training data to build the abstraction box. It considers the vectors extracted from the penultimate layer and the predicted classes as the features. In the abstraction box, the inputs are divided into two groups, correct inputs and incorrect inputs. Then, given new unlabeled data, DeepAbstraction checks whether their features belong to these two groups. If not, DeepAbstraction assigns these data to the third group—uncertain group. Finally, using uncertainty metrics, DeepAbstraction prioritizes the data from each group in order of incorrect, uncertain, and correct.

Different from prior works that primarily focused on computer vision tasks and feed-forward neural networks (FNNs), Liu et al. proposed the first method DeepState [71] to specifically target the fault detection problem of RNNs. DeepState extracts the predictions from the internal output state and builds a label sequence for each input. Then, DeepState defines a changing rate (CR) to measure the uncertainty of inputs based on the label changes of the collected sequence. A higher CR indicates a more uncertain input. DeepState uses the changing trend (CT) metric to help remove the redundant inputs with the same CRs. CT measures how two label sequences are different. In this way, the selected inputs are both uncertain and diverse.

Aghababaeyan et al. proposed a black-box method for fault detection [1]. Specifically, it first used VGG16 models to extract the features of mispredicted inputs from training and test sets. Then, it used Uniform Manifold Approximation and Projection (UMAP) method to reduce the dimension of collected features. After that, the hierarchical density-based spatial clustering of applications with noise (HDBSCAN) was used to cluster the faults into different groups. Finally, the authors found that, given new inputs, using inputs within one cluster to retrain the model can help fix the model with respect to that fault represented by this cluster.

To tackle more practical problems, Deng et al. proposed the Scenario-based Test Reduction and Prioritization (STRaP)[24] to detect faults in self-driving systems efficiently. Firstly, given a driving recording, STRaP converts messages in each time frame into vectors based on a driving scene schema. Here, the authors defined multiple driving scene schemas. for example, Is there any pedestrian crossing the road? 1 for yes and 0 for no. Then, STRaP slices the vectors transformed from the last step into segments based on their similarity. Finally, it prioritizes the vectors based on their coverage and the rarity of driving scene features.

Lastly, Zheng et al. proposed CertPri [122], a certifiable method for fault detection. The intuition behind CertPri is that a significant difference exists in the movement cost of correctly and incorrectly predicted test inputs. Here, the movement cost means the cost of moving the input to the class center. Interestingly, the movement cost of correctly predicted inputs is significantly higher than the cost of incorrectly predicted inputs. Thus, this cost can be used to distinguish the faults.

5.1.1.2 Learning-Based Methods. Wang et al. proposed Dissector [104] to detect unexpected conditions produced by faults. The intuition behind Dissector is that DNNs should have increasing confidence in the normal input (correctly predicted input) from the input layer to the output layer. Based on this intuition, Dissector slices the DNN model into multiple sub-models and adds an output layer to each sub-model. Then, Dissector collects the output (so-called snapshot) from each sub-model as the sequence of confidence by the original DNN model. Finally, the authors defined an SVscore to measure the snapshot validity and computed the final profile validity score of inputs by weighting all SVscores. A higher profile validity score indicates the input is more likely a normal one.

Based on mutation testing techniques, Wang et al. proposed the PRioritizing test inputs via Intelligent Mutation Analysis (PRIMA) [107] for fault detection. The intuition behind PRIMA is that the faults are near the decision boundary and, thus, more sensitive to the perturbation. Based on this intuition, the authors designed two types of mutation roles, model mutation and input mutation. Then, given input and these mutation rules, PRIMA collects the multiple features based on the killing information and leverages the learning-to-rank strategy to train a ranking model for fault prediction. Here, the famous XGBoost ranking algorithm has been used for building the ranking model.

He et al. proposed the Parallel Signal Routing Paths (PSRP) [39] to identify misclassified samples. PSRP contains three components, feature space compression, extraction of PSRP, and misclassified sample detection. In the first step, PSRP compresses the input by computing the mean value of a two-dimensional tensor produced by a convolution kernel. Then, PSRP trains an SVM model to solve a binary classification task for fault detection. The training data is the trace of compressed data from the first CNN layer to the last one.

To tackle other types of datasets, Stocco et al. proposed SelfOracle [97], an online misbehavior prediction method for self-driving cars. SelfOracle first uses an autoencoder to reconstruct the training inputs of self-driving cars. Then, it computes the mean pixel-wise squared error as the reconstruction error. After that, SelfOracle uses the maximum likelihood estimation to fit the sum of the squares of pixel-wise errors and estimates the parameters of a Gamma distribution. Finally, by setting a threshold, the Gamma distribution will be used to predict the probability of misbehavior of the unseen inputs. As stated by the authors, SelfOracle can be used for online misbehavior prediction and time-aware anomaly score prediction. Dang et al. proposed the GNN-oriented Test Prioritization (GraphPrior) [18], a fault detection method specifically designed for graph data. GraphPrior defined 10 rules for mutating GNN models, for example, adding self-loops to the nodes. After the model mutation, inputs are prioritized based on their killing score on this mutant, which means the input is more likely a fault if mutants have higher disagreements on the input.

5.1.2.1 Output-Based Methods. In 2017, Dan et al. introduced the well-known baseline for fault detection, maximum softmax probabilities based detection [41]. This work showed that it is promising to detect out-of-distribution data and misclassified faults by simply using the maximum softmax probabilities of outputs.

To enhance uncertainty-based methods, Malinin et al. proposed the Dirichlet Prior Networks (DPNs) [79] to model the predictive uncertainty of DNNs. Simply speaking, the DPN model directly parameterized the Dirichlet distribution as a prior for predicting the classification distribution on the probability simplex. The most important part is the loss function of DPNs which was defined as the KL divergence between the model and a sharp Dirichlet distribution on the appropriate class for in-distribution data, plus the model and a flat Dirichlet distribution for out-of-distribution data. Lastly, the existing uncertainty measurements, max probability and entropy were used to detect the faults based on the output of DPNs. Importantly, this work summarized the different types of uncertainty, model uncertainty, data uncertainty and distributional uncertainty which were confused by previous works.

Similar to mutation-based methods from SE [107], Chen et al. proposed a framework to use an ensemble of models to identify faults [10]. Specifically, the disagreement between the original model and the majority voting of the ensemble model was used as the indicator for fault detection. That means if the ensemble disagrees with the original model, the input has a higher ability to be a fault. Most importantly, to improve the performance of fault detection, the authors built the ensemble models iteratively and added the selected faults at each iteration to the training data, and then performed self-training in the ensemble model.

Lust et al. proposed a Gradient’s Norm (GraN) based method [73] to detect faults. GraN contains three steps, input pre-processing, feature extraction, and feature processing. In the first step, given an input image, GraN utilized image smoothing techniques to generate a new input. Then, GraN fed the smoothed image with the predicted label of the original image into the DNN model to calculate the gradients via backpropagation. In the last step, a logistic regression network was used to learn the connection between the gradient features extracted from the last step and the correctness of the inputs. The output of the regression model indicates the likelihood of the correctness of inputs.

Instead of directly using the prediction score to detect faults, Jiang et al. proposed the trust score [51] to identify the correctness of unlabeled test data. To do so, firstly, the authors removed the data that has a low density from the training set for each class. Here, any data representation and distance methods can be used for this process. After that, the trust score was defined to measure the reliability of inputs. Given an input, Trust scorewas calculated by the ratio between its distance from this input to the nearest class different from the predicted class and the distance to its predicted class. A lower trust score indicates that this input is more likely a fault.

5.1.2.2 Learning-Based Methods. Aigrain et al. proposed Introspection-Net [3] to use a 3-layer regression NN to predict the correctness of inputs. Introspection-Net is a binary classification model that takes the output logits from the original DNNs as inputs and produces a confidence value for the inputs that is, whether the classification is correct (output value of (1) or not (output value of 0). The experiments showed that Introspection-Net accompanied by adversarial training and data augmentation has a competitive fault detection ability with the Trust Score approach [51] and Softmax Baseline [41].

Li et al. proposed TestRank [66] to rank test inputs based on their likelihood of being a failure. Specifically, TestRank extracted two types of features from inputs, the output from the logits layer and the graph information that represents the distance of this input (here, cosine distance was used for the computation) to others. After that, a GNN model was used to learn the graph information and predict the learned contextual attributes of the data. Finally, given the correctness label (e.g., 0 for incorrect, 1 for correct) of inputs, TestRank utilized a simple binary classification model to learn the output information and the contextual attributes and predict the correctness of inputs.

Granese et al. proposed DOCTOR [32], which defined two types of discriminators to determine whether the input is a fault or not. The first one is based on the sum of squared softmax probabilities, and the second one is computed by the predicted model for the posterior class probability. Interestingly, the authors considered two scenarios for the evaluation, Totally Black Box (TBB) where only the predictions are available and Partially Black Box (PBB) where gradient information is allowed. In the PBB situation, the authors found adding a small perturbation brought advantages for fault detection.

Sensoy et al. proposed the risk-calibrated evidential classifiers [93] whose outputs are more sensitive to the faults. The purpose of such classifiers is to force the faults to be biased toward less risky categories to increase the fault detection rate. To do so, firstly, the authors utilized the Dirichlet Distributions to reform the outputs and therefore, to quantify the uncertainty of inputs against DNNs. After that, an evidential deep classifier was trained where the activation function of the classifier was changed to the exponential function for predicting the Dirichlet distribution for each sample. Finally, uncertainty methods were used on the evidential deep classifier for fault detection.

Qiu et al. proposed the Residual-based Error Detection (RED) [89] to enhance error fault score based on the original maximum class probability. Specifically, RED first assigned a target detection score to each training input according to whether it is correctly classified by the base model, that is, 1 for correct and 0 for incorrect. Then, it computed the residual between the target score and the maximum class probability produced by the original model. After that, a Residual prediction with an Input/Output kernel (RIO) model is trained to predict this residual. Finally, when new inputs come, RED combines the score produced by the RIO model and the output probability produced by the original for fault detection.

Zhu et al. [126] conducted a study and found that adding out-of-distribution (OOD) detection methods, for example, Outlier Exposure, to the training process, harms the performance of output-based fault detection. Then, based on this finding, the authors proposed a method called OpenMix to help the fault detection. Specifically, OpenMix changed the OE loss in Outlier Exposure to in/OOD (ID) Mixup-based loss to increase the exposure of low-density regions. In this way, the OOD detection methods have both good OOD detection ability and fault detection ability.

5.1.3.1 Output-Based Methods. Aghababaeyan et al. introduced DeepGD [2], a search-based test selection method for detecting faults. Specifically, DeepGD considered both uncertainty scores and diversity scores of inputs to conduct test prioritization. Gini [26] score is used for uncertainty calculation and geometric diversity on the features extracted from VGG models is applied for diversity calculation. An NSGA-II algorithm is used for optimization uncertainty and diversity to assign a final score to each input for prioritization.

Yang et al. proposed PROPHET [117] to detect faults in automated speech recognition (ASR) systems. In this work, the authors introduced a new fine-grained word-level error metric for evaluating the correctness of audio-to-text tasks. By comparing the reference text and predicted text, PROPHET labeled the token as 0 (1)if it is correctly (incorrectly) predicted. Then, it trained a BERT model to predict word errors. Given the new coming data, after feeding them to the trained BERT model, the ones that have higher word errors have higher probabilities be faults.

Lee et al. proposed a neuron activation similarity-based sample selection method [63] for fault detection. The method first collected the neuron activation information for each class from the training data and then checked the activation of the test data. If the difference between the similarities of training and test data is less than the threshold, these inputs are considered to be faults.

Wu et al. propose RNNtcs [112], a test prioritization method for RNNs. Given the unlabeled test data, RNNtcs first extracted the outputs of the test inputs and utilized HDBSCAN to cluster the outputs into different groups. Then, it utilized uncertainty methods (Least confidence was used in RNNtcs) to compute uncertainty scores for outliers identified by HDBSCAN and prioritized them accordingly. If the number of outliers is less than the labeling budgets, the uncertainty scores of other inputs in each cluster (normal data) were also calculated and prioritized. After that, the hidden state CR of RNN models was computed as the second uncertainty measurement. Finally, the budget number of inputs with higher hidden state CRs was selected from the prioritized sets in the order of outliers and normal data.

5.1.3.2 Learning-Based Methods. Chen et al. proposed ActGraph [9], an activation graph-based test prioritization method. ActGraph first extracted the activation value from the last K layers and built an adjacency matrix. Then, a GNN model is used to aggregate the activation features, and an aggregation function is used to obtain the center node feature. Here, ActGraph used \(Sum\left(\right)\) as the aggregation function. Finally, a learning-to-rank algorithm is applied to build a ranking model for test prioritization. XGBoost is applied in ActGraph as the ranking model.

The very first empirical study was conducted by Byun et al. [8]. In this work, the authors explored the effectiveness of three fault detection methods, Entropy (method 2 in Table 5), uncertainty in Bayesian neural networks (method 4 in Table 5), and DSA [54]. The key finding from this empirical study is that when the DNN under test has a higher test accuracy, these methods are more effective in detecting faults. Besides, Mosin et al. compared SA, autoencoder-based, and similarity-based fault detection methods [85]. They found that SA has the most effective fault detection ability among the considered methods. However, the similarity-based method is the most efficient method which is more than 1,000 times faster than the SA method.

Ma et al. evaluated the fault detection ability of 12 methods [76]. In their work, they divided existing fault detection methods into two groups, methods from ML testing literature including coverage-guided methods, SA-based methods, and so on, and model uncertainty-based methods, such as maximum probability (method 5 in Table 5). Importantly, the authors suggested using Silhouette coefficient [92] to detect faults and also defined two new methods, Variance, and Weighted Variance (methods 6 and 7 in Table 5). They found that uncertainty-based methods have stronger fault detection ability than SA and neuron coverage-based methods. Shi et al. also empirically studied the effectiveness of test case prioritization metrics (fault detection methods) [95]. In total, 11 fault detection methods have been considered by the authors including SA methods and uncertainty-based methods. Different from the previous work, this work suggested using model mutants to replace dropout prediction for fault detection and introduced three new methods called Variation Ratio (VR), Variation Ratio for Original (VRO), and Mutual Information (MI), which are listed as methods 9, 10, and 11 in Table 5. In addition to analyzing the effectiveness of existing methods, this study also explored the impact of the test suite size, mutation operators, and the number of mutants.

More recently, Weiss et al. [111] empirically showed that simple test prioritization methods perform better than SA and neuron coverage methods in terms of fault detection. Here, simple methods indicate those methods only rely on the model outputs, for example, using the maximum probability to detect faults directly (method 5 in Table 5).

Vazhentsev et al. specifically studied the existing fault detection methods on natural language processing [103]. Besides, two novel methods have been proposed in this work, which are based on the Diverse Determinantal Point Process (DDPP) Monte Carlo Dropout. DDPP sampled (and masked) a subset of neurons from the dropout layer where the activation values of this subset are similar to the activation values of the whole set. The authors improved the diversity of the sampled DPP masks by two methods, the first one, determinantal point process (DPP) sampled a set of “diverse” masks that activate different sets of neurons by using the Radial basis function (RBF)-similarity matrix of mask vectors. The second one selected the masks that yield the highest Probability variance (PV) scores on the given OOD dataset. This study showed that methods based on the Mahalanobis distance and spectral normalization of a weight matrix achieved the best results in NLP, and the proposed two methods have competitive results with the best ones.

Zhu et al. [125] conducted an empirical study to show that confidence calibration methods are useless or harmful for failure prediction. They studied five calibration methods method, for example, mixup, and found that after adding these methods, the existing fault detection methods have poorer detection performance. Inspired by this finding, the authors proposed to use flat minima methods for enhancing the existing fault detection methods. Two methods have been considered in this work stochastic weight averaging and sharpness-aware minimization. The experiments demonstrated that flat minima are useful for improving the detection ability of existing methods.

Hu et al. [46] empirically explored the robustness of fault detection methods. Here, robustness means how good the methods are in handling uncommon test inputs. The authors designed two types of uncommon inputs that the benchmark datasets (e.g., MNIST) do not have but could appear in the wild. The first type of test is high uncertainty but correctly predicted data, and the second one is low uncertainty but wrongly predicted data. Extensive experiments demonstrated that existing fault detection methods cannot distinguish faults and the first type of test data, and cannot detect the second type of fault.

Table 6 lists the evaluation metrics used for fault detection. We can see different works used different metrics and there is no uniform one. The most used metric is receiver operating characteristic, that is, 11 works used AUC-ROC for the evaluation.

Table 7 presents the types of data used for evaluating fault detection methods from existing works. Most works (27 out of 46) only tried to detect faults in the test set (Original test set) that split from the original dataset. Works [7, 9, 101, 107, 122] conducted relatively more comprehensive evaluations, which considered faults in original tests, adversarial examples, and synthetic distribution shifted datasets.

Although many works take an effort to solve fault detection problems, there are some pitfalls revealed in existing works. We analyze such pitfalls from three perspectives, datasets, evaluation metrics, and robustness issues.

We can see both SE (26 papers) and ML (13 papers) communities contributed to the study of the problem of fault detection in DNNs. It is worth comparing the research focuses of these two different communities.

The first work is MODE which proposed a differential analysis-based input selection method for debugging (via retraining) DNNs by Ma et al. [75]. MODE contains three main components. First, it selects the intermediate layer in the DNN that is important for causing under-fitting or over-fitting problems. Then, MODE builds a feature model by adding one output layer after the selected layer and uses this model to produce heat maps of benign and faulty data. Finally, MODE selects new data based on the heat maps and the characteristics of under-fitting and over-fitting, for example, for under-fitting bugs, it selects more faulty data from under-fitted classes. The authors have shown MODE has a strong ability to fix bugs in pre-trained DNNs.

Shen et al. proposed the Multiple-Boundary Clustering and Prioritization (MCP) [94] to enhance the DNN retraining. The basic idea of MCP is to select data near the decision boundary and control the diversity of the selected set. Specifically, MCP first computes the output probabilities of all test inputs. Then, MCP divides all the data into different clusters according to their top-2 predicted classes. Besides, for each data, MCP computes its priority in its belonging cluster as the ratio of the probability of the first class to the probability of the second class. Finally, MCP evenly selects inputs with high priorities from each cluster to form the final set as the output. Since MCP considers top-2 classes, the selected inputs are close to multiple (2) boundaries.

Different from other works that mainly focused on the clean accuracy of DNN models, Wang et al. introduced Robustness-Oriented Testing (RobOT) [105] to enhance the robustness of DNNs by retraining. This article designed two robustness-oriented testing metrics to guide the test selection process, zero-order loss (ZOL) and first-order loss (FOL). ZOL is calculated by the loss of the input directly, and FOL is based on the gradient of the input against the DNN model. The authors demonstrated that there is a strong correlation between the robustness of the model and the FOL score, and showed FOL is good guidance for selecting data to improve the robustness of DNNs. Besides, in the extension work of RobOT [11], the authors added both tabular and image domain data in the experiments and explored the usefulness of using RobOT to improve the fairness of DNNs.

In order to provide guidance for users to better use sampling-based retraining methods, Hu et al. [44] first empirically studied how existing methods perform on datasets with distribution shifts. They found that no method can constantly outperform others across all ranges of distribution shifts. When the candidate set contains more than 70% distribution shifted data, random selection performs better than other well-designed methods. To tackle the distribution shift problem, the authors proposed a distribution-aware test (DAT) selection. DAT has two key steps, firstly, it splits the candidate set into in-distribution data and OOD data based on OOD detectors. The DAT selects data that the model has low confidence in from the in-distribution set and selects diverse (diverse in terms of the predicted labels)data from the OOD set.

Li et al. proposed HybridRepair [65], a method smartly combining semi-supervised learning and sampling-based retraining to improve the performance of DNNs. HybridRepair has two main steps, (1) it uses data augmentation methods to generate data for each input and then average their predictions. After that, the averaged predictions will be used as a groud-ture to calculate the loss for each data. Finally, semi-supervised learning is performed to train the model. In this step, the authors also defined a local entropy metric to only select the less uncertain data for semi-supervised learning. (2) The second step tends to use fully-surprised learning on a selected subset. HybridRepair uses a feature extractor to obtain the important features of inputs and leverages these features to conduct clustering-based sample selection. In this step, the local entropy is also used for guiding the data selection (select data with higher local entropy).

Attaoui et al. proposed the Safety Analysis based on Feature Extraction (SAFE) [5] to retrain and improve DNNs. SAFE utilizes pre-trained image models to extract the features from existing faults first. Then it uses the DBSCAN algorithm to cluster the extracted features into different fault groups. After that, when new data come, engineers extract their features, and then select and label the data that are close to the fault clusters. Finally, the selected data are used to retrain the DNN models.

Hao et al. proposed the Multiple-Objective Optimization-Based Test Input Selection (MOTS) [38] to consider both uncertainty and diversity in the data selection process. For the uncertainty measurement, MOTS considers the entropy score of output probabilities. On the other hand, the Euclidean distance between two inputs is used for the diversity score. A multiple-objective optimization algorithm NSGA-II is used to store the Pareto optimal solution. Finally, MOTS selects the samples based on the aforementioned solution.

Guo et al. proposed the density-based robust sampling with entropy (DRE) [37]. DRE calculated the entropy score from the predicted probabilities of each data first. Then, it split the data space into multiple intervals and mapped each input to the interval according to its entropy score. Finally, DRE randomly selected inputs from each interval. The number of selected inputs from each interval was determined by the probability density function.

Empirical studies on sampling-based model retraining mainly focus on SA. Kim et al. showed how surprise SA performed on the industrial-level datasets [56]. They took the self-driving car as the use case and explored the effectiveness of SA in guiding DNN retraining. The study results demonstrated that SA is a promising metric for producing DNN models with good performance while reducing the labeling effort. Weiss et al. refined the well-known DNN testing metric, SA to reduce its computation cost [110]. Concretely, they proposed three methods to reduce the size of training data and then build the activation trace, The first one is uniform sampling which randomly selects a part of training data. The second one is unsurprising-first sampling which selects the data with the lowest surprise from each class. The last one is neighbor-free sampling. The authors suggested removing the redundant data if the distance between their activation traces to any activation trace of an already selected one is smaller than a threshold. Here, the considered data should be in the same class. The authors demonstrated that their refined SA metrics have similar effectiveness to the original one while significantly reducing computation costs.

Most of the works evaluated the retraining methods by comparing the accuracy of DNNs before and after retraining. Three works [11, 37, 105] considered and evaluated another property of DNNs – adversarial robustness. Adversarial robustness here means the ability of DNNs to correctly predict adversarial examples.

Different from the fault detection task one DNN model and a test set are enough for the evaluation, retraining must be conducted on a DNN model and three parts of data, (1) candidate data that is used for data selection, (2) data that are used to retrain the model, and (3) test data that used to evaluate the performance of the retrained model. Therefore, there could be different settings for these three groups of data. Table 9 lists the types of data that were used for the retraining process from existing works. Here, independent tests in column Test data means that the candidate data and test data are different. We can summarize that existing works mainly lie in five modes as follows. And most of the works (43%) chose to combine the training data and selected test data for retraining, and then evaluate the model using another independent test set.

Additionally, we noticed that similar to the fault detection works, all the sampling-based retraining methods from the SE community only considered datasets and models that were originally studied by the ML community. SE tasks were ignored by the SE community.

Model selection works mainly utilize correlation analysis metrics to measure the correctness between the idea performance ranking and the estimated performance ranking. Three metrics have been used in existing works, Spearman’s rank correlation coefficient, Jaccard similarity coefficient, and Kendall’s rank correlation.

Given n samples and \(s_1, s_2, \ldots , s_n\) , let \(r\left(s_1\right), r\left(s_2\right), \ldots , r\left(s_n\right)\) be the ground truth ranking and \(r^{\prime }\left(s_1\right), r^{\prime }\left(s_2\right), \ldots , r^{\prime }\left(s_n\right)\) be the estimated ranking.

Spearman’s rank correlation coefficient. (Used in [47, 84, 98])

Jaccard similarity coefficient. (Used in [47, 84] to measure the correctness of top-k rankings.)

Kendall’s rank correlation. (Used in [47, 98])where P and Q are the numbers of ordered and disordered pairs in \(\left\lbrace r\left(s_i\right), r^{\prime }\left(s_i\right)\right\rbrace\) , respectively. T and U are the numbers of ties in \(\left\lbrace r\left(s_i\right)\right\rbrace\) and \(\left\lbrace r^{\prime }\left(s_i\right)\right\rbrace\) , respectively.

Pitfalls. No comparison with performance estimation methods. The existing works have not clearly explained when we need to use model selection methods and when we need to use labeling-free performance estimation methods. The fact is that we can use performance estimation methods to assign performance to every single model and then rank them accordingly. However, since these model selection words have not conducted comparison experiments between their proposed methods with labeling-free performance estimation methods, it is doubtful whether these proposed model ranking methods are not necessarily needed.

Good Practice. When proposing new model selection methods, it is important to clarify why performance estimation methods cannot work in the studied scenario or conduct experiments to compare the proposed methods with state-of-the-art performance estimation methods to show their advantages. Besides, instead of using the aforementioned three evaluation metrics in Section 7.2.1, other recently proposed ranking evaluation metrics, such as normalized discounted cumulative gain (NDCG) [50] can be also considered in the experiments.

8.1.1.1 Sampling-Based Methods. Li et al. [69] first introduced the concept of efficient DNN testing to the SE community and proposed to select a subset of test data to assess the performance of DNNs based on the conditioning of the learned representation. It minimizes the distance between the whole test set and the selected set. Here, the distance is the probability distribution of the neuron outputs in the last hidden layer of the DNN models. The authors introduced two ways to measure the distribution, (1) output confidence-based distribution called CSS, and (2) cross entropy-based distribution called CES. The authors demonstrated that CES is more stable in different evaluation situations. Following the first work, Chen et al. introduced a clustering-based approach for model performance estimation, Practical ACcuracy Estimation (PACE) [12]. PACE first clusters all the test inputs into different groups using the HDBSCAN algorithm. Then, it utilizes the MMD-critic algorithm to select representative inputs from each cluster. Besides, PACE uses adaptive random exploration to collect inputs from the data that do not belong to any cluster and adds it to the already selected data. This step is useful to increase the diversity of the finally collected data. Importantly, the authors studied the impact of different features (which is an important factor for clustering) used to conduct the clustering.

Zhou et al. proposed a two-phase approach DeepReduce [124], to select test inputs and estimate the performance of DNNs accordingly. Firstly, DeepReduce uses a famous algorithm HGS to select a subset of test inputs that have the same neuron coverage as the whole inputs. Secondly, DeepReduce iteratively selects inputs to minimize the difference between the output distribution of selected data and the output distribution of the whole input. Here, the authors split the output of neurons in the last layer into K sections, and then use the percentage of data located in different sections as the output distribution. Besides, the KL Divergence is used to compute the difference in distribution. Finally, if the KL distance is smaller than a user-defined threshold, DeepReduce outputs the selected data.

8.1.1.2 Labeling-Free Methods. The only labeling-free performance estimation method in SE was Aries proposed by Hu et al. [48]. Aries is based on the distribution analysis of training and test data. Given the training data, Aries first uses dropout prediction to collect multiple outputs of each input. Then, it computes the distance of input to the decision boundaries based on the change rate of predicted labels across these multiple predictions. That means, if the change rate is higher, the input is closer to decision boundaries. After that, Aries splits the data space into multiple sections regarding the distance and computes the accuracy of each section using the data mapped in each section. Finally, given the new unlabeled test data, Aries maps them into each section and uses the corresponding accuracy to estimate the final accuracy of the set.

8.1.2.1 Sampling-Based Methods. Inspired by the problem of active learning, Kossen et al. introduced the novel testing framework, Active testing [60], which aims to sample test inputs to estimate the loss of the whole set of inputs to reduce the labeling effort. This is the first work that explained the importance of active testing and the difference between active testing and active learning. In this work, the authors proposed a surrogate model-guided acquisition strategy for test selection. This is based on the Monte Carlo estimator and acquisition distribution q. Here, q was defined differently for different tasks. For example, for classification tasks, \(q=1-\pi \left(y = y^*\left(x\right)|x\right)\) , where \(y^*\left(x\right)\) is the index of maximum probability and \(\pi\) is the surrogate model. Besides, to increase the estimation precision, the surrogate model is iteratively updated using the labeled test inputs by retraining. Following their previous work, Kossen et al. proposed the Active Surrogate Estimators (ASEs) [61], that actively learn a surrogate model and use this surrogate to select inputs to estimate the loss of the original model. This basic idea is similar to active testing [60], a new acquisition strategy, called Expected Weighted Disagreement (XWED), was proposed to select inputs. Simply speaking, XWED is a modified BALD [42]. It assigned a weight for each data point using its corresponding loss value from the original model and then applied BALD to select the inputs. Finally, the selected inputs were labeled and used for computing the total loss of the DNN model.

8.1.2.2 Labeling-Free Methods. Jiang et al. proposed the first work to use the correlation between the margin distribution of hidden layers and the model performance to predict the generalization gap [52]. Here, the margin distribution means the distance from the data to decision boundaries. The authors defined the decision boundary for each class pair (i, j) as \(D_{\left(i, j\right)} = \lbrace {x|p_{i}\left(x\right) = p_{j}\left(x\right)}\rbrace\) and approximated it as follows:where \(x^l\) means the representation of x from the lth later. Finally, the performance estimation was conducted by using a linear regression model to learn the relation between the distance and the performance. Similar to [52], DeChant et al. [19] also used the output of intermediate layers for performance estimation. The difference is this work trained a meta-model to predict accuracy. Specifically, it collected the intermediate and final outputs first. Then these outputs were labeled as correct or incorrect based on the correctness of their corresponding inputs. Finally, a binary classification model was trained using the outputs and labels. Given the new unlabeled inputs, their intermediate can be used to predict their correctness according to this trained binary classifier.

Different from [19] that relied on meta-model, Chuang et al. proposed to use domain-invariant representations (DIR) models to learn a latent, joint embedding of source and target data, and a predictor from the latent space to the output labels for target risk prediction [14]. A \(F_{G\Delta G}\) -divergence was defined and used for measuring the difference between two representation domains. Besides, in this work, the authors demonstrated that for DNNs, the selection of the intermediate layer to divide the model into encoder (which affects the complexity of produced embeddings) and predictor highly affects the trained DIR models for risk prediction.

Corneanu et al. proposed to use persistent topology measures to estimate the testing error of DNNs on unseen samples [16]. Specifically, firstly, given the DNN model and inputs, the correlations between each pair of neurons were computed by:where \(a_i\) is the activation value of a neuron, \(\overline{a}\) and \(S_{a}\) are the mean and standard deviation of all activate values. After that, a persistent diagram was computed based on the activation values using Vietoris-Rips filtration and persistent homology methods. This persistent diagram indicates the sequence status of birth and death of neurons. Then, topological summaries (average time and average density) of the persistent diagram were used to build a linear function to predict the performance gap.

Instead of training learners for performance estimation, Guillory et al. proposed a simple method that uses the difference of confidence (DoC) to estimate the performance of classifiers [36]. Here, the DoC was computed by the difference between the average maximum probabilities of training data and the average maximum probabilities of test data. Besides, the authors introduce a variant of DoC, the difference of average entropy (DoE) which uses the entropy of probabilities to replace the maximum probabilities in DoC. After getting the DoC or DoE, a linear regression model was used to learn the relation between DoC (or DoE) and the accuracy of models for performance estimation. Garg et al. also proposed a very simple method Average Thresholded Confidence (ATC) [29] to detect faults. ATC first utilized the validation set to find a threshold that the fraction of the confidence of inputs above the threshold matches the validation set accuracy. After that, given the enabled test data, ATC calculated such fraction based on the determined threshold and estimated the performance of DNN on this set accordingly.

Deng et al. introduced a new model evaluation paradigm—Automatic model Evaluation (AutoEval) [22]. AutoEval tends to assess the quality of DNN models without using the labeled data. To do so, the authors proposed a dataset-level regression method. Specifically, given a DNN model, and a training dataset, this method first generated a large number of metasets based on a randomly sampled seed set using image synthesis techniques. After that, the Frechet distance between the representation of the original training set and the met sets is calculated. Here, the representation is the mean and covariance of the image feature vectors. Finally, a regression model or NN model was used to learn the relation between the distance and model performance for further performance estimation. Following this work, the authors found that there is a strong linear relationship between semantic classification accuracy and the accuracy of the rotation prediction task which can be used for performance estimation [20]. Based on this finding, given a DNN model, the authors added an auxiliary rotation prediction into the model, in turn, building a multi-task model. Since the rotation prediction is an unsupervised task, when facing new unlabeled inputs, the model can predict their rotation head directly. Then, the linear relationship between the two accuracies can be used to predict the semantic prediction accuracy of DNNs.

Baek et al. discovered a phenomenon, Agreement-on-the-Line [6], which means ID vs. OOD agreement for pairs of DNNs has a linear correlation when the corresponding ID vs. OOD accuracy also lies on a line. Based on this phenomenon, the authors proposed two methods ALine-D and ALine-S to estimate the performance of DNNs on new unlabeled data. More concretely, the authors first conducted a large-scale empirical study and revealed that When ID vs. OOD accuracy observes a strong linear correlation ( \(\ge\) 0.95 R2 values), we see that ID vs. OOD agreement is also strongly linearly correlated with the similar slope and bias. Then, based on the finding, ALine-S and ALine-D have been proposed. ALine-S simply used the slope and bias from disagreement to estimate the accuracy of unlabeled data, while ALineD built a matrix of the combination of slope and bias and found the best solution for performance estimation.

Li et al. considered the label-imbalanced situation where the existing performance estimation methods could produce unexpected results [68]. To tackle this label-imbalanced issue, the authors plugged a class-specific temperature scaling (TS) technique into existing methods to improve their effectiveness. Specifically, the TS technique rescaled the output probabilities using the temperature parameter to fit the distribution of classes.

Unlike other works, Guan et al. brought the performance estimation to the instance segmentation problem in self-driving cars [34]. A distance-based method has been proposed in this work. Concretely, it extracted the representation vectors of ID and OOD inputs by instance segmentation detectors first, and then computed the Fréchet distance (FD) between these two representations. Here, the first-order mean and second-order covariance matrix of the representation vectors has been used for the distance calculation. Finally, regression models were used to learn the relation between the distance gap and the accuracy for further performance estimation.

Finally, Kleyko et al. proposed to use Perceptron theory to predict the performance of DNNs [57]. In the perceptron theory, the mean and standard deviation of the last hidden layer can be used to predict the accuracy of DNN models. Therefore, the authors train regression models to estimate the performance based on the last hidden layer outputs.

8.1.3.1 Sampling-Based Methods. Sun et al. proposed a three-step clustering-based method to estimate the model performance [99]. In this first step, the histograms of marginal distributions have been recorded to represent the input data as \(h_{shape}\) , that is, the matrix contains the number of values in each range of each feature dimension. After that, clustering methods were applied to divide the inputs into K groups, where K is the number of classes. The cluster centers will be selected as the first part of the labeled data as \(h_{cluster}\) . Then, the farthest points sampling (FPS) method was applied to select diverse data samples from the center to the marginal as \(h_{sample}\) . Finally, the combined features [ \(h_{shape}\) , \(h_{cluster}\) , and \(h_{sample}\) ] were used to train a regression model for performance estimation.

8.1.3.2 Labeling-Free Methods. Martin et al. found that there is a clear correlation between the weight matrix and the accuracy of DNNs [81] that can be used for performance estimation. Specifically, the authors empirically studied the linear correlation between the norm-based capacity control metrics and the power law-based metrics from the Theory of Heavy-Tailed self-regularization with the model performance. By using this correlation, it is able to estimate the accuracy of any DNN model. Following the same intuition, Unterthiner etal. directly used the weights of DNNs to predict their performance [102]. They empirically showed the correlation between the model parameters (flattened weights, weight statistics, and weight norms) and the model performance, and found that weight statistics are the best feature to build this correlation. Besides, they studied three types of estimators for learning such correlations, logit-linear (L-Linear) model, gradient boosting machine (GBM) using regression trees, and a fully connected DNN, and found DNN is the best option. Most importantly, the authors released a large dataset with 120k CNNS to support the research of performance estimation. Besides, Deng et al. found that there is a strong correlation between the dispersity of outputs (i.e., label-balance, the entropy of predicted probabilities was used in the work) and the accuracy of DNNs and utilized this phenomenon to predict the model performance [21]. In addition to using the dispersity value to predict performance directly, the nuclear norm [17] has been used to combine the model confidence and the dispersity. After that regression model was trained based on the combined values for final performance estimation.

Similar to work [6], Jiang et al. also found that there is a strong correlation between the disagreement and test errors of two DNNs that were trained by using the same architecture and training data but different random seed [53]. Thus, this finding can be used to estimate the performance of DNNs by only using the disagreement information from the model and its auxiliary model. Besides, the authors found that the ensembled model using these trained models has a well-calibrated nature. Zhu et al. found that OOD data could affect the effectiveness of AutoEval [23] and proposed to remove the OOD data from the unlabeled set before conducting performance estimation [127]. Here, an Energy-based OOD detection method that uses the energy score of output probabilities to detect OOD data has been used in the OOD detection process.

Xie et al. [113] empirically demonstrated that the distribution difference cannot be always useful for performance estimation, that is, existing methods are not always reliable. Besides, the authors defined a new score—dispersion score to replace the distribution difference. To do so, the inputs were divided into different groups first based on their pseudo labels. Then, the dispersion score was computed by the average distances between each cluster center and the center of all features, weighted by the sample size of each cluster. Finally, regression models were trained for performance estimation using the pairs of dispersion score and model performance.

In contrast to prior studies that assess model performance across all classes, Zhao et al. [121] conducted an empirical investigation to explore the effectiveness of existing performance estimation methods for individual classes. Based on the empirical study, they found that (1) existing methods have high estimation errors in each class especially when the subset size is small, (2) the overall performance of the methods can be further improved if we reduce their accuracy estimation errors in each class, and (3) there is still a large performance improvement room for each performance estimation method.

Elsahar et al. [25] empirically studied three methods, H-divergence, Confidence-based, and Reverse classification accuracy for predicting the performance drops of DNNs. H-divergence-based methods use another classification model to distinguish between training data and target data. Confidence-based methods use the difference of output probabilities to measure the performance drops. Reverse classification accuracy methods utilize the pseudo-label predicted by the original DNN to train a new DNN with the same architecture and training configuration. The performance drops are calculated by comparing the performance of original and newly trained DNNs. The authors found the H-divergence-based methods are better at predicting the performance drop on natural distribution shift while Confidence-based methods are good at predicting the performance drop on adversarial distribution shift.

Table 11 lists the evaluation metrics used for performance estimation. Basically, there are two types of metrics, the first one directly computes the difference between real model performance and estimated model performance, for example, MSE and Accuracy difference. The second one evaluates the correlation between the scores (computed by their proposed methods) and the real model performance, for example, Pearson correlation coefficient and Coefficient of determination Therefore, multiple data values (multiple models or datasets used to calculate such values) are needed for the computation of correlation.

Table 12 summarizes the types of datasets used for the evaluation in each work. Surprisingly, 32% of the works only evaluated their methods using in-distribution datasets. Besides, only six studies [6, 10, 21, 36, 99, 127] evaluated performance estimation methods on in-distribution dataset, synthetic distribution dataset, and natural distribution dataset.

We compare the works from SE and ML from the following three perspectives.