A Survey of Methods for Explaining Black Box Models A Survey of Methods for Explaining Black Box Models

In the analysis of the interpretability of predictive models, we can identify a set of dimensions to be taken into consideration, and that characterize the interpretability of the model [28].

Global and Local Interpretability: A model may be completely interpretable, i.e., we are able to understand the whole logic of a model and follow the entire reasoning leading to all the different possible outcomes. In this case, we are speaking about global interpretability. Instead, we indicate with local interpretability the situation in which it is possible to understand only the reasons for a specific decision: only the single prediction/decision is interpretable.

Time Limitation: An important aspect is the time that the user is available or is allowed to spend on understanding an explanation. The user time availability is strictly related to the scenario where the predictive model has to be used. Therefore, in some contexts where the user needs to quickly take the decision (e.g., a disaster is imminent), it is preferable to have an explanation simple to understand. While in contexts where the decision time is not a constraint (e.g., during a procedure to release a loan) one might prefer a more complex and exhaustive explanation.

Nature of User Expertise: Users of a predictive model may have different background knowledge and experience in the task: decision-makers, scientists, compliance and safety engineers, data scientists, and so on. Knowing the user experience in the task is a key aspect of the perception of interpretability of a model. Domain experts may prefer a larger and more sophisticated model over a smaller and sometimes more opaque one.

The works reviewed in the literature only implicitly specify if their proposal is global or local. Just a few of them take into account the nature of user expertise [34, 98, 104], and only few of them provide real experiments about the time required to understand an explanation [61, 129]. Instead, some of the works consider the “complexity” of an explanation through an approximation. For example, they define the model complexity as the model's size (e.g., tree depth, number of rules, etc.) [25, 38, 47, 98]. In the following, we further discuss issues related to the complexity of an explanation.

An interpretable model is required to provide an explanation. Thus, to realize an interpretable model, it is necessary to take into account the following list of desiderata, which are mentioned by a set of papers in the state of art [5, 28, 32, 45]:

• Interpretability: to which extent the model and/or its predictions are human understandable. The most addressed discussion is related to how the interpretability can be measured. In Reference [32] a component for measuring the interpretability is the complexity of the predictive model in terms of the model size. According to the literature, we refer to interpretability also with the name comprehensibility.
  
• Accuracy: to which extent the model accurately predicts unseen instances. The accuracy of a model can be measured using evaluation measures like the accuracy score, the F1-score [118], and so on. Producing an interpretable model maintaining competitive levels of accuracy is the most common target among the papers in the literature.
  
• Fidelity: to which extent the model is able to accurately imitate a black-box predictor. The fidelity captures how much is good an interpretable model in the mimic of the behavior of a black-box. Similarly to the accuracy, the fidelity is measured in terms of accuracy score, F1-score, and so on, but with respect to the outcome of the black box.
  

Moreover, besides these features are strictly related to interpretability, yet according to [5, 28, 32, 45] a data-mining and machine-learning model should have other important desiderata. Some of these desiderata are related to ethical aspects such as fairness and privacy. The first principle requires that the model guarantees the protection of groups against (direct or indirect) discrimination [100]; while the second one requires that the model does not reveal sensitive information about people [4]. The level of interpretability of a model together with the standards of privacy and non-discrimination that are guaranteed may impact on how much human users trust that model. The degree of trust on a model increases if the model is built by respecting constraints of monotonicity given by the users [77, 87, 123]. A predictor respecting the monotonicity principle is, for example, a predictor where the increase of the values of a numerical attribute tends to either increase or decrease in a monotonic way the probability of a record of being member of a class [32]. Another property that influences the trust level of a model is usability: people tend to trust more models providing information that assist them to accomplish a task with awareness. In this line, an interactive and queryable explanation results to be more usable than a textual and fixed explanation.

Furthermore, data mining and machine-learning models should also have other ordinary important required features such as reliability, robustness, causality, scalability, and generality. This means that a model should have the ability to maintain certain levels of performance independently from small variations of the parameters or of the input data (reliability/robustness) and that controlled changes in the input due to a perturbation affect the model behavior (causality). Moreover, since we are in the “Big Data era,” it is opportune to have models able to scale to large input data with large input spaces. Finally, since often in different application scenarios one might use the same model with different data, it is preferable to have portable models that do not require special training regimes or restrictions (generality).

In the state of the art a small set of existing interpretable models is recognized: decision tree, rules, linear models [32, 43, 98]. These models are considered easily understandable and interpretable for humans.

A decision system based on a decision tree exploits a graph structured like a tree and composed of internal nodes representing tests on features or attributes (e.g., whether a variable has a value lower than, equals to or grater than a threshold, see Figure 2) and leaf nodes representing a class label. Each branch represents a possible outcome [92]. The paths from the root to the leaves represent the classification rules. Indeed, a decision tree can be linearized into a set of decision rules with the if-then form [31, 90, 91]:

Fig. 2. Example of decision tree classifier.

More generally, a decision rule is a function that maps an observation to an appropriate action. Decision rules can be extracted by generating the so-called classification rules, i.e., association rules that in the consequence have the class label [3]. The most common rules are if-then rules where the if clause is a combination of conditions on the input variables. In particular, it may be formed by conjunctions, negations and disjunctions. However, methods for rule extraction typically take into consideration only rules with conjunctions. Other types of rules are: m-of-n rules, where given a set of $n$ conditions, if $m$ of them are verified, then the consequence of the rule is considered true [79]; list of rules, where given an ordered set of rules is considered true, the consequent of the first rule that is verified [135]; falling rule lists consists of a list of if-then rules ordered with respect to the probability of a specific outcome, and the order identifies the example to be classified by that rule [127]; decision sets, where an unordered set of classification rules is provided such that the rules are not connected by else statements, but each rule is an independent classifier that can assign its label without regard for any other rules [61].

The interpretation of rules and decision trees is different with respect to different aspects [32]. Decision trees are widely adopted for their graphical representation, while rules have a textual representation. The main difference is that textual representation does not provide immediately information about the more relevant attributes of a rule. However, the hierarchical position of the features in a tree gives this kind of clue.

Attributes’ relative importance could be added to rules by means of positional information. Specifically, rule conditions are shown by following the order in which the rule extraction algorithm added them to the rule. Even though the representation of rules causes some difficulties in understanding the whole model, it enables the study of single rules representing partial parts of the whole knowledge (“local patterns”), which are composable. Also in a decision tree, the analysis of each path separately from the leaf node to the root, enables users to focus on such local patterns. However, if the tree is very deep in this case it is a much more complex task.

A further crucial difference between rules and decision trees is that in a decision tree each record is classified by only one leaf node, i.e., the class predicted are represented in a mutually exclusive and exhaustive way by the set of leaves and their paths to the root node. However, a certain record can satisfy the antecedent of rules having as consequent a different class for that record. Indeed, rule-based classifiers have the disadvantage of requiring an additional approach for resolving such situations of conflicting outcome [133]. Many rule-based classifiers deal with this issue by returning an ordered rule list, instead of an unordered rule set. In this way it is returned the outcome corresponding to the first rule matching the test record and ignoring the other rules in the list. We notice that ordered rule lists may be harder to interpret than classical rules. In fact, in this model a given rule cannot be considered independently from the precedent rules in the list [133]. Another widely used approach consists in considering the top-k rules satisfying the test record where the ordering is given by a certain weight (e.g., Laplace accuracy). Then, the outcome of the rules with the average highest weight among the top-k is returned as predicted class [135].

Another set of approaches adopted to provide explanations are linear models [29, 39, 98]. This can be done by considering and visualizing the features importance, i.e., both the sign and the magnitude of the contribution of the attributes for a given prediction (see Figure 3). If the contribution of an attribute-value is positive, then it contributes by increasing the model's output. Instead, if the sign is negative then the attribute-value decreases the output of the model. If an attribute-value has an higher contribution than another, then it means that it has an higher influence on the prediction of the model. The produced contributions summarize the performance of the model, thus the difference between the predictions of the model and expected predictions, providing the opportunity of quantifying the changes of the model prediction for each test record. In particular, it is possible to identify the attributes leading to this change and for each attribute how much it contributed to the change. An intrinsic problem that linear models have when used for explanation is that when the model does not optimally fit the training data, it may use spurious features to optimize the error, and these features may be very hard to interpret for a human.

Fig. 3. Example of linear model and the returned features importance.

We point out that, in general, when an explanation for a prediction is provided, besides the explanation (satisfied rules, branch of the tree, features importance, etc.), it is often useful to analyze also instances that are exceptions with respect to the “boundaries” provided by the explanation, or with very few differences with respect to the prototypes returned as explanation. For example, instances covered by the rule body but with an outcome label different from the class of the outcome predicted. Even though this sort of exception analysis is hardly performed, it can be more informative than the direct explanation, and it can also provide clues about the application domain [37, 85].

Finally, as last remark, we underline that all the aforementioned techniques for providing explanations are effectively interpretable only when they have human-reasonable sizes. Indeed, the goodness of the explanation could be invalidated by its size and complexity. For example, when the linear model is high-dimensional, the explanation may be overwhelming. Moreover, if a too large set of rules, or a too deep and wide tree are returned they could not be humanly manageable even though they are perfectly capturing the internal logic of the black box for the classification.

In the literature, very little space is dedicated to a crucial aspect: the model complexity. The evaluation of the model complexity is generally tied to the model comprehensibility, and this is a very hard task to address. As a consequence, this evaluation is generally estimated with a rough approximation related to the size of the model. Moreover, complexity is often used as an opposed term to interpretability. Before analyzing the various notions of model complexity in the literature, we point out that, concerning the problem of black box explanation, the complexity is only related to the model and not to the training data that is generally unknown.

In Reference [38] the complexity is identified by the number of regions, i.e., the parts of the model, for which the boundaries are defined. In Reference [98] as complexity for linear models is adopted the number of non-zero weights, while for decision trees the depth of the tree. In Reference [25] the complexity of a rule (and thus of an explanation) is measured by the length of the rule condition, defined as the number of attribute-value pairs in the condition. Given two rules with similar frequency and accuracy, the rule with the smaller length may be preferred as it is more interpretable. Similarly, in the case of lists of rules the complexity is typically measured considering the total number of attribute-value pairs in the whole set of rules. However, this could not be a suitable way for measuring the model complexity, since in an ordered rule list different test records need distinct numbers of rules to be evaluated [32]. In this kind of model, a more honest measure could be the average number of conditions evaluated to classify a set of test records [84]. However, this is more a “measure of the explanation” of a list of rules, rather than a “measure of the complexity” of rule list's itself.

In the decision tree literature, the problems of over-fitting the training data and of “trading accuracy for simplicity” [10] have been addressed by a class of post-processing algorithms called simplification methods [93]. The most well-known simplification method is decision tree pruning, which relies on an error estimation function to compare the errors of the two alternatives: to prune a subtree or not [101]. Such methods are mainly intended for improving model generality.

Differently from the not flexible representation of decision trees where the prediction of a single record is mutually exhaustive and exclusive, rules characterization contains only significant clauses. That is, a record can satisfy more than a rule also with different outcome labels, and there could be records that are not covered by any rule. Moreover, rules do not capture insignificant clauses, while decision trees can also have insignificant branches. This happens because rule-based classifier generally select one attribute-value while expanding a rule, whereas decision tree algorithms usually select one attribute while expanding the tree [32]. Considering these aspects to estimate the complexity is very difficult. Consequently, even though a model equivalence exists, the estimation of the fact that a different representation for the same model (or explanation) is more complex than another can be very subjective with respect to the interpreter.

The types of data used for classification may have diverse nature. Different types of data present a different level of interpretability for a human. The majority of data mining and machine-learning techniques work on data organized in tables that algorithms may handle as matrices. The advantage of this type of data is that it is both easily managed by these algorithms, without requiring specific transformations, and enough simple to be interpreted by humans [43]. Whereas the disadvantage of tables is that the interpretation of the represented information requires understanding also the meta-data that allow us to associate a meaning to values in the tables.

Other forms of data that are more understandable than tables are images and texts. They indeed represent the most common and natural way people use to communicate in everyday life, and they do not require the understanding of any meta-structure useful for deriving a specific meaning. However, the processing of these data for predictive models requires their transformation into vectors that make them easier to be processed by algorithms. Indeed, on images and texts, the state of art techniques typically apply predictive models based on support vector machine, neural networks or deep neural networks that are usually hard to be interpreted. As a consequence, certain recognized interpretable models cannot be directly employed for this type of data to obtain an interpretable model or a human understandable explanation. Transformations using equivalences, approximations or heuristics are required in such a way that images and texts can be employed by prediction systems and used for providing the interpretation of the model and/or the prediction at the same time.

Finally, there exist other forms of data such as sequences, spatio-temporal data and complex networks that may be used by data mining and machine-learning algorithms. However, to the best of our knowledge, in the literature there is no work addressing the black box model explanation for data different from images, texts, and tabular data. The only exceptions are [39, 113] that using EEG data provide heatmaps with data point's relevance for the decision's outcome.⁵

In the following, we generalize the classification problem (see Figure 5). A predictor, also named model or classifier, is a function $b:\mathcal {X}^{(m)} \rightarrow \mathcal {Y}$, which maps data instances (tuples) $x$ from a feature space $\mathcal {X}^{(m)}$ with $m$ input features to a decision $y$ in a target space $\mathcal {Y}$. We write $b(x) = y$ to denote the decision $y$ predicted by $b$, and $b(X) = Y$ as a shorthand for $\lbrace b(x) \ |\ x \in X\rbrace = Y$. An instance $x$ consists of a set of $m$ attribute-value pairs $(a_i, v_i)$, where $a_i$ is a feature (or attribute) and $v_i$ is a value from the domain of $a_i$. The domain of a feature can be continuous or categorical. The target space $\mathcal {Y}$ (with dimensionality equals to one) contains the different labels (classes or outcomes) and also in this case the domain can be continuous or categorical. Note that, in case of ordinal classification labels in $Y$ have an order. A predictor $b$ can be a machine-learning model, a domain-expert rule-based system, or any combination of algorithmic and human knowledge processing. In the following, we denote by $b$ a black box predictor, whose internals are either unknown to the observer or they are known but uninterpretable by humans. Examples include neural networks, SVMs, ensemble classifiers, or a composition of data-mining and hard-coded expert systems. Instead, we denote with $c$ an interpretable predictor, whose internal processing yielding a decision $c(x)$ can be given a symbolic interpretation comprehensible by a human, i.e., for which a global or a local explanation is available. Examples of such predictors include rule-based classifiers, decision trees, decision sets, and rational functions.

Fig. 5. Classification problem: given a train $D_{\mathit {train}}$ and a test $D_{\mathit {test}}$ set the black box model is learned on the train (black box learner) generating the black box predictor $b$ that can applied on the test set $D_{\mathit {test}}$ to obtain the prediction $Y = \bigcup _{x\in D_{\mathit {test}}} b(x)$.

In supervised learning [118], a training dataset $D_{\mathit {train}}$ is used for training a predictor $b$, and a test dataset $D_{\mathit {test}}$ is used for evaluating the performance of $b$. In the black box explanations’ problems, treated in this survey, $D_{\mathit {train}}$ is usually unknown. Given $D_{\mathit {test}} = \lbrace X, \hat{Y}\rbrace$, the evaluation consists of observing for each pair of data record and target value $(x, \hat{y}) \in D_{\mathit {test}}$ the matches between $\hat{y}$ and $b(x)= y$. The $\mathit {accuracy}$ is the percentage of matches over the size of the test dataset.

The predictive performances of both the black box $b$ and the interpretable predictor $c$ can be evaluated through the $\mathit {accuracy}$ measure over the test dataset. For the interpretable predictor $c$, a second measure is the $\mathit {fidelity}$, which evaluates how well $c$ mimicks the black box predictor $b$ on the test dataset. Formally, fidelity is the percentage of matches $c(x) = b(x)$, for $(x, \hat{y}) \in D_{\mathit {test}}$, over the size of the test dataset. Note that the $\mathit {fidelity}$ score can be interpreted as the accuracy of the interpretable predictor $c$ with respect to predictions of the black box $b$. The measures $\mathit {accuracy}$ and $\mathit {fidelity}$ can be easily extended to more refined ones such as precision, recall, F1-score [118].

The black box explanation problem consists in providing a global explanation of the black box model through an interpretable and transparent model. This model should be both able to mimic the behavior of the black box and it should also be understandable by humans. In other words, the interpretable model approximating the black box must be globally interpretable. We formalize this problem by assuming that the interpretable global predictor is derived from the black box and some dataset of instances $X$. The dataset $X$ provided by the user is a sampling of the domain $\mathcal {X}^{(m)}$, and it may include actual class values (hence allowing to evaluate accuracy of the interpretable model). As extreme possibilities, $X$ can be empty or $X$ can precisely be the training dataset used to learn $b$. The process of extracting the interpretable predictor may further expand $X$, e.g., by random perturbation or random sampling. For such test instances, only the predictions of the black box are known—see Figure 6. Therefore, we define the model explanation problem as follows.

Fig. 6. Model explanation problem example. Starting from test instances in $X$, first query the black box, and then extract an interpretable global predictor from $\lbrace X, b(X)\rbrace$ in the form of a decision rule classifier.

A large set of papers reviewed in this survey describe various designs for the function $f$ to solve the explanation problem, including many approaches in the expansion of the dataset $X$ for the purpose of training the interpretable model $c_g$. Such models can include decision trees or decision rule classifiers [43]. The domain of explanations $\mathcal {E}$ in such cases consists of decision trees and sets of rules, respectively. An example is shown in Figure 6. The definition above also accounts for an explanation logic $\varepsilon _g(\cdot ,\cdot)$, which further processes the interpretable global predictor, e.g., to provide an abstraction of $c_g$ tailored at users with different background knowledge. As an example, the explanation logic may condense a decision tree to extract a feature importance vector, summarizing the contribution of features to the decisions of the global model (see Section 6.3).

Given a black box and an input instance, the outcome explanation problem consists in providing an explanation for the outcome of the black box on that instance. It is not required to explain the whole logic underlying the black box but only the reason for the prediction on a specific input instance. We formalize this problem by assuming that first an interpretable local model $c_l$ is build from the black box $b$ and the instance $x$, and then an explanation is derived from $c_l$—see Figure 7.

Fig. 7. Outcome explanation problem example. For a test instance $x$, the black box decision $b(x)$ is explained by building an interpretable local predictor $c_l$, e.g., a decision rule classifier. The local explanation $\varepsilon _l(c_l, x)$ is the specific rule used to classify $x$.

As an example, the local predictor $c_l$ can be a decision tree built from a neighborhood of $x$, and an explanation $e$ can be the path of the decision tree followed by attribute values in $x$ [32]. Another example is shown in Figure 7. We will survey recent works adopting very diversified approaches that instantiate the outcome explanation problem.

The model inspection problem consists in providing a representation (visual or textual) for understanding some specific property of the black box model or of its predictions. Example properties of interest include sensitivity to attribute changes, and identification of components of the black box (e.g., neurons in DNN) responsible for specific decisions. We define this problem as follows.

For example, the function $f$ may be based on sensitivity analysis that, by observing the changes occurring in the predictions when varying the input of $b$, returns a set of visualizations (e.g., partial dependence plots [55] or variable effect characteristic curve [19]), highlighting the feature importance for the predictions. See the example in Figure 8. The key element differentiating the inspection problem from the model explanation problem is that the latter requires the extraction of an interpretable global predictor, while the former concentrates the analysis of specific properties of the black box without requiring a global understanding of it. This is readily checked by contrasting Figures 8 and 6.

Fig. 8. Model inspection problem example. Query the black box on test instances $X$, and then extract a sensitivity analysis plot.

Fig. 9. Transparent box design problem example. A decision rule classifier learned from a training dataset is globally interpretable predictor. Moreover, the rule that applies on a given test instance is a local explanation of the predictor's decision.

The transparent box design problem consists in directly providing a model that is locally or globally interpretable.

An example interpretable predictor $c$ is a decision tree classifier. A local explanation for an instance $x$ can be derived as a decision rule with conclusion $c(x)$ and with premise the conditions from the tree path followed according to the attributed values of $x$. A global explanation can be the decision tree itself, or, as discussed in the case of model explanation, a feature importance vector condensed from the decision tree. See Figure 9 as example.

In summary, according to our problem definitions, when stating that a method is able to open the black box, we are referring to one of the following statements: (i) it explains the model, (ii) it explains the outcome, (iii) it can inspect the black box internally, (iv) it provides a transparent solution.

Before proceeding in the detailed analysis and classification of papers proposing method $f$ for understanding black boxes $b$, we present in this section the most widely used approach to solve the three black box model explanation problems. We refer to this approach as reverse engineering, because the black box predictor $b$ is queried with a certain test dataset to create an oracle dataset that in turn will be used to train the comprehensible predictor (see Figure 10). The name “reverse engineering” comes from the fact that we can only observe the input and output of the black box.

Fig. 10. Reverse engineering approach: the learned black box predictor $b$ is queried with a test dataset $D=\lbrace X, Y\rbrace$ to produce an oracle $\hat{Y}$, which associate to each record $x \in X$, a label that is not real but assigned by the black box.

With respect to the black box model and outcome explanation problems, the possibility of action tied with this approach relies on the choice of adopting a particular type of comprehensible predictor, and in the possibility of querying the black box with input records created in a controlled way and/or by using random perturbations of the initial train or test dataset. Regarding the random perturbations of the input used to feed the black box, it is important to recall that recent studies discovered that DNN built for classification problems on texts and images can be easily fooled (see Section 2). Changes in an image that are imperceptible to humans can lead a DNN to label the record as something else. Thus, according to these discoveries, the methods treating images or text, in theory, should not be enabled to use completely random perturbations of their input. However, this is not always the case in practice [98].

Such a reverse engineering approach can be classified as generalizable or not (or pedagocial vs. decompositional as described in Reference [76]). We say that an approach is generalizable when a purely reverse engineering procedure is followed, i.e., the black box is only queried with different input records to obtain an oracle used for learning the comprehensible predictor (see Figure 11(left)). In other words, internal peculiarities of the black box are not exploited to build the comprehensible predictor. Thus, if an approach is generalizable, even though it is presented to explain a particular type of black box, in reality, it can be used to interpret any kind of black box predictor. That is, it is an agnostic approach for interpreting black boxes. However, we say that an approach is not generalizable if it can be used to open only that particular type of black box for which it was designed for (see Figure 11(right)). For example, if an approach is designed to interpret random forest and internally use a concept of distance between trees, then such an approach cannot be used to explain predictions of a NN. A not generalizable approach cannot be black box agnostic.

Fig. 11. (Left) Generalizable reverse engineering approach: internal peculiarities of the black box $b$ are not exploited to build the comprehensible predictor $c$. (Right) Not Generalizable reverse engineering approach: the comprehensible predictor $c$ is the result of a procedure involving internal characteristics of the black box $b$.

In Tables 6, 7, and 8, we keep track of these aspects with the two features General and Random. With General, we indicate if an explanatory approach can be generalized for every black box, while with Random, we indicate if any kind of random perturbation or permutation of the original dataset is used by the explanatory approach.

In light of these concepts, as the reader will discover below, a further classification not explicitly indicated emerges from the analysis of these papers. This fact can be at the same time a strong point or a weakness of the current state of the art. Indeed, we highlight that the works for opening the black box are realized for two cases. The first (larger) group contains approaches proposed to tackle a particular problem (e.g., medical cases) or to explain a particular type of black box, that is, the solutions are specific for the problem instance. The second group contains general purpose solutions that try to be general as much as possible and propose agnostic and generalizable solutions.

The following set of works address the model explanation problem implementing in different ways the function $f$. However, all these works adopt a decision tree as comprehensible global predictor $c_g$, and consequently represent the explanation $\varepsilon _g$ with the decision tree itself. We point out that all the methods presented in this section work on tabular data.

Another commonly used interpretable and easily understandable model is the set of rules. When a set of rules describing the logic behind the black box model is returned the interpretability is provided at a global level. In the following, we present a set of reference works solving the model explanation problem by implementing in different ways function $f$, and by adopting any kind of decision rules as comprehensible global predictor $c_g$. Hence, the global explanation $\varepsilon _g$ changes accordingly to the type of rules extracted by $c_g$. Also, all the methods presented in this section work on tabular data.

6.2.2 Explanation of Support Vector Machines. In the following are shown methods $f$ for explaining Support Vector Machine (SVM) [118] still returning a comprehensible global predictor $c_g$ consisting in a rule-based classifier.

The authors of Reference [82] propose the SVM+Prototypes (SVM+P) procedure $f$ for rule extraction $c_g$ from support vector machines $b$. It works as follows: it first determines the decision function by means of a SVM, then a clustering algorithm is used to find out a prototype vector for each class. By using geometric methods, these points are joined with the support vectors for defining ellipsoids in the input space that can be transformed into if-then rules.

Fung et al., in Reference [33], describe as function $f$ an algorithm based on constraint programming for converting linear SVM $b$ (and other hyperplane-based linear classifiers) into a set of non overlapping and interpretable rules $c_g$. These rules are asymptotically equivalent to the original linear SVM. Each iteration of the algorithm for extracting the rules is designed to solve a constrained optimization problem having a low computational cost. We underline that this black box explanation solution $f$ is not generalizable and can be employed only for Linear SVM-like black boxes.

In Reference [76] the authors propose a qualitative comparison of the explanations returned by techniques for extraction of rules from SVM black boxes (e.g., SVM+P [82], Fung method [33]) against the redefining of methods designed for explaining neural networks, i.e., C4.5 [118], Trepan [22], and G-REX [44]. How we anticipated in the previous section, the authors delineate the existence of two type of approaches to extract rules: pedagogical and decompositional (see Figure 12). Pedagogical techniques $f$ directly extract rules that relate the inputs and outputs of the predictor (e.g., Reference [22, 44]), while decompositional approaches are closely intertwined with the internal structure of the SVM (e.g., Reference [33, 82]). We recall that, in Table 2, we identify with the term generalizable the pedagogical approaches.

Fig. 12. From Reference [76]: pedagogical (a) and decompositional (b) rule extraction techniques.

Recent approaches for interpretation are agnostic (AGN) with respect to the black box to be explained. In this section, we present a set of works solving the model explanation problem by implementing function $f$ such that any type of black box $b$ can be explained. These approaches do not return a specific comprehensible global predictor $c_g$, thus the type of explanation $\varepsilon _g$ change with respect to $f$ and $c_g$. By definition all these approaches are generalizable.

Probably the first attempt of an agnostic solution was proposed in Reference [69]. Lou et al. propose a method $f$, which exploits Generalized Additive Models (GAMs) and it is able to interpret regression splines (linear and logistics), single trees and tree ensembles (bagged trees, boosted trees, boosted bagged trees and random forests). GAMs are presented as the gold standard for intelligibility when only univariate terms are considered. Indeed, the explanation $\varepsilon _c$ is returned as the importance of the contribution of the individual features in $b$ together with their shape function, such that the impact of each predictor can be quantified. A shape function is the plot of a function capturing the linearities and nonlinearities together with its shape. It works on tabular data. A refinement of the GAM approach is proposed by the same authors in Reference [70]. A case study on health care showing the application of the GAM the refinement is presented in Reference [16]. In particular, this approach is used for the prediction of the pneumonia risk and hospital 30-day readmission.

In Reference [40], the authors present an iterative algorithm $f$, named GoldenEye, which is based on data randomization (within class permutation, data permutation, etc.) and on finding groups of attributes whose interactions have an impact on the predictive power. The attributes and the dependencies among the grouped attributes represent the global explanation $\varepsilon _g$.

In Reference [58], Partition Aware Local Model (PALM) is presented to implement $f$. PALM is a method able to learn and summarize the structure of the training dataset to help the machine-learning debugging. PALM mimes a black box $b$ using a meta-model for partitioning the training dataset, and a set of sub-models for approximating and miming the patterns within each partition. As meta-model it uses a decision tree ($c_g$) so that the user can examine its structure and determine if the rules detected follow the intuition or not, and link efficiently problematic test records to the responsible train data. The sub-models linked to the leaves of the tree can be arbitrarily complex models able to catch elaborate local patterns, but yet interpretable by humans. Thus, with respect to the final sub-models PALM is not only black box agnostic but also explanator agnostic. Moreover, PALM is also data agnostic; i.e., it can work on any kind of data.

In Reference [142] is introduced FIRM (Feature Importance Ranking Measure) for implementing function $f$ as an extension of the POIMs [111]. FIRM takes the underlying correlation structure of the features into account and it is able to discover the most relevant ones. The model explanation $\varepsilon _g$ is provided through a vector of features importance. FIRM is general as it can be applied to a very broad family of classifiers. In summary, the FIRM score depends on the conditional expected value for a feature and the importance of the feature is measured as the variability of the conditional expected score.

A further extension of FIRM is proposed in Reference [124]. Vidovic et al. propose the Measure of Feature Importance (MFI). As FIRM, MFI can be applied to any classifier. MFI is intrinsically non-linear and can detect features that by itself are inconspicuous and only impact the prediction function through their interaction with other features. Moreover, MFI can be used for solving both the black box model explanation and the outcome explanation.

In Reference [121], a solution for the model explanation problem is presented. It adopts an approach that cannot be classified as one of the previous. The proposed approach $f$ uses the internals of a random forest model $b$ to produce recommendations on the transformation of true negative examples into positively predicted examples. These recommendations, which are strictly related to the feature importance, corresponds to the comprehensible global predictor $c_g$. In particular, the function $f$ aims at transforming a negative instance into a positive instance by analyzing the path on the trees in the forest predicting such instance as positive or negative. The explanation of $b$ is provided by means of the helpfulness of the features in the paths adopted for changing the instance outcome from negative to positive.

Another approach hard to classify is Reference [111], based on References [142] and [124]. The authors explain SVM black boxes $b$ through the concept of POIMs: Positional Oligomiter Importance Matrices (i.e., the global explanation function provided $\varepsilon _g$). The problem faced in Reference [111] is specific for the classification of DNA sequences and $k$-mers. The POIMs $f$ is a scoring system that assigns a score to each $k$-mer and it can be used to rank and visualize the $k$-mer scoring system.

In the following works the opened black box $b$ is a DNN and the explanation is provided by using a Saliency Mask (SM) as comprehensible local predictor $c_l$, i.e., a subset of the original record, which is mainly responsible for the prediction. For example, as a salient mask, we can consider the part of an image or a sentence in a text. A saliency image summarizes where a DNN looks into an image for recognizing their predictions. The function $f$ to extract the local explanation $\varepsilon _l$ is always not generalizable and often strictly tied with the particular type of network, i.e., convolutional, recursive, and so on. We point out that some of the papers described in this section could also be categorized as methods for solving the model inspection problem and thus presented in Section 8 and vice-versa. Another aspect that is worth to highlight is the slight difference between saliency masks and features importance. Indeed, they can be seen as two different ways of building the same explanation: in both cases a value that expresses the importance of a feature/area is provided.

Reference [134] introduces an attention-based model $f$, which automatically identifies the contents of an image. The black box is a neural network that consists of a combination of a Convolutional NN (CNN) for the features extraction and a Recursive NN (RNN) containing Long Short Term Memory (LSTM), nodes producing the image caption by generating a single word for each iteration. The explanation $\varepsilon _l$ of the prediction is provided through a visualization of the attention (area of an image, see Figure 13(left)) for each word in the caption. A similar result is obtained by Fong et al. in Reference [30]. In this work, the authors propose a framework $f$ of explanations $c_l$ as meta-predictors. In their view, an explanation $\varepsilon _l$, and thus a meta-predictor, is a rule that predicts the response of a black box $b$ to certain inputs. Moreover, they use saliency maps as explanations for black boxes to highlight the salient part of the images (see Figure 13(right)).

Fig. 13. Saliency masks for explanation of deep neural network. (Left) From [134] the elements of the image highlighted. (Right) From Reference [30] the mask and the level of accuracy on the image considering and not considering the learned mask.

Similarly, another set of works produce saliency masks incorporating network activations into their visualizations. This kind of approaches $f$ are named Class Activation Mapping (CAM). In Reference [139], global average pooling in CNN (the black box $b$) is used for generating the CAM. A CAM (the local explanation $\varepsilon _l$) for a particular outcome label indicates the discriminative active region that identifies that label. Reference [106] defines its relaxed generalization Grad-CAM, which visualizes the linear combination of a late layer's activations and label-specific weights (or gradients for Reference [139]). All these approaches invoke different versions of back propagation and/or activation, which results in aesthetically pleasing, heuristic explanations of image saliency. Their solution is not black box agnostic limited to NN, but it requires specific architectural modifications [139] or access to intermediate layers [106]. However, in Reference [109] is proposed a method $f$ for visualizing saliency mask $\varepsilon _l$ specialized for CNN black boxes $b$.

Another set of works base their explanation on saliency masks $\varepsilon _l$ and on using a technique $f$, which involves the backpropagtion from the output to the input layer. This technique is called Layer-wise Relevance Propagation (LRP) and consists in assigning a relevance score fore each layer backpropagating the effect of a decision on a certain image up to the input level. Thus, LRP can also be seen as a way for obtaining the features importance, then visualized through saliency masks. The first method exploiting this technique is the Pixel-wise Decomposition method (PWD) described in Reference [7]. In this work, the saliency masks are named heatmaps. PWD is presented as an explanator of nonlinear classifiers and in particular of neural networks. Input images are encoded in $f$ using the bag of (visual) word features and the contribution of every pixel is shown as an heatmap that shows the focus of the black box $b$ for taking a certain decision. In Reference [113], LRP is specifically adopted for working on DNN trained to classify EEG analysis data. An evolution of LRP is the Deep Taylor Decomposition (DTD) method illustrated in Reference [78]. It tries to overcome a functional approach, where the explanation results form the local analysis of the prediction function [109], and where the explanation is obtained by running a backward pass in that graph [7, 137]. Also DTD is used for interpreting multilayer neural networks by composing the network decision into contributions (relevance propagation) of its input elements. In [103] it is presented a comparison between explanations provided with LRP and more traditional deconvolution and sensitivity analysis. Another work following this line of research is Reference [107], which focuses on the activation differences. It is worth to mention that perhaps the first step toward LRP was Reference [64], with the Contribution Propagation (CP) method $f$. Similar to LRP, Contribution Propagation, inspired by Reference [89], calculates the contributions backwards from the output level to the input in such a way that for each instance, CP explains how important each part of the record was for that decision.

In Reference [143] the authors present a probabilistic methodology $f$ for explaining classification decisions made by DNN. The method can be used to produce a saliency map for each instance and also for each node of the neural network (in this sense it solves the model inspection problem). In particular, the saliency mask highlights the parts of either the image (i.e., the features) of the input that constitute most evidence for (or against) the activation of the given output or internal node. This approach uses a technique that exploits the difference analysis of activation maximization on the network nodes producing a relevance vector with the relative importance of the input features.

Another approach $f$ that can be used to solve both the outcome explanation problem and the model inspection in case of image classification with CNN is VisualBackProp (VBP) [11]. VBP visualizes which sets of pixels $\varepsilon _l$ of the input image contribute most to the prediction. The method uses the intuition that the feature maps contain less and less irrelevant information to the prediction decision when moving deeper into the network. VBP was initially developed as a debugging tool for CNN-based systems for steering self-driving cars. This makes VBP a valuable inspection tool that can be easily used during both training and inference. VBP obtains visualization similar to the LRP approach while achieving orders of magnitude speed-ups.

Finally, with respect to texts, in Reference [65] the authors develop an approach $f$ that incorporates rationales as part of the learning process of $b$. A rationale is a simple subset of words representing a short and coherent piece of text (e.g., phrases), and alone must be sufficient for the prediction of the original text. A rational is the local explanator $\varepsilon _l$ and provides the saliency of the text analyzed, i.e., indicates the reason for a certain outcome.

In this section, we present the agnostic solutions proposed for the outcome explanation problem implementing function $f$ such that any type of black box $b$ can be explained. All these approaches are generalizable by definition and return a comprehensible local predictor $c_l$. Thus, in some cases they can also be employed for diversified data types.

A first attempt of describing an agnostic method $f$ for explaining black box systems $b$ is described in Reference [89] with ExplainD (Explain Decision). This framework $f$ is specially designed for Naive Bayes, SVM and Linear Regression black boxes, but it is in principle generalizable to every black box. ExplainD uses the concept of additive models to weight the importance of the features of the input dataset. It provides a graphical explanation of the decision process by visualizing the feature importance for the decisions, the capability to speculate on the effect of changes to the data, and the capability, wherever possible, to drill down and audit the source of the evidence.

Then, in Reference [29], Strumbelj et al. describe a method $f$ for explaining individual predictions of any type of black box $b$ based on Naive Bayes models. The proposed approach exploits notions from coalitional game theory, and explains the predictions utilizing the contribution of the value of different individual features $\varepsilon _l$ (see Figure 3). The method is agnostic with respect to the black box used and is tested only on tabular data.

In Reference [98] is presented the Local Interpretable Model-agnostic Explanations (LIME) approach $f$, which does not depend on the type of data, nor on the type of black box $b$ to be opened, nor on a particular type of comprehensible local predictor $c_l$ or explanation $\varepsilon _l$. The main intuition of LIME is that the explanation may be derived locally from the records generated randomly in the neighborhood of the record to be explained, and weighted according to their proximity to it. In their experiments, the authors adopt linear models as comprehensible local predictor $c_l$ returning the importance of the features as explanation $\varepsilon _l$. As black box $b$, the following classifiers are tested: decision trees, logistic regression, nearest neighbors, SVM, and random forest. A weak point of this approach is the required transformation of any type of data in a binary format that is claimed to be human interpretable: Moreover, in practice the explanation is only provided through linear models and their features importance. References [97] and [96] propose some extensions of LIME with an analysis of particular aspects and cases not overcoming the aforementioned limitations.

A similar approach is presented in Reference [122], where Turner et al. design the Model Explanation System (MES) $f$ that augments black box predictions with explanations by using a Monte Carlo algorithm. In practice, they derive a scoring system for finding the best explanation based on formal requirements and consider that the explanations $\varepsilon _l$ are simple logical statements, i.e., decision rules. The authors test logistic regression and SVMs as black box $b$.

An extension of LIME using decision rules as local interpretable classifier $c_l$ is presented in Reference [99]. The Anchor $f$ uses a bandit algorithm that randomly constructs the anchors with the highest coverage and respecting a user-specified precision threshold. An anchor explanation is a decision rule that sufficiently tie a prediction locally such that changes to the rest of the features values do not matter, i.e., similar instances covered by the same anchor have the same prediction outcome. Anchor is applied on tabular, images and textual datasets. Reference [110] is an antecedent of Anchor for tabular data only. It adopts a simulated annealing approach that randomly grows, shrinks, or replaces nodes in an expression tree (the comprehensible local predictor $c_l$). It was meant to return black box decision in forms of “programs.”

A recent proposal that overcomes both LIME and Anchor in terms of performance and clarity of the explanations is LORE (LOcal Rule-based Explanations) [37]. LORE implements function $f$ by learning a local interpretable predictor $c_l$ on a synthetic neighborhood generated through a genetic algorithm approach. Then, it derives from the logic of $c_l$, represented by a decision tree, an explanation $e$ consisting of: a decision rule explaining the reasons of the decision, and a set of counterfactual rules, suggesting the changes in the instance's features that lead to a different outcome.

In Reference [39] is presented a solution for the outcome explanation problem not easily classifiable in one of the previous categories. Haufe et al. propose an approach $f$ for determining the origin of neural processes in time or space. Function $f$ transforms nonlinear models (NLM) in terms of multivariate classifiers (the black box $b$ of this work) into linear interpretable models (the comprehensible linear predictor $c_l$) from which is possible to interpret the features $\varepsilon _l$ for a specific prediction. In particular, the $f$ described in Reference [39] enables the neurophysiological interpretation of the parameters of nonlinear models.

In this section, we review the works solving the black box inspection problem by implementing function $f$ using Sensitivity Analysis (SA) for the visual representation $r$. Sensitivity analysis studies the correlation between the uncertainty in the output of a predictor and that one in its inputs [102]. The following methods mostly work on tabular datasets. We emphasize that besides model inspection, sensitivity analysis is also used for outcome explanation (e.g., References [78, 103]).

Sensitivity analysis for “illuminating” the black box was first proposed by Olden in Reference [83], where a visual method for understanding the mechanism of NN is described. In particular, they propose to assess the importance of axon connections and the contribution of input variables by means of sensitivity analysis and Neural Interpretation Diagram (NID) to remove not significant connections and improve the network interpretability.

In Reference [8] the authors propose a procedure based on Gaussian Process Classification (GDP), which allows explaining the decisions of any classification method through an explanation vector. That is, the procedure $f$ is black box agnostic. The explanation vectors $r$ are visualized to highlight the features that were most influential for the decision of a particular instance. Thus, we are dealing with an inspection for outcome explanation $\varepsilon _l$.

In Reference [24], Datta et al. introduce a set of Quantitative Input Influence (QII) measures $f$ capturing how much inputs influence the outputs of black box predictors. These measures provide a foundation for transparency reports $r$ of black box predictors. In practice, the output consists in the feature importance for outcome predictions.

Reference [115] studies the problem of attributing the prediction of a DNN (the black box $b$) to its input features. Two fundamental axioms are identified: sensitivity and implementation invariance. These axioms guide the design of an attribution method $f$, called Integrated Gradients (IG), that requires no modification to the original network. Differently from the previous work, this approach is tested on different types of data.

Finally, Cortez in References [18, 19] uses sensitivity analysis based and visualization techniques $f$ to explain black boxes $b$. The sensitivity measures are variables calculated as the range, gradient, variance of the prediction. Then, the visualizations $r$ realized are barplots for the features importance, and Variable Effect Characteristic curve (VEC) [20] plotting the input values (x-axis) versus the (average) outcome responses (see Figure 14(left)).

Fig. 14. (Left) From Reference [18], VEC curve and histogram for the pH input feature (x-axis) and the respective high-quality wine probability outcome (left of y-axis) and frequency (right of y-axis). (Right) From Reference [55], Age at enrollment shown as line plot (top) and partial dependence bar (middle). Color denotes the predicted risk of the outcome.

In this section, we report a set of approaches solving the model inspection problem by implementing a function $f$ that returns a Partial Dependence Plot (PDP). The partial dependence plot $r$ returned by $f$ is a tool for visualizing the relationship between the response variable and predictor variables in a reduced feature space. All the approaches presented in this section are black box agnostic and are tested on tabular datasets.

In Reference [42], the authors present an approach $f$ aimed at evaluating the importance of non-additive interactions between any set of features. The implementation uses the Variable Interaction Network (VIN) visualization generated from the use of ANOVA statistical methodology (a technique to calculate partial dependence plots). VIN allows to visualize in $r$ the importance of the features together with their interdependencies.

Goldstein et al. provide in Reference [35] a technique $f$ that extends classical PDP named Individual Conditional Expectation (ICE) to visualize the model approximated by a black box $b$ that helps in visualizing the average partial relationship between the outcome and some features. ICE plots $r$ improves PDP by highlighting the variation in the fitted values.

In Reference [55], Krause et al. introduce random perturbations on the black box $b$ input values to understand to which extent every feature impact the prediction through a visual inspection $r$ using the PDPs $f$. The main idea of Prospector is to observe how the output varies by varying the input changing one variable at a time. It provides an effective way to understand which are the most important features for a valuable interpretation (see Figure 14(right)).

In Reference [2], the authors propose a method $f$ for auditing (i.e., inspecting) black box predictors $b$, studying to which extent existing models benefit of specific features in the data. This method does not assume any knowledge on the models behavior. In particular, the method $f$ focuses on indirect influence and visualizes in $r$ the global inspection through an obscurity vs. accuracy plot (the features are obscured one after the other).

Yet, the dependence of a black box $b$ on its input features is relatively quantified by the procedure $f$ proposed in Reference [1], where the authors present an iterative procedure based on Orthogonal Projection of Input Attributes (OPIA).

The works we present in this section solve the black box inspection problem by implementing function $f$ using Activation Maximization (AM) for finding the activation neurons and the instances (generally images $r$) characterizing the decision. We remark the very soft difference between these works and those of Section 7.1. Indeed, activation maximization can be used also to find out the pixels and areas on which the black box $b$ focused for taking the decision (e.g., References [11, 143]).

In Reference [136] are proposed two tools for visualizing and interpreting DNNs and for understanding what computations DNNs perform at intermediate layers and which neurons are activated. These tools visualize in $r$ the activations of each layer of a trained CNN during the processing of images or videos. Moreover, they visualize the features of the different layers by regularized optimization in image space. Yosinski et al. found that by analyzing the live activations and observing as they change in correspondence of different inputs, helps to generate an explanation on the DNNs behaviour.

Shwartz-Ziv et al. in Reference [108] proposes $f$ that shows the effectiveness of the Information Plane visualization of DNNs $r$, by highlighting that the empirical error minimization of each stochastic gradient descent phase epoch is always followed by a slow representation compression. This is a very useful result that can be exploited for explaining DNNs.

Similarly to the works presented in 7.1, in Reference [137], Zeiler et al. backtrack the network computations to identify which image patches are responsible for certain neural activations. Simonyan in Reference [109], demonstrated that Zeiler's method can be interpreted as a sensitivity analysis of the network input/output relation.

Explaining the outcome of black boxes is not the main topic of Reference [112], as the paper is more focused in showing that homogeneous and not complicated CNN can reach the state of art performance. However, the intermediate steps of CNN are analyzed introducing a new variant of the “deconvolution approach” for visualizing in $r$ the features learned by $b$.

The method $f$ proposed in Reference [80] by Nguyen et al. for understanding the inner workings of CNN is based on activation maximization. In practice, it is aimed to retrieve in a visual representation $r$ what each neuron has learned to detect by synthesizing an input image that highly activates the neuron. This work proposes a way to improve the qualitative activation maximization using a Deep Generator Network (DGN). Thus, DGN-AM generates synthetic images that look almost real and reveals the features learned by each neuron in an interpretable way.

In Reference [72], the authors analyze the visual information contained in DNN internal representations trying to reconstruct the input image from its encoding. They provide a general framework $f$ that is able to invert image representations. As a side effect this method allows to show that several layers in CNN retain photographically accurate information about the image. Moreover, in Reference [73] this work is extended by comparing various family of methods analyzed for understanding “representations.” Activation maximization is one of these methods. This task is assessed using the result of the natural pre-images reconstructed by Reference [72]. Moreover, these natural pre-images can be helpful in studying for what black box models are discriminative outside the domain of the images used to train them. In Reference [72], natural pre-images are extracted using an approach based on regularized energy minimization.

Finally, it is worth mentioning that in Reference [95] is presented the discovery that a single neuron unit of a DNN can perform alone sentiment analysis after the training of the network reaching the same level of performance of strong baselines.

We present here a solution for the black box inspection problem that adopts an approach $f$ that can be categorized as none of the previous ones. Reference [119] shows the extraction of a visual interpretation $r$ of a DNN using a decision tree. The method TreeView $f$ works as follows. Given the black box $b$ as a DNN, it first decomposes the feature space into K (user defined) overlapping factors. Then, it builds a meta feature for each of the K clusters and a random forest that predicts the cluster labels. Finally, it shows a surrogate decision tree from the forest as an approximation of the black box.

In this section, we present the most relevant state of the art works solving the transparent box design problem by means of comprehensible predictors $c$ based on rules. In these cases, $c_g$ is a comprehensible global predictor providing the whole set of rules leading to any possible decision: a global explanator $\varepsilon _g$ is made available by $c_g$. All the methods presented in this section work on tabular data.

In Reference [135], the authors propose the approach $f$ named CPAR (Classification based on Predictive Association Rules), combining the positive aspects of both associative classification and traditional rule-based classification. Indeed, following the basic idea of FOIL [94], CPAR does not generate a large set of candidates, as in associative classification, and applies a greedy approach for generating rules $c_g$ directly from training data.

In Reference [127], Wang and Rudin propose a method $f$ to extract falling rule lists $c_g$ (see Section 3.3) instead of classical rules. The falling rule lists extraction method $f$ relies on a Bayesian framework.

In Reference [66], the authors tackle the problem to build a system for medical scoring that is interpretable and characterized by high accuracy. To this end, they propose Bayesian Rule Lists (BRL) $f$ to extract the comprehensible global predictor $c_g$ as a decision list. A decision list consists of a series of if-then statements discretizing the whole feature space into a set of simple and directly interpretable decision statements.

A Bayesian approach is followed also in Reference [114]. The authors propose algorithms $f$ for learning Two-Level Boolean Rules (TLBR) in Conjunctive Normal Form or Disjunctive Normal Form $c_g$. Two formulations are proposed. The first one is an integer program whose objective function combines the total number of errors and the total number of features used in the rule. The second formulation replaces the 0-1 classification error with the Hamming distance from the current two-level rule to the closest rule that correctly classifies a sample. In Reference [62], the authors propose a method $f$ exploiting a two-level boolean rule predictor to solve the model explanation, i.e., the transparent approach is used in the reverse engineering approach to explain the black box.

Yet another type of rule is exploited in Reference [61]. Here, Lakkaraju et al. propose a framework $f$ for generating prediction models, which are both interpretable and accuratem, by extracting Interpretable Decision Sets (IDS) $c_g$, i.e., independent if-then rules. Since each rule is independently applicable, decision sets are simple, succinct, and easily to be interpreted. In particular, this approach can learn accurate, short, and non-overlapping rules covering the whole feature space.

Rule Sets are adopted in Reference [130] as comprehensible global predictor $c_g$. The authors present a Bayesian framework $f$ for learning Rule Sets. A set of parameters is provided to the user to encourage the model to have a desired size and shape to conform with a domain-specific definition of interpretability. A Rule Set consists of a small number of short rules where an instance is classified as positive if it satisfies at least one of the rules. The rule set provides reasons for predictions, and also descriptions of a particular class.

Finally, in Reference [75] an approach $f$ is designed to learn both sparse conjunctive and disjunctive clause rules from training data through a linear programming solution. The optimization formulation leads the resulting rule-based global predictor $c_g$ (1Rule) to automatically balance accuracy and interpretability.

In this section, we present the design of a set of approaches $f$ for solving the transparent box design problem returning a comprehensible predictor $c_g$ equipped with a human understandable global explanator function $\varepsilon _g$. A prototype, also referred to with the name artifact or archetype, is an object that is representative of a set of similar instances. A prototype can be an instance $x$ part of the training set $D = \lbrace X, Y \rbrace$, or it can lie anywhere in the space $\mathcal {X}^m \times \mathcal {Y}$ of the dataset $D$. Having only prototypes among the observed points is desirable for interpretability, but it can also improve the classification error. As an example of a prototype, we can consider the record minimizing the sum of the distances with all the other points of a set (like in K-Medoids) or the record generated averaging the value of the features of a set of points (like in K-Means) [118]. Different definitions and requirements to find a prototype are specified in each work.

In Reference [9], Bien et al. design the transparent Prototype Selection (PS) approach $f$ that first seeks for the best prototype (two strategies are proposed), and then assigns the points in $D$ to the label corresponding to the prototype. In particular, they face the problem of recognizing hand written digits. In this approach, every instance can be described by more than one prototype, and more than a prototype can refer to the same label (e.g., there can be more than one prototype for digit zero, more than one for digit one, etc.). The comprehensible predictor $c_g$ provides a global explanation in which every instance must have a prototype corresponding to its label in its neighborhood; no instances should have a prototype with a different label in its neighborhood, and there should be as few prototypes as possible.

Kim et al. in References [51, 52] design the Bayesian Case Model (BCM) comprehensible predictor $c_l$ able to learn prototypes by clustering the data and to learn subspaces. Each prototype is the representative sample of a given cluster, while the subspaces are sets of features that are important in identifying the cluster prototype. That is, the global explanator $\varepsilon _g$ returns a set of prototypes together with their fundamental features. Possible drawbacks of this approach are the high number of parameters (e.g., number of clusters) and various types of probability distributions that are assumed to be correct for each type of data. Reference [49] proposes an extension of BCM that exploits humans interaction to improve the prototypes. Finally, in Reference [50], the approach is further expanded to include criticisms, where a criticism is an instance that does not fit the model very well, i.e., a counter-example part of the cluster of a prototype.

With respect to prototypes and DNN, [72] already analyzed in Section 8.3 proposes a method for changing the image representations to use only information from the original image and from a generic natural image prior. This task is mainly related to image reconstruction rather than black box explanation, but it is realized with the aim of understanding the example to which the DNN $b$ is related to producing a certain prediction by realizing a sort of artificial image prototype. Therefore it is worth to highlight that there is a significant amount of work in understanding the representation of DNN by means of artifact images [48, 125, 131].

We conclude this section presenting how Reference [30] deals with artifacts in DNNs. Finding a single representative prototype by perturbation, deletion, preservation, and similar approaches has the risk of triggering artifacts of the black box. As discussed in Section 8.4, NN and DNN are known to be affected by surprising artifacts. For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the DNN from the original image, and also with respect to texts [67] inserts typos and random sentences in real texts that are classified arbitrarily. These examples demonstrate that it is possible to find particular inputs that can drive the DNN to generate nonsensical or unexpected outputs. While not all artifacts look “unnatural,” nevertheless, they form a subset of images that are sampled with negligible probability when the network is normally operated. In our opinion, two guidelines should be followed to avoid such artifacts in generating explanations for DNNs, and for every black box in general. The first one is that powerful explanations should, just like any predictor, generalize as much as possible. Second, the artifacts should not be representative of natural perturbations.

We present here a solution for the transparent box design problem that cannot be easily categorized with the previous groups. In Reference [128], Wang et al. propose a method $f$ named OT-SpAMs based on oblique tree sparse additive models for obtaining a global interpretable predictor $c_g$ as a decision tree. OT-SpAMs divides the feature space into regions using a sparse oblique tree splitting and assigns local sparse additive experts (leaf of the tree) to individual regions. Basically, OT-SpAMs passes from complicated trees/linear models to an explainable tree $\varepsilon _g$.