Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts

We consider adversarial explanation contexts where an AI decision system is used to make decisions about individuals. Prominent examples are university admissions, job and loan applications, or bail and sentencing decisions. Under existing and planned legislation, such as the EU Artificial Intelligence Act, the creator of the system ought to provide information about how the system comes to its decisions (see Section 3 below for a detailed discussion of the legal background). The creator of the system is the entity that has built the machine learning system and uses it to support decision making.¹ The creator could be a private company (such as a bank) or a public entity (such as a university). The decision subject is the person about whom the automated system makes a decision: the person who applies for a loan, or the person who applies to for university admission. After the decision has been communicated, the explanation recipient asks for an explanation, which is communicated by the explanation provider. The explanation recipient could be the decision subject herself, or an external examiner who is supposed to investigate the decisions or explanations on behalf of the decision subject or to defend her interests. The explanation provider is typically the creator of the system.²

In our technical discussion, we assume that the inputs $x \in \mathbb {R}^d$ of a decision algorithm are given in tabular form. Each dimension of the input encodes a different property of a person, for example age, income, etc. Typically, the number of dimensions d is large: persons are described by dozens or hundreds of features. A machine learning algorithm is used to learn a decision function $f: \mathbb {R}^d \rightarrow \mathbb {R}$. The resulting decision y = f(x) for input x could be a binary decision (“receives the loan” or “does not receive the loan”) or a numeric risk score on which such a decision is based, as in the often discussed COMPAS algorithm to predict recidivism risk. We focus on supervised machine learning, where f is learned based on training data consisting of pairs (x₁, y₁),..., (x_n, y_n) with x_i the training points and y_i the training labels. An explanation algorithm E is an algorithm operating on a decision function with the purpose of explaining it. We focus on local post-hoc explanation algorithms: The explanation algorithm E gets queried with a data point x and the corresponding decision y, and produces an explanation E(x, y). Internally, the algorithm has access to the decision function f, and in some cases also to the training data. The explanation E(x, y) is supposed to explain why the decision function f came to decide y for x. The explanation can be in linguistic form. For example, “The low income of Mr. Smith was relevant for the refusal of the loan” or “Mr. Smith would have received the loan had his income been 10.000 Euros higher”.

In this paper we consider local post-hoc explanation algorithms such as LIME, SHAP, and DiCE [30, 34, 41]. The explanations generated by these algorithms do not provide a global or holistic view of the decision function f but merely try to explain individual decisions y = f(x). The often-cited advantage of these algorithms is that they work, at least in principle, for any decision function [7, 41]. Different algorithms take different approaches as to what constitutes an explanation: LIME and SHAP provide feature attributions that aim to quantify the influence of the different input-features for the particular decision. Feature attributions correspond to the linguistic form “The low income of Mr. Smith was relevant for the refusal of the loan”. Another approach is to provide counterfactual explanations [60]. These explanations are based on searching for a sufficiently close or the closest alternative point x′ to the actual input point x that yields a decision y′ = f(x′) that differs from the original decision y = f(x). Comparing the two we can arrive at factors that are relevant to the decision [24]. The resulting counterfactual explanations have the linguistic form “Mr. Smith would have received the loan had his income been 10.000 Euros higher”.

The current draft of the AIA defines AI systems as “software (...) that can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with”. Generally, the AIA regulates AI on the basis of its perceived risk by introducing four different categories of AI. Most relevant to our discussion are the two categories of systems that are high-risk, as opposed to systems that are not high-risk (the remaining two categories are practices that are subject to qualified prohibitions, and a residual category of AI systems that includes law enforcement software, emotion recognition system, biometric categorisation systems and deep fakes) [54]. The stronger the risk, the heavier regulatory obligations apply, also regarding transparency and interpretability.

There are two categories of high-risk AI systems. First, AI systems that relate to products that are already subject to supranational harmonisation, namely AI systems intended to be used as a safety component of a product, which are themselves products covered by Union harmonising legislation or which are required to undergo third-party conformity assessments. Second, a list of systems that are currently considered to carry a high-risk such as, for instance, biometric identification systems, systems for the management and operation of critical infrastructure, those used in education and employment, some law enforcement systems as well as others (see further Art 3(1) of the draft AIA). Article 13 governs explainability for high-risk AI systems, which have to be “designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system's output and use it appropriately”. Furthermore, users (the entity deploying the AI) need to have access to instructions for use in an appropriate digital format that contains information about the characteristics, capabilities and limitations of performance, including information about the level of accuracy, robustness and cybersecurity, risks to health, safety or fundamental rights, specifications for the input data, expected lifetime of the AI system and necessary maintenance measures. Finally, human oversight must be ensured. These measures are designed to minimize risks to health, safety or fundamental rights. Human oversight shall either be (i) identified and built into the system by the provider before it is placed on the market or put into service, or (ii) identified by the provider before the system is placed on the market or put into service but only implemented by the user. In its current version, the AIA would thus require that high-risk AI systems are sufficiently transparent to enable the interpretation of the system's output. Is this an explainability obligation? Recital 47 sheds some light on how to interpret these notions. It specifies that high-risk AI systems should be transparent to a “certain degree” to “address the opacity that may make certain AI systems incomprehensible to or too complex for natural persons”. To this end, users “should be able to interpret the system output and use it appropriately” through the provision of “relevant documentation and instructions of use”. This does not read like an obligation to make systems explainable in the sense that the way in which data has been processed must be entirely traceable. Rather, the AIA would require that an “interpretation” of the output must be facilitated through sufficient transparency. Importantly, this does not necessarily seem to imply that an absolute truth must be identified post-hoc (see Sections 4.1 and 4.2 below) but rather the overall functioning of the system and how it comes to an output. The draft AIA leaves open the question of what transparency and interpretability imply from a technical perspective. This certainly includes the elements listed in its Article 16 such as technical documentation, keeping logs or quality management systems. Article 13 leaves open whether there are additional requirements and what, exactly, interpretability requires from a technical perspective. If input data ought to be entirely traceable, “black-box” systems cannot be used in high-risk applications. This highlights that it is important to think about the objectives of transparency and explainability. If these can be achieved through alternative means, excluding black-box systems such as deep neural networks from high-risk scenarios (such as healthcare as devices falling under the Medical Devices Regulation qualify as high-risk) might unduly hinder innovation in important domains.

Article 52 AIA creates some general transparency obligations for AI systems that are not high-risk. These are general disclosure obligations such as to (i) inform users that they are interacting with an AI system unless this is obvious from context, (ii) users of an emotion recognition system or biometric categorization system shall inform natural persons exposed thereto, (iii) deep fakes must be disclosed as such. Some exceptions apply where the AI is used in the context of law enforcement. These are thus obligations of transparency that require disclosure that AI is used, as opposed to how it is used.

To summarize, the draft AIA would thus not, in its current form, create a general explainability obligation for machine learning systems. Such an obligation clearly is not foreseen in relation to AI systems that are not qualified as high-risk. Arguably, there is also no explainability obligation in relation to high-risk AI systems. Rather, what is required is transparency of the system's functioning and output generation. This transparency must make these elements interpretable but not necessarily amount to the provision of an explanation as it is commonly understood in computer science.

The GDPR creates some general transparency requirements that form part of the data controller's (the entity that determines the purposes and means of processing) general informational obligations vis-à-vis the data subject (the natural person that personal data relates to). In addition, it also contains a specific regime for “solely automated data processing”. In contrast to the draft AIA, which creates vague obligations resting on the user, the GDPR creates specific rights for the individual subjected to such decisions.

Article 13 requires that data controllers provide specific information to data subjects where personal data is collected from them at the time of collection such as whether “automated decision-making” is used, and, if so, provide “meaningful information about the logic involved ³, as well as the significance and the envisaged consequences of such processing”. Article 14(1)(h) creates the same obligation in cases where data is not directly collected from the data subject. Pursuant to Recital 62 this information does not have to be provided where it is redundant, or where compliance proves impossible or involves a disproportionate effort. The same wording can also be found in Article 15, which deals with the data subject's right to access data. Whereas Articles 13 and 14 relate to the pre-processing stage, data subjects can exercise their rights under Article 15 at any time, including after processing has taken place. This raises the question of whether – despite the identical wording of these provisions – Article 15 may substantively require something different when referring to the “logic” of the automated decision-making process.

There is no general right to an explanation under the GDPR. Some explainability requirements may, however, arise in respect of machine learning algorithms that produce legal effect or similarly significantly affect a data subject. Article 22 creates a qualified prohibition of “solely automated data processing”, including profiling. This implies that such techniques can only be used in some circumstances, namely (i) where necessary to enter into or perform a contract between the data subject and controller, (ii) where it is authorized by law or where the data subject has provided explicit consent. In these circumstances automated processing can take place, but the data subject has the right to human intervention and to express her point of view and to contest the decision. Recital 71 mentions an additional element, namely that the data subject has the right “to obtain an explanation” after human review of the decision “and to challenge this decision”.⁴ Recitals, however, do not have the same legally binding force as the text of the GDPR itself.

Over the past years there has been a vivid academic debate around whether the reference to “an explanation” in Recital 71 amounts to a “right to an explanation” that data subjects can exercise via-à-vis controllers [59] [32] [45] [14]. The Article 29 Working Party's guidance suggests that Article 22, read in conjunction with Recital 71, should be understood to require that controllers (i) tell data subjects that they are engaging in automated decision making, (ii) deliver meaningful information about the logic, and (iii) explain the processing's significance and envisaged consequences. The information provided should include details about the categories of data; why data is seen as pertinent; how profiles are built; why the profile is relevant for the decision-making process and how it is used to reach a decision about the data subject. The last three criteria appear to apply to profiling only [36]. Information with respect to the “logic” means “simple ways to tell the data subject about the rationale behind, or the criteria relied on in reaching the decision”. What is required is “not necessarily a complex explanation of the algorithms used or disclosure of the full algorithm”. Nonetheless, the information transmitted to the data subject should be sufficiently comprehensive to “understand the reasons for the decision”. Thus, an explanation of algorithms or disclosure of the full algorithm isn't “necessarily” required and that the controller ought to find “simple ways to tell the data subject about the rationale behind, or the criteria relied on in reaching the decision”. Unfortunately, this guidance leaves a lot of room for doubt regarding what exactly is required of controllers. In any event the GDPR does not create a general right to an explanation but applies only to automated decision-making that legally affect the data subject or have similarly significant effects on them.

While there is a persistent myth that EU law requires that all decisions based on AI are “explainable” our analysis has painted a more nuanced picture. First, there is no overarching explainability norm that would apply to any usage of AI. To what degree secondary law requires explanations has not been authoritatively settled. Ultimately, the Court of Justice of the European Union will need to settle this question in respect of the GDPR. Concerning the draft AIA, however, legislators should clarify in the final text whether explainability is a free-standing legal obligation in respect of high-risk AI systems or whether it should rather be understood as a sub-component of transparency. As shown above, it is indeed possible to read references to explainability as elements of the broader transparency obligation. Article 13 AIA is explicitly about transparency, but the reference that this transparency must allow users to “interpret the system's output” has been understood as an explainability obligation by some. Further iterations should clarify the link between transparency and explainability to enhance legal certainty. An analysis of the history behind the AIA confirms the lack of precision of the AIA itself. The EU High Level Expert Group on AI's report on the one hand portrayed explainability as a component of transparency. On the other hand, it repeatedly referred to another concept, “explicability”, which was introduced as an ethical principle and as the “procedural dimension” of fairness. In contrast, the AIA White Paper made no reference to explainability other than to mention that symbolic reasoning could help make deep neural networks more explainable. This part of the AIA legislative history underlines the lack of consensus about what exactly explainability is. Similarly, the GDPR could also be read as referring to explainability as a sub-component of transparency. Articles 12-15 derive from the core data protection principle of transparency in Article 5(1)(a) and likewise, one reading of Article 22 in conjunction with relevant recitals could also be understood as a more general transparency rather than explainability obligation.

This, of course, raises the question of what transparency means and what it should enable. There is broad consensus that the GDPR requires that decisions reached through automated decision making be justifiable. Indeed, Hildebrandt has highlighted that data protection requires “the justification of such decision-making rather than an explanation in the sense of its heuristics” (p. 113 in [18]). Kamimski and Urban deem that justification should enable “understanding, revealing and making challengeable the normative grounds of a decision” (p. 1980 in [21]). Wachter, Mittelstadt and Russell have argued that explainability is ultimately designed to help the data subject understand, contest and alter decisions and that this could also be achieved by counterfactual explanations [60]. If explainability is merely one means of achieving transparency, there needs to be a more thorough discussion as to what other, alternative, means of achieving transparency there are, particularly in situations where explainability strictu sensu proves impossible. Considering the lack of consensus as to how the legislative texts of the AIA and the GDPR ought to be interpreted and applied in practice, it is helpful to consider their underlying objectives.

The vague formulation of explainability rights, coupled with uncertainty regarding their function makes it legitimate to ask whether explanations serve any meaningful purpose. Indeed, as Edwards and Veale [14] have argued, “the search for a legally enforceable right to an explanation may be at best distracting and at worst nurture a new kind of transparency fallacy”. This is essentially a warning that if explainability obligations just become a box-ticking exercise, they might give a misleading appearance of compliance rather than to be of any real value to the decision subject. In addition, explainability rights in the GDPR inevitably also suffer from the general shortcomings of the low enforcement of the GDPR.

In order to better understand the above-examined norms we propose to consider their underlying objectives. Before discussing legislative history let us recapitulate what philosophers have identified as main objectives for algorithmic explanations.⁵ One major motivation for explainability of AI systems is the hope that this may foster trust in these systems [10, 26, 35, 57]. This has been called the “Explainability-Trust” hypothesis [22].⁶ The hypothesis is controversial, and it is not exactly clear how explanations would induce trust. The underlying rationale seems to rest on an analogy with human interactions. Consider decisions made by human experts. When the decision doesn't satisfy us, we are drawn to ask for an additional explanation. Given such an explanation, we may check whether it conforms to our expectations about good decision making. If so, this may be a ground for further trusting the decision maker. This is not a one-shot process, but an ever evolving interaction on a long term time-scale. We tend to trust a person that proved repeatedly to predict correctly, make good decisions, or provide well informed explanations. The trust raising potential of an explanation however requires that we can submit explanations and decisions to tests, possibly by delegating it to other experts. The trust raising potential of a single explanation thus presupposes that the explanation provider stays in the information-exchange on the long run: only then does she have an incentive to provide a correct explanation, since an incorrect one would lead to a loss of trust in the long run but not in a one-shot exchange. If an algorithm rather than a human expert makes a decision, we might have similar expectations. We would like to engage in a similar information exchange with an algorithm as we engage with humans. The demand for an explanation is then a demand for a piece of communicative interaction. The hope that this builds trust stems from the intuition that the interaction with the algorithm is similar to the interaction among humans, as depicted above. This assumption may however fail either because the algorithmic explanations cannot be submitted to sensible tests or because the exchange is one-shot and not long run. In the first case, explanations loose their trust raising potential. In the second, the explanation provider may not have the incentive to tell the truth. A second implicit motivation for explainability stems from the idea that information provided by explanations can be used to perform actions, and may in fact be needed for such actions. In the adversarial setting, a data subject might want to use an explanation to contest a decision [7, 60], by claiming, or arguing that the decision is not right, not good, or not fair. The data subject might also want to use the explanation for recourse, in order to do better next time [4, 55, 60] (see also [2, 29]).⁷ But such explanations are only of value when true or correct. A false explanation will not help in doing better next time, and may even be devised such as to render a decision incontestable.

The two motivations from philosophy — building trust and enabling recipients to act — can also be found as objectives in the legal texts. The EU High Level Expert Group on AI described explainability as one tool to achieve trust in AI systems [35].⁸ The AIA provides that explainability norms are designed to allow users to fully understand the capacities and limitations of high-risk systems, leading again to trust. Partly related to trust, one can understand explainability as a tool for risk management, in line with the AIA's overall risk-based approach. Indeed, for high-risk AI systems, transparency must be ensured by monitoring the system's operation, detect signs of anomalies, dysfunctions and unexpected performance in order to counteract automation bias or to potentially intervene in the system (the idea of a “stop button”). The European Commission White Paper also emphasized the risk-based approach and stressed that due to the potential scale of AI systems [11]: a hidden bias or an incorrect assumption of an AI system, say deciding on tens of thousands of university admission decisions, will have a large systemic effect. This differentiates large-scale AI systems from human decision-making systems. In philosophy explanations are considered as a tool towards future actions. Similarly, the legal discussion also portrays explainability as an enabling right. The High-Level Expert Group on AI has drawn attention to the fact that to be able to contest decisions, they must be traceable. Also outside the AIA and the GDPR, explainability serves a related purpose. In consumer protection law, explainability is linked to the unequal power dynamics between the business and the consumer. In the public administration, it has been argued that being subjected to an intransparent black-box decision would undermine human dignity and is also to be avoided, unlike in the private sector, individuals cannot vote with their feet and go elsewhere.

Overall, the motivations for explanations seem to presuppose that such explanations are true or correct. Only then does a single explanation raise trust, and only then can an explanation be used to perform the intended actions, such as contesting or recourse. We will, however, see in the next section that this truth-presupposition for explanations fails in adversarial scenarios of algorithmic post-hoc explanations.

Learning and explanation algorithms only have access to a coarse-grained description of the real world. Their vocabulary is restricted to certain features, and possible relations between them. The “experience” of such algorithms given by the finite training data is formulated in the restricted vocabulary and provides only a small window to the world. Overall, the algorithm's representation of the real world is coarse-grained and incomplete.⁹ The learning algorithm just sees features and training labels. The explanation algorithm, additionally, sees the learning algorithm's association between input and output. This is what we call “the world of the explanation algorithm”, and this is all what it can exploit. As a consequence, all the explanation algorithm could talk about are geometric properties in the world of the algorithm: distances of points to the decision surface, proximity between points, their true or predicted labels, the gradient of the decision function at a point, the necessary change of a feature to change the decision, etc. Although a true explanation for a decision might exist in the real world, it might not be represented in the data or other aspects of the algorithm's world, which could thus not provide any such explanation. This is even the case in a cooperative setting. Consider the example of a medical diagnosis of a disease for which a true (say, causal) explanation exists in the real world. If the learning algorithm was trained on feature-based data such as age, blood pressure, etc, the explanation algorithm could suggest that age was the cause. However, in reality the cause for the disease may not be age, but rather a smoking habit that was not represented in the data. So even if a true explanation exists (say, a cause) this may neither be identifiable nor expressible by the explanation algorithm.

Even within the limited world that the explanation algorithm has access to, a “true internal reason” why the learned decision function comes to a certain decision does generally not exist. This is particularly the case for complicated black-box functions. Even machine learning experts digging into the learning algorithm or properties of the function could not reveal a unique true reason. All we can do is to provide vague approximations of how the algorithm arrives at its decision, by summarizing which features contributed how much to the decision (the approach of LIME and SHAP), or whether a change in some features would alter the decision (the approach of counterfactual explanations). For example, in the case of a loan rejection, we might want to know whether it was rather our low income or our postal code which determined the decision, and whether we could change something about the decision, if in the future we had a higher income or moved to another area. However, these explanation attempts are all subject to choices. A mathematically unique way to determine how much each feature of a complicated black-box function contributed to the decision does not exist. Consequently, all feature attribution methods rely on particular assumptions and mechanisms in order to construct explanations: LIME, for example, looks at the gradient of the decision function at the point to be explained [15, 41]. SHAP compares the point with other datapoints from a reference population [16, 30]. Yet another approach would be to re-train the classifier on subsets of features or to use counterfactual feature importance, where one looks at the distance to the decision surface in various directions. All these mechanisms and choices seem plausible but, as we will see in the next section, they all deliver different explanations.

Different explanation algorithms lead to different explanations [25]. This is true even if the algorithms have access to exactly the same information (the geometry of the data, the learned decision function, etc). In an adversarial context, this is problematic because it means that the creator of the system can modify the explanations by choosing a particular explanation algorithm. In practice, different explanation algorithms lead to different explanations even on the most simple machine learning problems. In high dimensions, that is in real-world problems, the difference between the explanations obtained from two different explanation algorithms can be so significant that the explanations are entirely different. This is illustrated in Figure 1. The figure depicts the feature attribution explanations that four different explanation algorithms determined for the same individual. From the difference between the four panels in Figure 1 it is quite clear that different explanation algorithms can lead to markedly different explanations, even if they all attempt to explain the same decision for the same individual.¹⁰ Details on the machine learning problem, dataset and explanation algorithms can be found in the supplement.

That different explanation algorithms lead to different explanations is also true for counterfactual explanation methods [34, 60]. Indeed, there is a variety of ways in which the optimization problem can be set up, which in turn leads to different explanations. However, already a single counterfactual explanation method can lead to a large number of counterfactual explanations. In a cooperative context, being able to generate many different counterfactual explanations for the same individual can be beneficial [34]. In an adversarial context this is problematic because there is no principled way to choose among different counterfactual explanations, and the adversary is again awarded considerable discretion to determine explanations. In realistic, high-dimensional applications, the number of potential counterfactual explanations can quickly become very large. Let us illustrate this point on the German Credit Dataset. The German Credit Dataset is a 20-dimensional dataset with features on credit history and personal characteristic. The task is to predict credit risk in binary form. How many different counterfactual explanations exist for a single individual? With a common black-box decision function, more than 100 different counterfactual explanations exist for each individual.

At its core, the fundamental difficulty of explainable machine learning is then the same as in other fields of unsupervised learning: the lack of a ground truth explanation impedes the development of an algorithmic framework to automatically evaluate explanations. Every explanation algorithm needs to make assumptions about which properties of the decision function it seeks to highlight. As a result, it is possible to develop sanity checks for explanation algorithms and exclude unreasonable approaches [3, 9], but not to discern whether any of two post-hoc explanations is “more correct”, which would be equivalent to discussing whether any of two different clusterings is “more correct” [58].

Even for a single explanation algorithm, there can be many different parameter choices that all lead to different explanations. LIME explanations, for example, depend on the bandwidth and the number of perturbations [15, 27, 46]. The uniqueness properties of Shapley values non-withstanding, there is a multiplicity of ways in which Shapley values can be operationalized to generate explanations [51]. Counterfactual explanation algorithms depend on the underlying metric chosen to represent closeness (e.g. Euclidean distance vs. L1-norm)¹¹ as well as additional hyperparameters to weight-off between closeness and prediction, and, at least in principle, any number of additional penalty terms [34]. In certain cases, it might be possible to come up with good default parameter choices. For example, recent work has demonstrated how to choose the bandwidth parameter of LIME in a principled way or quantify uncertainty in the resulting explanations [27, 46, 64]. It is also possible to exclude explanation algorithms and parametrizations that are completely unreasonable, for example because they are not sensitive to the decision function [3, 9]. This nevertheless leaves an ever-increasing number of plausible explanation algorithms and corresponding parametrizations. Quite generally, different explanation algorithms vary among many different dimensions, and there is an ever increasing number of suggestions as to how black-box functions might be explained. This can be seen, for example, in the recent work of Covert et al. [12], who summarize 25 existing methods in a unified framework. As already discussed above, there are no fundamental reasons that impede us from using any particular method.¹²

Even if we fix a particular explanation method and its parameters, the generated explanations still depend on the exact shape of the learned decision boundary. In high dimensions, there are often many different black-box functions that solve a particular classification problem to a desired accuracy, that is they represent the data sufficiently well. However, these functions often lead to different explanations. To a certain extent, we may say that the exact shape of the learned decision boundary is arbitrary, but since the explanations depend on it, these turn out to be arbitrary as well. One of the reasons for the sensitivity of the explanation to the function's shape is that many explanation methods evaluate the function f at datapoints that are outside the data distribution or at points that are unlike most points from the data distribution. In the adversarial scenario, this is problematic because the adversary can freely modify the values of the function f outside the data distribution without changing the classification behavior. Recent work has demonstrated that this property can be used to explicitly manipulate and attack explanation methods [47, 48]. But even without explicit attacks, there are many different choices, in particular hyperparameter and architecture choices, that influence the shape of the decision boundary, and thus the resulting explanations. For an external examiner, this presents a challenging problem: while certain explicit attacks on explanation methods could in principle be detected through code review (see also Section 5.2), it is far less clear how one would argue about choosing one classifier over another, or any particular choice of hyperparameters. This problem is illustrated in Figure 3. Here, we solved the same machine learning problem both with linear regression and a random forest. The two methods have comparable performance on the test set, where 94% of their predictions agree. Nevertheless, the explanations obtained for the two different decision functions can be quite different – even for points that receive the same prediction.

Turning to counterfactual explanations, it is well-known that these depend on the exact shape of the decision boundary. Let us give an example, again using the German Credit Dataset. Consider two different decision functions, a gradient boosted tree and logistic regression. If we generate a number of diverse counterfactual explanations [34] for a typical individual with respect to one decision function, are these also counterfactual explanations with respect to the other decision function (at least as long as both functions arrive at the same decision)? In this simple experiment less than 50% of counterfactual explanations that work for the gradient boosted tree also work for logistic regression. As discussed above, the fact that the explanations depend on the exact shape of the decision boundary is problematic because it allows the creator of the system to influence the resulting explanations. The particular choice of the decision function can even determine whether certain types of counterfactual explanations exist at all. Let us give an example on the Wisconsin Breast Cancer Dataset. To demonstrate the dependence on the decision boundary, we consider again two different decision functions, linear regression and a random forest. For linear regression, there exist a large number of counterfactual explanations that modify only a single variable. For the random forest, it is impossible to find any such counterfactual explanations. This is despite the fact that both classifiers exhibit similarly low test error.

In recent years, there has been an increased focus on the composition of datasets, for example on the representation of different sociodemographic groups in machine learning datasets [6, 37]. In many real-world problems such as credit lending, the criteria for choosing an appropriate dataset are not clear. In both cooperative and adversarial contexts, the creator of the system has to make numerous choices, many of which can have significant effects on both the shape of the learned decision boundary and the generated explanations. For example, Anders et al. [5] have shown that gradient-based explanations can be manipulated by adding additional variables to the dataset. In this section, we highlight the additional role that the dataset can have on algorithmic explanations, even when keeping the learned decision boundary constant. Indeed, while some explanation algorithms such as LIME only rely on the learned decision boundary, other methods such as SHAP and some counterfactual explanation methods make additional use of the data in order to generate explanations. The relevant dataset could be the training data, but it could also be a different dataset. We refer to it as the reference dataset. While the usage of such a dataset to generate explanations can be seen as a remedy to the vagaries of high dimensions, or as a possibility to generate counterfactual explanations that look like they come from the data, this approach is problematic as long as the adversary determines the composition of the dataset. The reason is that whether certain datapoints are included in the dataset or not can determine whether an explanation algorithm provides one or another explanation. Figure 4 illustrates this with a simple example: By deciding between two different reference datasets, one can effectively decide whether one ore another feature was relevant to the decision.

It is extremely important to understand that an explanation algorithm is based on many human choices that are shaped by human objectives and preferences. While many choices are plausible, there is no objective reason to prefer one algorithm over the other, or one explanation over the other. Apart from the explanation algorithm and its particular parameters, explanations are influenced by human choices such as the selection of the classifier and the composition of the dataset. In adversarial contexts it implies that the adversary can choose, among many different plausible explanations, one that suits their incentives. This complicated situation makes it particularly difficult for external observers, including judges and regulatory bodies, to determine whether an explanation is acceptable. Explanation algorithms appear to provide objective explanations, yet as explained above this is not the case (compare Section 4.2).

To determine whether the adversary's explanations actually correspond to the used decision function f instead of being arbitrary justifications not related to the decision process, the examiner needs to be able to query the decision function and the generated explanations.¹³ This includes a fair amount of related knowledge, such as which variables are input to the algorithm, but excludes explicit access to the decision function, explanation algorithm, source code and training dataset. A related but slightly more limited version of this scenario arises when individuals jointly collect the decisions and explanations from the creator of the system. In this minimalist scenario, the examiner can validate the internal consistency of the provided explanations. Researchers have proposed a number of criteria that the examiner can test for such as faithfulness to the model, robustness to local perturbations, as well as necessity and sufficiency notions for individual feature attributions [3, 24, 56]. The examiner might also want to perform tests as to whether the provided explanations have been manipulated [48]. More importantly however, even just with the ability to query the decision function, the examiner can ignore the explanations and directly investigate the decision function for problematic properties. For example, the examiner could conduct a systematic evaluation of, say, fairness metrics such as equal opportunity and demographic parity, based on an independent reference dataset of her choice (see [6] for these and other notions of fairness and discrimination). Indeed, because the adversary designing the explanation algorithm has no interest in choosing explanations that highlight any discriminatory behavior of the decision algorithm, the examiner is well-advised to simply ignore the explanations and test the decision algorithm directly. Although such tests might be similar to certain explanation algorithms, what is important is that the examiner (as opposed to the creator) designs and implements them. Note that we are not saying that the minimalist scenario actually allows the examiner to assess all legally relevant properties of the decision function. What exactly can be assessed with querying access is a question that still requires more research. Our point is that once we have querying access, the explanations are useless.

At the opposite end of the minimalist scenario is the fully transparent scenario where the examiner is allowed to investigate the decision function, source code and training data. An examiner could then scrutinize whether the explanation algorithms have been implemented according to the state of the art with sensible parameter choices. This directly rules out the possibility for the creator of the system to manipulate explanations. Are post-hoc explanations useful in the transparent scenario, perhaps because the examiner now has the tools to verify whether the adversary has chosen the “correct” explanations? As we have already discussed above, the problem is that there is no notion of “correct” explanation (Sections 4.2 and 4.3). Thus, except for notions of internal consistency [3, 24], there is, in general, nothing the examiner can say about the explanations. Another issue, already observed in Sections 4.5 and 4.6, are hyperparameter choices and decisions regarding the composition of the dataset. For these decisions, it is highly non-trivial to come up with uniquely reasonable defaults: If the adversary has found a particular neural network architecture with hyperparameters that generalize well on the adversary's own dataset, how exactly could the examiner argue that this is inappropriate? Nevertheless, all of these choices can influence the resulting explanations, even if we fix a particular explanation algorithm. Of course, the examiner could scrutinize the source code, re-train the system with different parameters, perform tests on the data, and generate alternative explanations. Some have argued that this might be sufficient in order to assess a variety of legal requirements [23]. While we think that more research is needed on what can be realistically achieved in the fully transparent scenario, it is quite clear that the examiner can, at least in principle, perform a variety of powerful tests (whether this is achievable in practice, based on the limited resources of an examiner, is yet a different story). At any rate, just as in the minimalist scenario, the examiner is well-advised to examine and test the system on her own, and to ignore the explanations provided by the adversary creator.