Algorithmic Transparency via Quantitative

Input Influence

Anupam Datta, Shayak Sen, and Yair Zick

Abstract Algorithmic systems that employ machine learning are often opaque—
it is difficult to explain why a certain decision was made. We present a formal
foundation to improve the transparency of such decision-making systems. Specif-
ically, we introduce a family of Quantitative Input Influence (QII) measures that
capture the degree of input influence on system outputs. These measures provide a
foundation for the design of transparency reports that accompany system decisions
(e.g., explaining a specific credit decision) and for testing tools useful for internal
and external oversight (e.g., to detect algorithmic discrimination). Distinctively,
our causal QII measures carefully account for correlated inputs while measuring
influence. They support a general class of transparency queries and can, in
particular, explain decisions about individuals and groups. Finally, since single
inputs may not always have high influence, the QII measures also quantify the
joint influence of a set of inputs (e.g., age and income) on outcomes (e.g. loan
decisions) and the average marginal influence of individual inputs within such a
set (e.g., income) using principled aggregation measures, such as the Shapley value,
previously applied to measure influence in voting.

Abbreviation

QII Quantitative Input Influence

A. Datta () • S. Sen

Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, USA
e-mail: danupam@cmu.edu; shayaks@cmu.edu
Y. Zick
School of Computing, National University of Singapore, 21 Lower Kent Ridge Rd, Singapore,
Singapore
e-mail: zick@comp.nus.edu.sg

© Springer International Publishing AG 2017 71

T. Cerquitelli et al. (eds.), Transparent Data Mining for Big and Small Data,
Studies in Big Data 11, DOI 10.1007/978-3-319-54024-5_4
72 A. Datta et al.

Symbols

 Influence
A Algorithm
QA Quantity of Interest

1 Introduction

Algorithmic decision-making systems that employ machine learning and related

statistical methods are ubiquitous. They drive decisions in sectors as diverse as
Web services, health-care, education, insurance, law enforcement and defense [2,
5, 6, 31, 34]. Yet their decision-making processes are often opaque. Algorithmic
transparency is an emerging research area aimed at explaining decisions made by
algorithmic systems.
The call for algorithmic transparency has grown in intensity as public and private
sector organizations increasingly use large volumes of personal information and
complex data analytics systems for decision-making [35]. Algorithmic transparency
is important for several reasons. First, it is essential to enable identification of harms,
such as discrimination, introduced by algorithmic decision-making (e.g., high
interest credit cards targeted to protected groups) and to hold entities in the decision-
making chain accountable for such practices. This form of accountability can
incentivize entities to adopt appropriate corrective measures. Second, transparency
can help detect errors in input data which resulted in an adverse decision (e.g.,
incorrect information in a user’s profile because of which insurance or credit was
denied). Such errors can then be corrected. Third, by explaining why an adverse
decision was made, algorithmic transparency can provide guidance on how to
reverse it (e.g., by identifying a specific factor in the credit profile that needs to
be improved).
Our Goal While the importance of algorithmic transparency is recognized, work on
computational foundations for this research area has been limited. In this direction,
this chapter focuses on a concrete algorithmic transparency question:
How can we measure the influence of inputs (or features) on decisions made by an
algorithmic system about individuals or groups of individuals?

We wish to inform the design of transparency reports, which include answers

to transparency queries of this form. To be concrete, let us consider a predictive
policing system that forecasts future criminal activity based on historical data;
individuals high on the list receive visits from the police. An individual who receives
a visit from the police due to such an algorithmic decision can reasonably expect an
explanation as to why. One possible explanation would be to provide a complete
breakdown of the decision making process, or the source code of the program.
However, this is unrealistic. Decision making algorithms are often very complex
and protected as intellectual property.
Algorithmic Transparency via Quantitative Input Influence 73

Instead of releasing the entire source code of the decision making algorithm,
our proposed transparency report provides answers to personalized transparency
queries about the influence of various inputs (or features), such as race or recent
criminal history, on the system’s decision. These transparency reports can in turn be
studied by oversight agencies or the general public in order to gain much needed
insights into the use of sensitive user information (such as race or gender) by the
decision making algorithm; as we later discuss, transparency reports can also point
out potential systematic differences between groups of individuals (e.g. race-based
discrimination). Our transparency reports can also be used to identify harms and
errors in input data, and guide the design of better decision making algorithms.
Our Model We focus on a setting where a transparency report is generated with
black-box access to the decision-making system1 and knowledge of the input dataset
on which it operates. This setting models the kind of access available to a private or
public sector entity that proactively publishes transparency reports. It also models a
useful level of access required for internal or external oversight of such systems to
identify harms introduced by them. For the former use case, our approach provides
a basis for design of transparency mechanisms; for the latter, it provides a formal
basis for testing. Returning to our predictive policing example, the law enforcement
agency that employs such a system could proactively publish transparency reports,
and test the system for early detection of harms like race-based discrimination. An
oversight agency could also use transparency reports for post hoc identification of
harms.
Our Approach We formalize transparency reports by introducing a family of
Quantitative Input Influence (QII) measures that capture the degree of influence
of inputs on outputs of the system. Three desiderata drive the definitions of these
measures.
First, we seek a formalization of a general class of transparency reports that
allows us to answer many useful transparency queries related to input influence,
including but not limited to the example forms described above about the system’s
decisions about individuals and groups.
Second, we seek input influence measures that appropriately account for corre-
lated inputs—a common case for our target applications. For example, consider a
system that assists in hiring decisions for a moving company. Gender and the ability
to lift heavy weights are inputs to the system. They are positively correlated with
one another and with the hiring decisions. Yet transparency into whether the system
uses weight lifting ability or gender in making its decisions (and to what degree)
has substantive implications for determining if it is engaging in discrimination (the
business necessity defense could apply in the former case [16]). This observation
makes us look beyond correlation coefficients and other associative measures.

1
By “black-box access to the decision-making system” we mean a typical setting of software
testing with complete control of inputs to the system and full observability of the outputs.
74 A. Datta et al.

Third, we seek measures that appropriately quantify input influence in settings

where any input by itself does not have significant influence on outcomes but a
set of inputs does. In such cases, we seek measures of the joint influence of a
set of inputs (e.g., age and income) on a system’s decision (e.g., to serve a high-
paying job ad). We also seek measures of the marginal influence of an input within
such a set (e.g., age) on the decision. This notion allows us to provide finer-grained
transparency about the relative importance of individual inputs within the set (e.g.,
age vs. income) in the system’s decision.
We achieve the first desideratum by formalizing a notion of a quantity of interest.
A transparency query measures the influence of an input on a quantity of interest,
which represents a property of the system for a given input distribution. Our
formalization supports a wide range of statistical properties including probabilities
of various outcomes in the output distribution and probabilities of output distribution
outcomes conditioned on input distribution events. Examples of quantities of
interest include the conditional probability of an outcome for a particular individual
or group, and the ratio of conditional probabilities for an outcome for two different
groups (a metric used as evidence of disparate impact under discrimination law in
the US [16]).
We achieve the second desideratum by formalizing causal QII measures. These
measures (called Unary QII) model the difference in the quantity of interest when
the system operates over two related input distributions—the real distribution and
a hypothetical (or counterfactual) distribution that is constructed from the real
distribution in a specific way to account for correlations among inputs. Specifically,
if we are interested in measuring the influence of an input on a quantity of interest
of the system behavior, we construct the hypothetical distribution by retaining the
marginal distribution over all other inputs and sampling the input of interest from
its prior distribution. This choice breaks the correlations between this input and all
other inputs and thus lets us measure the influence of this input on the quantity of
interest, independently of other correlated inputs. Revisiting our moving company
hiring example, if the system makes decisions only using the weightlifting ability
of applicants, the influence of gender will be zero on the ratio of conditional
probabilities of being hired for males and females.
We achieve the third desideratum in two steps. First, we define a notion of joint
influence of a set of inputs (called Set QII) via a natural generalization of the
definition of the hypothetical distribution in the Unary QII definition. Second, we
define a family of Marginal QII measures that model the difference on the quantity
of interest as we consider sets with and without the specific input whose marginal
influence we want to measure. Depending on the application, we may pick these
sets in different ways, thus motivating several different measures. For example, we
could fix a set of inputs and ask about the marginal influence of any given input in
that set on the quantity of interest. Alternatively, we may be interested in the average
marginal influence of an input when it belongs to one of several different sets that
significantly affect the quantity of interest. We consider several marginal influence
aggregation measures from cooperative game theory originally developed in the
context of influence measurement in voting scenarios and discuss their applicability
in our setting. We also build on that literature to present an efficient approximate
algorithm for computing these measures.
Algorithmic Transparency via Quantitative Input Influence 75

Recognizing that different forms of transparency reports may be appropriate

for different settings, we generalize our QII measures to be parametric in its key
elements: the intervention used to construct the hypothetical input distribution; the
quantity of interest; and the difference measure used to quantify the distance in
the quantity of interest when the system operates over the real and hypothetical
input distributions. This generalized definition provides a structure for exploring the
design space of transparency reports.
We illustrate the utility of the QII framework by developing two machine learning
applications on real datasets: an income classification application based on the
benchmark adult dataset [27], and a predictive policing application based on the
National Longitudinal Survey of Youth [32]. In particular, we analyze transparency
reports for these two applications to demonstrate how QII can provide insights into
individual and group classification outcomes.

2 Unary QII

Consider the situation discussed in the introduction, where an automated system

assists in hiring decisions for a moving company. The input features used by this
classification system are : Age, Gender, Weight Lifting Ability, Marital Status and
Education. Suppose that, as before, weight lifting ability is strongly correlated with
gender (with men having better overall lifting ability than women). One particular
question that an analyst may want to ask is: “What is the influence of the input
Gender on positive classification for women?”. The analyst observes that 20% of
women are approved according to his classifier. Then, she replaces every woman’s
field for gender with a random value, and notices that the number of women
approved does not change. In other words, an intervention on the Gender variable
does not cause a significant change in the classification outcome. Repeating this
process with Weight Lifting Ability results in a 20% increase in women’s hiring.
Therefore, she concludes that for this classifier, Weight Lifting Ability has more
influence on positive classification for women than Gender.
By breaking correlations between gender and weight lifting ability, we are able
to establish a causal relationship between the outcome of the classifier and the
inputs. We are able to identify that despite the strong correlation between a negative
classification outcome for women, the feature Gender was not a cause of this
outcome.
A mechanism for breaking correlations to identify causal effects is called an
intervention in the literature on causality. In this chapter, we introduce a particular
randomized intervention, and we describe in Sect. 3 how other interventions may be
useful depending on the causal question asked of the system.
In the example above, instead of the entire population of women, the analyst may
be interested in a particular individual, say Alice, and ask “What is the influence
of the input Gender on Alice’s rejection?”. The analyst answers this question by
applying the same intervention, but only on Alice’s data, and observes that keeping
every other input feature fixed and replacing Gender with a random value has no
effect on the outcome.
76 A. Datta et al.

In general, QII supports reasoning at such different scales by being parametric

in a quantity of interest. A quantity of interest is a statistic of the system, which
represents the subject of the causal question being asked. For example, a suitable
quantity of interest corresponding to the question “What is the influence of the input
Gender on positive classification for women?” is average rate of positive outcomes
for women. Similarly, a suitable quantity of interest for “What is the influence of the
input Gender on Alice’s rejection?” is just Alice’s classification outcome. QII also
supports reasoning about more complex statistics such as ones which measure group
disparity. For example, the analyst observes that the rate of positive classification is
20% higher in men than in women, and wishes to ask “What is the influence of
Income on men getting more positive classification than women?”. In this case, a
natural quantity of interest is the difference in the classification rates of men and
women.
We now present the formal definition of QII while omitting some technical details
for which we refer the interested reader to the full technical paper [14].
We are given an algorithm A . A operates on inputs (also referred to as features
for ML systems),QN D f1; : : : ; ng. Every i 2 N, can take on various states, given by
Xi . We let X D i2N Xi be the set of possible feature state vectors, let Z be the set
of possible outputs of A . For a vector x 2 X and set of inputs S  N, xjS denotes
the vector of inputs in S. Inputs are assumed to be drawn from some probability
distribution that represents the population. The random variable X represents a
feature state vector drawn from this probability distribution.
For a random variable X representing the feature vector, a randomized interven-
tion on feature i is denoted by the random variable Xi Ui , where Ui is new random
variable that is independently drawn from the prior distribution of feature i. For
example, to randomly intervene on age, we replace the age for every individual with
a random age from the population.
A quantity of interest QA .X/ of some algorithm A is a statistic of the algorithm
over the population X is drawn from. As discussed above, examples of quantities of
interest are the average rate of positive classification for a group, or the classification
outcome of an individual.
The Quantitative Input Influence of an input on a quantity of interest is just the
difference in the quantity of interest with or without the intervention.
Definition 1 (QII) For a quantity of interest QA ./, and an input i, the Quantitative
Input Influence of i on QA ./ is defined to be

QA .i/ D QA .X/  QA .Xi Ui /:

We now instantiate this definition with different quantities of interest to illustrate

the above definition in three different scenarios.
QII for Individual Outcomes One intended use of QII is to provide personalized
transparency reports to users of data analytics systems. For example, if a person is
denied a job due to feedback from a machine learning algorithm, an explanation
of which factors were most influential for that person’s classification can provide
valuable insight into the classification outcome.
Algorithmic Transparency via Quantitative Input Influence 77

For QII to quantify the use of an input for individual outcomes, we define the
quantity of interest to be the classification outcome for a particular individual. Given
a particular individual x, we define Qxind ./ to be E.c./ D 1 j X D x/. The influence
measure is therefore:

xind .i/ D E.c.X/ D 1 j X D x/  E.c.Xi Ui / D 1 j X D x/ (1)

When the quantity of interest is not the probability of positive classification but
the classification that x actually received, a slight modification of the above QII
measure is more appropriate:

xind  act .i/ D E.c.X/ D c.x/ j X D x/  E.c.Xi Ui / D c.x/ j X D x/

D 1  E.c.Xi Ui / D c.x/ j X D x/ D E.c.Xi Ui / 6D c.x/ j X D x/
(2)
The above probability can be interpreted as the probability that feature i is pivotal
to the classification of c.x/. Computing the average of this quantity over X yields:
P
x2X Pr.X D x/E.i is pivotal for c.X/ j X D x/ D E.i is pivotal for c.X//: (3)

We denote this average QII for individual outcomes as defined above, by

ind  avg .i/, and use it as a measure for importance of an input towards classification
outcomes.
QII for Group Outcomes For more general findings, the quantity of interest may
be the classification outcome for a set of individuals. Given a group of individuals
Y  X , we define QY grp ./ to be E.c./ D 1 j X 2 Y /. The influence measure is
therefore:

Y
grp .i/ D E.c.X/ D 1 j X 2 Y /  E.c.Xi Ui / D 1 j X 2 Y / (4)

QII for Group Disparity Instead of simple classification outcomes, an analyst

may be interested in more nuanced properties of data analytics systems such as
the presence of disparate impact. Disparate impact compares the rates of positive
classification within protected groups defined by gender or race. In employment,
its absence is often codified as the ‘80% rule’ which states that the rate of selection
within a protected demographic should be at least 80% of the rate of selection within
the unprotected demographic. The quantity of interest in such a scenario is the ratio
in positive classification outcomes for a protected group Y from the rest of the
population X n Y .

E.c.X/ D 1 j X 2 Y /
E.c.X/ D 1 j X 62 Y /

However, the ratio of classification rates is unstable at low values of positive

classification. Therefore, for the computations in this paper we use the difference in
78 A. Datta et al.

classification rates as our measure of group disparity.

QY
disp ./ D jE.c./ D 1 j X 2 Y /  E.c./ D 1 j X 62 Y /j (5)

The QII measure of an input group disparity, as a result is:

Y Y Y
disp .i/ D Qdisp .X/  Qdisp .Xi Ui /: (6)

More generally, group disparity can be viewed as an association between

classification outcomes and membership in a group. QII on a measure of such
association (e.g., group disparity) identifies the variable that causes the association
in the classifier. Proxy variables are variables that are associated with protected
attributes. For addressing concerns of discrimination such as digital redlining, it is
important to identify which proxy variables actually introduce group disparity. It is
straightforward to observe that features with high QII for group disparity are proxy
variables that cause group disparity. Therefore, QII on group disparity is a useful
diagnostic tool for determining discrimination. The use of QII in identifying proxy
variables is explored experimentally in Sect. 5.1.

3 Transparency Schema

The forms of transparency reports discussed so far are specific instances of a general
schema for algorithmic transparency via input influence that we describe now.
Recall that the influence of an input is the difference in a quantity of interest before
and after an intervention. This definition features the following three elements of
our transparency schema.
• A quantity of interest, which captures the aspect of the system we wish to gain
transparency into.
• An intervention distribution, which defines how the alternate distribution is
constructed from the true distribution.
• A difference measure, which quantifies the difference between two quantities of
interest.
The particular instantiation of elements in this schema is determined by the
question the analyst wishes to answer about the system. Table 1 contains some
examples of these instantiations and we describe the elements in detail below.
The first element, the quantity of interest, is determined by the subject of the
question. In the previous section, the systems considered were deterministic, and
the quantities of interest considered were represented by a number such as an
expectation or a probability. However, for more complex systems, the analyst may
be interested in richer quantities about the system. For instance, if the system is
randomized, a particular quantity of interest is the distribution over outcomes for a
given individual.
Algorithmic Transparency via Quantitative Input Influence 79

Table 1 Examples of Quantity of interest Intervention Difference

transparency schema
elements Individual outcome Constant Subtraction
Average outcome Prior Absolute difference
Distribution Earthmover distance
Group disparity Mutual information

The second element is the causal intervention used to compute influence. The
particular randomized intervention considered in this paper, Xi Ui , where Ui is
drawn from the prior distribution of feature i, is suitable when no alternatives are
suggested in the analyst’s question. For example, the question “What is the influence
of age on Bob’s classification?” does not suggest any particular alternative age. On
the other hand, when a particular alternative is suggested, a constant intervention is
more suited. For example, for the question “Was age influential in Bob’s rejection
and Carl’s acceptance?” Carl’s age is the natural alternate to consider, and Ui is
drawn from the constant distribution corresponding to Carl’s age.
Finally, we use a difference measure to compare the quantity of interest before
and after the intervention. Apart from subtraction, considered above, examples of
other suitable difference measures are absolute difference, and distance measures
over distributions. The absolute difference measure can be employed when the
direction of change of outcomes before and after an intervention is unimportant.
When the quantity of interest is the distribution over outcomes, subtraction does
not make sense, and a suitable measure of difference should be employed, such
as the earthmover distance or the mutual information between the original and the
intervened distributions.

4 Set and Marginal QII

In many situations, intervention on a single input variable has no influence on the

outcome of a system. Consider, for example, a two-feature setting where features
are age (A) and income (I), and the classifier is c.A; I/ D .A D old/ ^ .I D high/. In
other words, the only datapoints that are labeled 1 are those of elderly persons with
high income. Now, given a datapoint where A D young; I D low, an intervention on
either age or income would result in the same classification. However, it would be
misleading to say that neither age nor income have an influence over the outcome;
changing both the states of income and age would result in a change in outcome.
Equating influence with the individual ability to affect the outcome is unin-
formative in real datasets as well: Fig. 1 is a histogram of feature influence
on individual outcomes for a classifier learned from the adult dataset [27].2

2
The adult dataset contains approximately 31k datapoints of users’ personal attributes, and
whether their income is more than $50k per annum; see Sect. 5 for more details.
80 A. Datta et al.

Fig. 1 A histogram of the

highest specific causal
influence for some feature
across individuals in the adult
dataset. Alone, most inputs
have very low influence

For most individuals, all features have zero influence: changing the state of one
feature alone is not likely to change the outcome of a classifier. Of the 19,537
datapoints we evaluate, more than half have x .i/ D 0 for all i 2 N; indeed, changes
to outcome are more likely to occur if we intervene on sets of features. In order
to better understand the influence of a feature i 2 N, we should measure its effect
when coupled with interventions on other features. We define the influence of a set
of inputs as a straightforward extension of individual input influence. Essentially,
we wish the influence of a set of inputs S  N to be the same as when the set of
inputs is considered to be a single input; when intervening on S, we draw the states
of i 2 S based on the joint distribution of the states of features in S.
We define the random variable XS US representing an intervention on a set of
features instead of a single feature. XS US has the states of features in N n S fixed
to their original values in x, but features in S take on new values according the
joint marginal distribution over the features in S. This is identical to the intervention
considered in unary QII, when the features in S are viewed as a single composite
feature. Using this generalization of an intervention to sets of features, we can define
set QII that measures the influence of a set of features.
Definition 2 (Set QII) For a quantity of interest Q, and an input i, the Quantitative
Input Influence of set S  N on Q is defined to be

Q .S/ D Q.X/  Q.XS US /:

Considering the influence of a set of inputs raises a number of interesting

questions due to the interaction between inputs. First among these is how does one
measure the individual effect of a feature, given the measured effects of interventions
on sets of features. One natural way of doing so is by measuring the marginal effect
of a feature on a set.
Algorithmic Transparency via Quantitative Input Influence 81

Definition 3 (Marginal QII) For a quantity of interest Q, and an input i, the

Quantitative Input Influence of input i over a set S  N on Q is defined to be

Q .i; S/ D Q.XS US /  Q.XS[fig US[fig /:

Notice that marginal QII can also be viewed as a difference in set QIIs: Q .S [
fig/  Q .S/. Informally, the difference between Q .S [ fig/ and Q .S/ measures the
“added value” obtained by intervening on S [ fig, versus intervening on S alone.
The marginal contribution of i may vary significantly based on S. Thus, we are
interested in the aggregate marginal contribution of i to S, where S is sampled from
some natural distribution over subsets of N n fig. In what follows, we describe
a few measures for aggregating the marginal contribution of a feature i to sets,
based on different methods for sampling sets. The primary method of aggregating
the marginal contribution is the Shapley value [39]. The less theoretically inclined
reader can choose to proceed to Sect. 5 without a loss in continuity.

4.1 Cooperative Games and Causality

In this section, we discuss how measures from the theory of cooperative games
define measures for aggregating marginal influence. In particular, we observe that
the Shapley value [39] is uniquely characterized by axioms that are natural in our
setting. Definition 2 measures the influence that an intervention on a set of features
S  N has on the outcome. One can naturally think of Set QII as a function v W
2N ! R, where v.S/ is the influence of S on the outcome. With this intuition in
mind, one can naturally study influence measures using cooperative game theory,
and in particular, using the Shapley value. The Shapley value can be thought of as
an influence aggregation method, which, given an influence measure v W 2N ! R,
outputs a vector  2 Rn , whose i-th coordinate corresponds in some natural way to
the aggregate influence, or aggregate causal effect, of feature i.
The original motivation for game-theoretic measures is revenue division [29,
Chapter 18]: the function v describes the amount of money that each subset of
players S  N can generate; assuming that the set N generates a total revenue
of v.N/, how should v.N/ be divided amongst the players? A special case of
revenue division that has received significant attention is the measurement of voting
power [40]. In voting systems with multiple agents (think of political parties in a
parliament), whose weights differ, voting power often does not directly correspond
to agent weights. For example, the US presidential election can roughly be modeled
as a cooperative game where each state is an agent. The weight of a state is the
number of electors in that state (i.e., the number of votes it brings to the presidential
candidate who wins that state). Although states like California and Texas have
higher weight, swing states like Pennsylvania and Ohio tend to have higher power
in determining the outcome of elections.
82 A. Datta et al.

A voting system can be modeled as a cooperative game: players are voters, and
the value of a coalition S  N is 1 if S can make a decision (e.g. pass a bill,
form a government, or perform a task), and is 0 otherwise. Note the similarity to
classification, with players being replaced by features (in classification, functions
of the form v W 2N ! f0; 1g are referred to as boolean functions [33]). The
game-theoretic measures of revenue division are a measure of voting power: how
much influence does player i have in the decision making process? Thus the notions
of voting power and revenue division fit naturally with our goals when defining
aggregate QII influence measures: in both settings, one is interested in measuring
the aggregate effect that a single element has, given the actions of subsets.
Several canonical influence measures rely on the fundamental notion of marginal
contribution. Given a player i and a set S  N n fig, the marginal contribution of
i to S is denoted mi .S; v/ D v.S [ fig/  v.S/ (we simply write mi .S/ when v
is clear from the context). Marginal QII, as defined above, can be viewed as an
instance of a measure of marginal contribution. Given a permutation  2 ˘.N/ of
the elements in N, we define Pi . / D fj 2 N j  .j/ <  .i/g; this is the set of i’s
predecessors in  . We can now similarly define the marginal contribution of i to
a permutation  2 ˘.N/ as mi . / D mi .Pi . //. Intuitively, one can think of the
players sequentially entering a room, according to some ordering  ; the value mi . /
is the marginal contribution that i has to whoever is in the room when she enters it.
Generally speaking, game theoretic influence measures specify some reasonable
way of aggregating the marginal contributions of i to sets S  N. In our setting,
we argue for the use of the Shapley value. Introduced by the late Lloyd Shapley,
the Shapley value is one of the most canonical methods of dividing revenue in
cooperative games. It is defined as follows:

1 X
'i .N; v/ D E Œmi . / D mi . /
nŠ
2˘.N/

Intuitively, the Shapley value describes the following process: players are sequen-
tially selected according to some randomly chosen order  ; each player receives a
payment of mi . /. The Shapley value is the expected payment to the players under
this regime.

4.2 Axiomatic Treatment of the Shapley Value

In this work, the Shapley value is our function of choice for aggregating marginal
feature influence. To justify our choice, we provide a brief exposition of axiomatic
game-theoretic value theory. We present a set of axioms that uniquely define the
Shapley value, and discuss why they are desirable in the QII setting.
Algorithmic Transparency via Quantitative Input Influence 83

The Shapley value satisfies the following properties:

Definition 4 (Symmetry) We say that i; j 2 N are symmetric if v.S [ fig/ D v.S [
fjg/ for all S  N n fi; jg. A value  satisfies symmetry if i D j whenever i and j
are symmetric.
Definition 5 (Dummy) We say that a player i 2 N is a dummy if v.S [ fig/ D v.S/
for all S  N. A value  satisfies the dummy property if i D 0 whenever i is a
dummy.
Definition 6 (Monotonicity) Given two games hN; v1 i; hN; v2 i, a value  satisfies
strong monotonicity if mi .S; v1 /  mi .S; v2 / for all S implies that i .N; v1 / 
i .N; v2 /, where a strict inequality for some set S  N implies a strict inequality for
the values as well.
All of these axioms take on a natural interpretation in the QII setting. Indeed, if
two features have the same probabilistic effect, no matter what other interventions
are already in place, they should have the same influence. In our context, the dummy
axiom says that a feature that never offers information with respect to an outcome
should have no influence. Monotonicity makes intuitive sense in the QII setting:
if a feature has consistently higher influence on the outcome in one setting than
another, its measure of influence should increase. For example, if a user receives
two transparency reports (say, for two separate loan applications), and in one report
gender had a consistently higher effect on the outcome than in the other, then the
transparency report should reflect this.
[47] offers an characterization of the Shapley value, based on these monotonicity
assumption.
Theorem 1 ([47]) The Shapley value is the only function that satisfies symmetry,
dummy and monotonicity.
To conclude, the Shapley value is a unique way of measuring aggregate influence in
the QII setting, while satisfying a set of very natural axioms.

5 Experimental Evaluation

We illustrate the utility of the QII framework by developing two simple machine
learning applications on real datasets. In Sect. 5.1, we illustrate the distinction
between different quantities of interest on which Unary QII can be computed. We
also illustrate the effect of discrimination on the QII measure. In Sect. 5.2, we
analyze transparency reports of three individuals to demonstrate how Marginal QII
can provide insights into individuals’ classification outcomes.
We use the following datasets in our experiments:
• adult[27]: This standard machine learning benchmark dataset is a subset of US
census data classifying individual income; it contains factors such as age, race,
gender, marital status and other socio-economic parameters. We use this dataset
to train a classifier that predicts the income of individuals from other parameters.
Such a classifier could potentially be used to assist in credit decisions.
84 A. Datta et al.

• arrests[32]: The National Longitudinal Surveys are a set of surveys con-

ducted by the Bureau of Labor Statistics of the United States. In particular,
we use the 1997 youth survey, covering young persons born in the years 1980–
1984. The survey covers various aspects of an individual’s life such as medical
history, criminal records and economic parameters. From this dataset, we extract
the following features: age, gender, race, region, history of drug use, history of
smoking, and history of arrests. We use this data to train a classifier that predicts
arrests history to simulate predictive policing, where socio-economic factors are
used to decide whether individuals should receive a visit from the police. A
similar application is described in [21].
The two applications described above are hypothetical examples of decision
making aided by algorithms that use potentially sensitive socio-economic data, and
not real systems that are currently in use. We use these classifiers to illustrate the
subtle causal questions that our QII measures can answer.
We use the following standard machine learning classifiers in our dataset:
Logistic Regression, SVM with a radial basis function kernel, Decision Tree,
and Gradient Boosted Decision Trees (see [7] for an excellent primer to these
classifiers); with the exception of Logistic Regression, a linear classifier, the other
three are nonlinear and can potentially learn very complex models.

5.1 Unary QII Measures

In Fig. 2, we illustrate the use of different Unary QII measures. Figure 2a, b show the
Average QII measure (Eq. (3)) computed for features of a decision forest classifier.
For the income classifier trained on the adult dataset, the feature with highest
influence is Marital Status, followed by Occupation, Relationship and Capital Gain.
Sensitive features such as Gender and Race have relatively lower influence. For the
predictive policing classifier trained on the arrests dataset, the most influential
input is Drug History, followed by Gender, and Smoking History. We observe that
influence on outcomes may be different from influence on group disparity.
QII on Group Disparity Figure 2c, d show influences of features on group disparity
(Eq. (6)) for two different settings. Figure 2c shows feature influence on group
disparity by Gender in the adult dataset; Fig. 2d shows the influence on group
disparity by Race in the arrests dataset. For the income classifier trained on
the adult dataset, we observe that most inputs have negative influence on group
disparity; randomly intervening on most inputs would lead to a reduction in group
disparity. In other words, a classifier that did not use these inputs would be fairer.
Interestingly, in this classifier, marital status—rather than gender—has the highest
influence on group disparity by gender.
For the arrests dataset, most inputs have the effect of increasing group
disparity if randomly intervened on. In particular, Drug history has the highest
positive influence on disparity in arrests. Although Drug history is correlated
with race, using it reduces disparate impact by race, i.e. makes fairer decisions.
Algorithmic Transparency via Quantitative Input Influence 85

(a) QII of inputs on Outcomes for the adult (b) QII of inputs on Outcomes for the arrests
dataset dataset

(c) QII of Inputs on Group Disparity by Gender (d) Influence on Group Disparity by Race in the
in the adult dataset arrests dataset

Fig. 2 QII measures for the adult and arrests datasets. (a) QII of inputs on outcomes for
the adult dataset, (b) QII of inputs on Outcomes for the arrests dataset, (c) QII of Inputs on
group disparity by gender in the adult dataset, (d) Influence on group disparity by race in the
arrests dataset

In both examples, features correlated with the sensitive attribute are the most
influential for group disparity according to the sensitive attribute rather than the
sensitive attribute itself. It is in this sense that QII measures can identify proxy
variables that cause associations between outcomes and sensitive attributes.
QII with Artificial Discrimination We simulate discrimination using an artificial
experiment. We first randomly assign “ZIP codes” to individuals in our dataset
(these are simply arbitrary numerical identifiers). Then, to simulate systematic bias,
we make an f fraction of the ZIP codes discriminatory in the following sense: all
individuals in the protected set are automatically assigned a negative classification
outcome. We then study the change in the influence of features as we increase f .
Figure 3a, shows that the influence of Gender increases almost linearly with f .
Recall that Marital Status was the most influential feature for this classifier without
any added discrimination. As f increases, the importance of Marital Status decreases
as expected, since the number of individuals for whom Marital Status is pivotal
decreases.
86 A. Datta et al.

(a) Change in QII of inputs as discrimination by (b) Change in QII of inputs as discrimination by
Zip Code increases in the adult dataset Zip Code increases in the arrests dataset

Fig. 3 The effect of discrimination on QII. (a) Change in QII of inputs as discrimination by Zip
code increases in the adult dataset, (b) Change in QII of inputs as discrimination by Zip code
increases in the arrests dataset

5.2 Personalized Transparency Reports

To illustrate the utility of personalized transparency reports, we study the clas-

sification of individuals who received potentially unexpected outcomes. For the
personalized transparency reports, we use classification outcomes obtained from
decision forests, though one obtains a similar outcome using other classifiers.
The influence measure that we employ is the Shapley value. In more detail, the
coalitional function we use is v.S/ D QA .S/, with QA being EŒc./ D 1 j X D x;
that is, the marginal contribution of i 2 N to S is given by mi .S/ D EŒc.XS / D 1 j
X D x  EŒc.XS[fig / D 1 j X D x.
We emphasize that some features may have a negative Shapley value; this should
be interpreted as follows: a feature with a high positive Shapley value often increases
the certainty that the classification outcome is 1, whereas a feature whose Shapley
value is negative is one that increases the certainty that the classification outcome
would be zero.
Mr. X: The first example is of an individual from the adult dataset, to whom
we refer as Mr. X (Fig. 4). The learnt classifier classifies his income as low.
This result may be surprising to him: he reports high capital gains ($14k), and
only 2.1% of people with capital gains higher than $10k are reported as low
income. In fact, he might be led to believe that his classification may be a
result of his ethnicity or country of origin. Examining his transparency report in
Fig. 4, however, we find that the most influential features that led to his negative
classification were Marital Status, Relationship and Education.
Mr. Y: The second example, to whom we refer as Mr. Y (Fig. 5), has even higher
capital gains than Mr. X. Mr. Y is a 27 year old, with only Preschool education,
and is engaged in fishing. Examination of the transparency report reveals that
the most influential factor for negative classification for Mr. Y is his Occupation.
Interestingly, his low level of education is not considered very important by this
classifier.
Algorithmic Transparency via Quantitative Input Influence 87

Age 23
Workclass Private
Education 11th
Education-Num 7
Marital Status Never-married
Occupation Craft-repair
Relationship Own-child
Race Asian-Pac-Islander
Gender Male
Capital Gain 14344
Capital Loss 0
Hours per week 40
Country Vietnam

Fig. 4 Mr. X’s profile and transparency report for negative classification

Age 27
Workclass Private
Education Preschool
Education-Num 1
Marital Status Married-civ-spouse
Occupation Farming-fishing
Relationship Other-relative
Race White
Gender Male
Capital Gain 41310
Capital Loss 0
Hours per week 24
Country Mexico

Fig. 5 Mr. Y’s profile and transparency report for negative classification

Mr. Z: The third example, who we refer to as Mr. Z (Fig. 6) is from the arrests
dataset. History of drug use and smoking are both strong indicators of arrests.
However, Mr. X received positive classification by this classifier even without
any history of drug use or smoking. Upon examining his classifier, it appears that
race, age and gender were most influential in determining his outcome. In other
words, the classifier that we train for this dataset (a decision forest) has picked
up on the correlations between race (Black), and age (born in 1984) to infer that
this individual is likely to engage in criminal activity. Indeed, our interventional
approach indicates that this is not a mere correlation effect: race is actively being
used by this classifier to determine outcomes. Of course, in this instance, we
have explicitly offered the race parameter to our classifier as a viable feature.
However, our influence measure is able to pick up on this fact, and alert us of the
problematic behavior of the underlying classifier. More generally, this example
illustrates a concern with the black box use of machine learning which can lead
to unfavorable outcomes for individuals.
88 A. Datta et al.

Birth Year 1984

Drug History None
Smoking History None
Census Region West
Race Black
Gender Male

Fig. 6 Mr. Z’s profile and transparency report for positive classification

6 QII for Fairness

Due to the widespread and black box use of machine learning in aiding decision
making, there is a legitimate concern of algorithms introducing and perpetuating
social harms such as racial discrimination [4, 35]. As a result, the algorithmic
foundations of fairness in personal information processing systems have received
significant attention recently [10, 12, 15, 22, 48]. While many of the algorithmic
approaches [10, 17, 22, 48] have focused on group parity as a metric for achieving
fairness in classification, Dwork et al. [15] argue that group parity is insufficient as a
basis for fairness, and propose an approach which prescribes that similar individuals
should receive similar classification outcomes. However, this approach requires a
similarity metric for individuals which is often subjective and difficult to construct.
QII does not suggest any normative definition of fairness. Instead, we view QII as
a diagnostic tool to aid fine-grained fairness determinations. In fact, QII can be used
in the spirit of the similarity based definition of [15] by comparing the personalized
privacy reports of individuals who are perceived to be similar but received different
classification outcomes, and identifying the inputs which were used by the classifier
to provide different outcomes.
When group parity is used as a criteria for fairness, QII can identify the features
that lead to group disparity, thereby identifying features being used by a classifier
as a proxy for sensitive attributes. The determination of whether using certain
proxies for sensitive attributes is discriminatory is often a task-specific normative
judgment. For example, using standardized test scores (e.g., SAT scores) for college
admissions decisions is by and large accepted. However, SAT scores may act as
a proxy for several protected attributes, leading to concerns of unfairness [43, 45].
Our goal is not to provide such normative judgments. Rather we seek to provide fine-
grained transparency into input usage (e.g., what’s the extent to which SAT scores
influence decisions), which is useful to make determinations of discrimination from
a specific normative position.
Algorithmic Transparency via Quantitative Input Influence 89

7 Related Work

QII enables algorithmic transparency via a family of game-theoretic causal influence

measures. In this section we compare with related work along a number of
different dimensions. First, we compare with related approaches to measuring causal
influence, associations, and briefly discuss an emerging direction that measures
proxy influence via a combination of associative and causal techniques. Next, we
discuss orthogonal approaches to algorithmic transparency: interpretable machine
learning and experimentation on web services. Finally, we discuss other fields in
which similar game-theoretic measures have been applied.
Quantitative Causal Measures Causal models and probabilistic interventions have
been used in a few other settings. While the form of the interventions in some
of these settings may be very similar, our generalization to account for different
quantities of interests enables us to reason about a large class of transparency queries
for data analytics systems ranging from classification outcomes of individuals to
disparity among groups. Further, the notion of marginal contribution which we use
to compute responsibility does not appear in this line of prior work.
Janzing et al. [20] use interventions to assess the causal importance of relations
between variables in causal graphs; in order to assess the causal effect of a relation
between two variables, X ! Y (assuming that both take on specific values X D x
and Y D y), a new causal model is constructed, where the value of X is replaced with
a prior over the possible values of X. The influence of the causal relation is defined
as the distance between the joint distributions of all the variables in the two causal
models with and without the value of X replaced. The approach of intervening with
a random value from the prior is similar to our approach of constructing XS US .
Independently, there has been considerable work in the machine learning com-
munity to define importance metrics for variables, mainly for the purpose of feature
selection (see [19] for a comprehensive overview). One important metric is called
Permutation Importance [9], which measures the importance of a feature towards
classification by randomly permuting the values of the feature and then computing
the difference of classification accuracies before and after the permutation. Replac-
ing a feature with a random permutation can be viewed as sampling the feature
independently from the prior.
Measures of Association One can think of our results as a causal alternative
to quantitative information flow. Quantitative information flow is a broad class
of metrics that quantify the information leaked by a process by comparing the
information contained before and after observing the outcome of the process.
Quantitative Information Flow traces its information-theoretic roots to the work of
Shannon [38] and Rényi [36]. Recent works have proposed measures for quantifying
the security of information by measuring the amount of information leaked from
inputs to outputs by certain variables; we point the reader to [41] for an overview,
and to [11] for an exposition on information theory. Quantitative Information Flow
is concerned with information leaks and therefore needs to account for correlations
90 A. Datta et al.

between inputs that may lead to leakage. The dual problem of transparency, on the
other hand, requires us to destroy correlations while analyzing the outcomes of a
system to identify the causal paths for information leakage. Measures of association
have also been widely used for detecting discrimination (see Sect. 6).
Proxy Influence An emerging direction in this space is the identification of proxy
or indirect use of sensitive features as opposed to direct causal use, as captured
by QII. Adler et al. [1] quantify the indirect influence of an attribute by obscuring
the attribute (along with associations) from a dataset and comparing the prediction
accuracy of a model before and after obscuring. In a different approach, Datta
et al. (Use privacy in data-driven systems: theory and experiments with machine
learnt programs, Unpublished Manuscript) identify intermediate computations in a
program that are associated with an attribute and use QII to measure the causal
influence of the intermediate computations on the outcome.
Interpretable Machine Learning An orthogonal approach to adding interpretability
to machine learning is to constrain the choice of models to those that are inter-
pretable by design. This can either proceed through regularization techniques such
as Lasso [44] that attempt to pick a small subset of the most important features, or
by using models that structurally match human reasoning such as Bayesian Rule
Lists [26], Supersparse Linear Integer Models [46], or Probabilistic Scaling [37].
Since the choice of models in this approach is restricted, a loss in predictive accuracy
is a concern, and therefore, the central focus in this line of work is the minimization
of the loss in accuracy while maintaining interpretability. On the other hand, our
approach to interpretability is forensic. We add interpretability to machine learning
models after they have been learnt. As a result, our approach does not constrain the
choice of models that can be used.
Experimentation on Web Services Another emerging body of work focuses on
systematic experimentation to enhance transparency into Web services such as
targeted advertising [3, 12, 18, 24, 25]. The setting in this line of work is different
since they have restricted access to the analytics systems through publicly available
interfaces. As a result they only have partial control of inputs, partial observability
of outputs, and little or no knowledge of input distributions. The intended use of
these experiments is to enable external oversight into Web services without any
cooperation. Our framework is more appropriate for a transparency mechanism
where an entity proactively publishes transparency reports for individuals and
groups. Our framework is also appropriate for use as an internal or external oversight
tool with access to mechanisms with control and knowledge of input distributions,
thereby forming a basis for testing.
Game-Theoretic Influence Measures Recent years have seen game-theoretic influ-
ence measures used in various settings. Datta et al. [13] also define a measure for
quantifying feature influence in classification tasks. Their measure does not account
for the prior on the data, nor does it use interventions that break correlations between
sets of features. In the terminology of this paper, the quantity of interest used by
Datta et al. [13] is the ability of changing the outcome by changing the state of
Algorithmic Transparency via Quantitative Input Influence 91

a feature. Strumbelj and Kononenko [42] also use the Shapley value to compute
influences for individual classification. This work greatly extends and generalizes
the concepts presented in [13] and [42], by both accounting for interventions
on sets, and by generalizing the notion of influence to include a wide range of
system behaviors, such as group disparity, group outcomes and individual outcomes.
Essentially, the instantiations of our transparency schema define a wide range of
transparency reports.
Game theoretic measures have been used by various research disciplines to
measure influence. Indeed, such measures are relevant whenever one is interested
in measuring the marginal contribution of variables, and when sets of variables are
able to cause some measurable effect. Prominent domains where game theoretic
measures have recently been used are terrorist networks [28, 30], protein inter-
actions [8], and neurophysical models [23]. The novelty in our use of the game
theoretic power indices lies in the conception of a cooperative game via a valuation
function .S/, defined by a randomized intervention on input S. Such an intervention
breaks correlations and allows us to compute marginal causal influences on a wide
range of system behaviors.

8 Conclusion & Future Work

In this chapter, we present QII, a general family of metrics for quantifying the
influence of inputs in systems that process personal information. In particular,
QII lends insights into the behavior of opaque machine learning algorithms by
allowing us to answer a wide class of transparency queries ranging from influence
on individual causal outcomes to influence on disparate impact. To achieve this,
QII breaks correlations between inputs to allow causal reasoning, and computes
the marginal influence of inputs in situations where inputs cannot affect outcomes
alone. Also, we demonstrate that QII can be efficiently approximated, and can be
made differentially private with negligible noise addition in many cases.
We do not consider situations where inputs do not have well understood seman-
tics. Such situations arise often in settings such as image or speech recognition,
and automated video surveillance. With the proliferation of immense processing
power, complex machine learning models such as deep neural networks have
become ubiquitous in these domains. Similarly, we do not consider the problem of
measuring the influence of inputs that are not explicitly provided but inferred by the
model. Defining transparency and developing analysis techniques in such settings is
important future work.
92 A. Datta et al.

References

1. Adler, P., Falk, C., Friedler, S., Rybeck, G., Schedegger, C., Smith, B., Venkatasubramanian,
S.: Auditing black-box models for indirect influence. In: Proceedings of the 2016 IEEE
International Conference on Data Mining (ICDM), ICDM’16, pp. 339–348. IEEE Computer
Society, Washington (2016)
2. Alloway, T.: Big data: Credit where credit’s due (2015). http://www.ft.com/cms/s/0/7933792e-
a2e6-11e4-9c06-00144feab7de.html
3. Barford, P., Canadi, I., Krushevskaja, D., Ma, Q., Muthukrishnan, S.: Adscape: harvesting and
analyzing online display ads. In: Proceedings of the 23rd International Conference on World
Wide Web, WWW’14, pp. 597–608. ACM, New York (2014)
4. Barocas, S., Nissenbaum, H.: Big data’s end run around procedural privacy protections.
Commun. ACM 57(11), 31–33 (2014)
5. Big data in education (2015). https://www.edx.org/course/big-data-education-
teacherscollegex-bde1x
6. Big data in government, defense and homeland security 2015–2020 (2015). http://www.
prnewswire.com/news-releases/big-data-in-government-defense-and-homeland-security-
2015---2020.html
7. Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics).
Springer, New York (2006)
8. Bork, P., Jensen, L., von Mering, C., Ramani, A., Lee, I., Marcott, E.: Protein interaction
networks from yeast to human. Curr. Opin. Struct. Biol. 14(3), 292–299 (2004)
9. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
10. Calders, T., Verwer, S.: Three naive Bayes approaches for discrimination-free classification.
Data Min. Knowl. Disc. 21(2), 277–292 (2010)
11. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (2012)
12. Datta, A., Tschantz, M., Datta, A.: Automated experiments on ad privacy settings: a tale
of opacity, choice, and discrimination. In: Proceedings on Privacy Enhancing Technologies
(PoPETs 2015), pp. 92–112 (2015)
13. Datta, A., Datta, A., Procaccia, A., Zick, Y.: Influence in classification via cooperative game
theory. In: Proceedings of the 24th International Joint Conference on Artificial Intelligence
(IJCAI 2015), pp. 511–517 (2015)
14. Datta, A., Sen, S., Zick, Y.: Algorithmic transparency via quantitative input influence: theory
and experiments with learning systems. In: Proceedings of 37th Symposium on Security and
Privacy (Oakland 2016), pp. 598–617 (2016)
15. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In:
Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS 2012),
pp. 214–226 (2012)
16. E.G. Griggs v. Duke Power Co., 401 U.S. 424, 91 S. Ct. 849, 28 L. Ed. 2d 158 (1977)
17. Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S.: Certifying
and removing disparate impact. In: Proceedings of the 21th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, KDD’15, pp. 259–268. ACM, New
York (2015)
18. Guha, S., Cheng, B., Francis, P.: Challenges in measuring online advertising systems. In:
Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, IMC’10,
pp. 81–87. ACM, New York (2010)
19. Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res.
3, 1157–1182 (2003)
20. Janzing, D., Balduzzi, D., Grosse-Wentrup, M., Schölkopf, B.: Quantifying causal influences.
Ann. Statist. 41(5), 2324–2358 (2013)
21. Jelveh, Z., Luca, M.: Towards diagnosing accuracy loss in discrimination-aware classification:
an application to predictive policing. In: Fairness, Accountability and Transparency in Machine
Learning, pp. 137–141 (2014)
Algorithmic Transparency via Quantitative Input Influence 93

22. Kamishima, T., Akaho, S., Sakuma, J.: Fairness-aware learning through regularization
approach. In: Proceedings of the 2011 IEEE 11th International Conference on Data Mining
Workshops (ICDMW 2011), pp. 643–650 (2011)
23. Keinan, A., Sandbank, B., Hilgetag, C., Meilijson, I., Ruppin, E.: Fair attribution of functional
contribution in artificial and biological networks. Neural Comput. 16(9), 1887–1915 (2004)
24. Lécuyer, M., Ducoffe, G., Lan, F., Papancea, A., Petsios, T., Spahn, R., Chaintreau, A.,
Geambasu, R.: Xray: enhancing the web’s transparency with differential correlation. In:
Proceedings of the 23rd USENIX Conference on Security Symposium, SEC’14, pp. 49–64.
USENIX Association, Berkeley (2014)
25. Lecuyer, M., Spahn, R., Spiliopolous, Y., Chaintreau, A., Geambasu, R., Hsu, D.: Sunlight:
Fine-grained targeting detection at scale with statistical confidence. In: Proceedings of the 22nd
ACM SIGSAC Conference on Computer and Communications Security, CCS’15, pp. 554–566.
ACM, New York (2015)
26. Letham, B., Rudin, C., McCormick, T.H., Madigan, D.: Interpretable classifiers using rules and
Bayesian analysis: building a better stroke prediction model. Ann. Appl. Stat. 9(3), 1350–1371
(2015)
27. Lichman, M.: UCI machine learning repository (2013). http://archive.ics.uci.edu/ml
28. Lindelauf, R., Hamers, H., Husslage, B.: Cooperative game theoretic centrality analysis of
terrorist networks: the cases of Jemaah Islamiyah and Al Qaeda. Eur. J. Oper. Res. 229(1),
230–238 (2013)
29. Maschler, M., Solan, E., Zamir, S.: Game Theory. Cambridge University Press, Cambridge
(2013)
30. Michalak, T., Rahwan, T., Szczepanski, P., Skibski, O., Narayanam, R., Wooldridge, M., Jen-
nings, N.: Computational analysis of connectivity games with applications to the investigation
of terrorist networks. In: Proceedings of the 23rd International Joint Conference on Artificial
Intelligence (IJCAI 2013), pp. 293–301 (2013)
31. Murdoch, T.B., Detsky, A.S.: The inevitable application of big data to health care. http://jama.
jamanetwork.com/article.aspx?articleid=1674245
32. National longitudinal surveys (2017). http://www.bls.gov/nls/
33. O’Donnell, R.: Analysis of Boolean Functions. Cambridge University Press, New York (2014)
34. Perry, W.L., McInnis, B., Price, C.C., Smith, S.C., Hollywood, J.S.: Predictive policing: the
role of crime forecasting in law enforcement operations. RAND Corporation, Santa Monica
(2013)
35. Podesta, J., Pritzker, P., Moniz, E., Holdern, J., Zients, J.: Big data: seizing opportunities,
preserving values. Technical Report, Executive Office of the President - the White House
(2014)
36. Rényi, A.: On measures of entropy and information. In: Proceedings of the Fourth Berkeley
Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory
of Statistics, pp. 547–561. University of California Press, Berkeley (1961)
37. Rüping, S.: Learning interpretable models. Ph.D. Thesis, Dortmund University of Technology
(2006). Http://d-nb.info/997491736
38. Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423
(1948)
39. Shapley, L.: A value for n-person games. In: Contributions to the Theory of Games, vol. 2,
Annals of Mathematics Studies, No. 28, pp. 307–317. Princeton University Press, Princeton
(1953)
40. Shapley, L.S., Shubik, M.: A method for evaluating the distribution of power in a committee
system. Am. Polit. Sci. Rev. 48(3), 787–792 (1954)
41. Smith, G.: Quantifying information flow using min-entropy. In: Proceedings of the 8th
International Conference on Quantitative Evaluation of Systems (QEST 2011), pp. 159–167
(2011)
42. Strumbelj, E., Kononenko, I.: An efficient explanation of individual classifications using game
theory. J. Mach. Learn. Res. 11, 1–18 (2010)
94 A. Datta et al.

43. The National Center for Fair and Open Testing: 850+ colleges and universities that do not
use SAT/ACT scores to admit substantial numbers of students into bachelor degree programs
(2015). http://www.fairtest.org/university/optional
44. Tibshirani, R.: Regression shrinkage and selection via the lasso: a retrospective. J. R. Stat. Soc.
Ser. B 73(3), 273–282 (2011)
45. University, G.W.: Standardized test scores will be optional for GW applicants (2015). https://
gwtoday.gwu.edu/standardized-test-scores-will-be-optional-gw-applicants
46. Ustun, B., Tracà, S., Rudin, C.: Supersparse linear integer models for interpretable classifica-
tion. ArXiv e-prints (2013). http://arxiv.org/pdf/1306.5860v1
47. Young, H.: Monotonic solutions of cooperative games. Int. J. Game Theory 14(2), 65–72
(1985)
48. Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning fair representations.
In: Proceedings of the 30th International Conference on Machine Learning (ICML 2013),
pp. 325–333 (2013)