Applied Econometrics:

Causal Inference and Research Design,

Professor Scott Cunningham, Spring 2017

Figure: xkcd
 Basic information

Professor Scott Cunningham’s contact information:

scunning@gmail.com or scott cunningham@baylor.edu
phone: (254) 537-2239
Office hours: MW: 11:00am to 12:15am in my office
 Overview
Part 1: The Core
Part 2: Selection on observables
Administrative
Part 3: Selection on unobservables
Empiricism in Microeconomics
Part 4: Advanced material
Conclusion
References

Structure and Assessment

The fundamental theme linking all lectures will be the estimation

of causal effects
Part 1 (8 lectures) covers “the core” of applied econometrics
Part 2 (10 lectures) covers “selection on observables”
methodologies
Part 3 (10 lectures) covers “selection on unobservables”
methodologies

Cunningham Causal Inference

 Assessment

Your grade will be based on three things:

4 problem sets (includes replications) weight: 10 percent
each.
You will need to purchase a copy of STATA (grad plan) or use
the school’s licensed version on the network as the problem
sets require programming.
1 presentation weight: 20 percent
2 exams weight: 20 percent each. Final is cumulative. No
make up exams.
 Required readings

Textbooks
1 Angrist and Pischke (2009) Mostly Harmless Econometrics
(MHE)
2 Morgan and Winship (2014) Counterfactuals and Causal
Inference (MW)
Readings:
We will also discuss a number of papers in each lecture, each
of which you will need to learn inside and out.
Lecture slides and reading lists are available
The exam will cover material from all required readings.
 Content of Part 1: The Core

Part 1 includes 2 lectures on each of the following topics:

1 Statistics and Econometrics Review
Expectation operator, mean, covariance, variance
Ordinary Least Squares (derivation, properties, intuition, etc.)
Conditional expectation function and linear regression
2 Modern theories of causality
Rubin causal model
Pearl’s directed acyclical graphical (DAG) model
3 Randomized experiments
 Content of Part 2: Selection on observables

Part 2 includes lectures on each of the following topics

1 Regression
2 Stratification
3 Matching
4 Regression discontinuity design
 Content of Part 3: Selection on unobservables

Part 3 includes lectures on the following topics

1 Panel methods, differences-in-differences and synthetic control
2 Instrumental variables
 Overview
Part 1: The Core
Part 2: Selection on observables
Administrative
Part 3: Selection on unobservables
Empiricism in Microeconomics
Part 4: Advanced material
Conclusion
References

Introduction: OLS Review

Derivation of the OLS estimator

Algebraic properties of OLS
Statistical Properties of OLS
Variance of OLS and standard errors

Cunningham Causal Inference

 Terminology
y x
Dependent Variable Independent Variable
Explained Variable Explanatory Variable
Response Variable Control Variable
Predicted Variable Predictor Variable
Regressand Regressor
LHS RHS

The terms “explained” and “explanatory” are probably best, as

they are the most descriptive and widely applicable. But
“dependent” and “independent” are used often. (The
“independence” here is not really statistical independence.)
We said we must confront three issues:
1 How do we allow factors other than x to affect y ?
2 What is the functional relationship between y and x?
3 How can we be sure we are capturing a ceteris paribus
relationship between y and x?
We will argue that the simple regression model

y = β0 + β1 x + u (1)
addresses each of them.
The SLR model is a population model. When it comes to
estimating β1 (and β0 ) using a random sample of data, we
must restrict how u and x are related to each other.
What we must do is restrict the way u and x relate to each
other in the population.
First, we make a simplifying assumption (without loss of
generality): the average, or expected, value of u is zero in the
population:

E (u) = 0 (2)
where E (·) is the expected value operator.
The presence of β0 in

y = β0 + β1 x + u (3)

allows us to assume E (u) = 0. If the average of u is different from

zero, say α0 , we just adjust the intercept, leaving the slope the
same:
y = (β0 + α0 ) + β1 x + (u − α0 ) (4)
where α0 = E (u). The new error is u − α0 and the new intercept
is β0 + α0 . The important point is that the slope, β1 , has not
changed.
An assumption that meshes well with our introductory treatment
involves the mean of the error term for each “slice” of the
population determined by values of x:

E (u|x) = E (u), all values x (5)

where E (u|x) means “the expected value of u given x”.
Then, we say u is mean independent of x.
Suppose u is “ability” and x is years of education. We need,
for example,

E (ability |x = 8) = E (ability |x = 12) = E (ability |x = 16)

so that the average ability is the same in the different portions

of the population with an 8th grade education, a 12th grade
education, and a four-year college education.
Because people choose education levels partly based on
ability, this assumption is almost certainly false.
Suppose u is “land quality” and x is fertilizer amount. Then
E (u|x) = E (u) if fertilizer amounts are chosen independently
of quality.
Combining E (u|x) = E (u) (the substantive assumption) with
E (u) = 0 (a normalization) gives

E (u|x) = 0, all values x (6)

Called the zero conditional mean assumption. Because the
conditional expected value is a linear operator, E (u|x) = 0 implies

E (y |x) = β0 + β1 x (7)
which shows the population regression function is a linear
function of x.
The straight line in the graph on the next page is what
Wooldridge calls the population regression function, and
what Angrist and Pischke call the conditional expectation
function
E (y |x) = β0 + β1 x
The conditional distribution of y at three different values of x
are superimposed. for a given value of x, we see a range of y
values: remember, y = β0 + β1 x + u, and u has a distribution
in the population.
 Deriving the Ordinary Least Squares Estimates

Given data on x and y , how can we estimate the population

parameters, β0 and β1 ?
Let {(xi , yi ) : i = 1, 2, ..., n} be a random sample of size n
(the number of observations) from the population.
Plug any observation into the population equation:

yi = β0 + β1 xi + ui (8)

where the i subscript indicates a particular observation.

We observe yi and xi , but not ui (but we know it is there).
We use the two population restrictions:

E (u) = 0 (9)
Cov (x, u) = 0 (10)

to obtain estimating equations for β0 and β1 . We talked about the

first condition. The second condition means that x and u are
uncorrelated. Both conditions are implied by

E (u|x) = 0 (11)
With E (u) = 0, Cov (x, u) = 0 is the same as E (xu) = 0. Next we
plug in for u:

E (y − β0 − β1 x) = 0 (12)
E [x(y − β0 − β1 x)] = 0 (13)

These are the two conditions in the population that effectively

determine β0 and β1 .
So we use their sample counterparts (which is a method of
moments approach to estimation):

n
X
−1
n (yi − β̂0 − β̂1 xi ) = 0 (14)
i=1
n
X
−1
n xi (yi − β̂0 − β̂1 xi ) = 0 (15)
i=1

where β̂0 and β̂1 are the estimates from the data.
These are two linear equations in the two unknowns β̂0 and β̂1 .
Pass the summation operator through the first equation:
n
X
−1
n (yi − β̂0 − β̂1 xi ) (16)
i=1
n
X n
X n
X
−1 −1 −1
=n yi − n β̂0 − n β̂1 xi (17)
i=1 i=1 i=1
n n
!
X X
= n−1 yi − β̂0 − β̂1 n−1 xi (18)
i=1 i=1

= y − β̂0 − β̂1 x (19)

We use the standard notation y = n−1 ni=1 yi for the average of
P
the n numbers {yi : i = 1, 2, ..., n}. For emphasis, we call y a
sample average.
We have shown that the first equation,
n
X
n−1 (yi − β̂0 − β̂1 xi ) = 0 (20)
i=1

implies

y = β̂0 + β̂1 x (21)

Now, use this equation to write the intercept in terms of the slope:

β̂0 = y − β̂1 x (22)

Plug this into the second equation (but where we take away the
division by n):
n
X
xi (yi − β̂0 − β̂1 xi ) = 0 (23)
i=1
so
n
X
xi [yi − (y − β̂1 x) − β̂1 xi ] = 0 (24)
i=1

Simple algebra gives

n
" n #
X X
xi (yi − y ) = β̂1 xi (xi − x) (25)
i=1 i=1
So, the equation to solve is
n
" n #
X X
(xi − x)(yi − y ) = β̂1 (xi − x)2 (26)
i=1 i=1
Pn
If i=1 (xi − x)2 > 0, we can write

Pn
(x − x)(yi − y ) Sample Covariance(xi , yi )
Pn i
β̂1 = i=1 2
= (27)
i=1 (xi − x) Sample Variance(xi )
The previous formula for β̂1 is important. It shows us how to
take the data we have and compute the slope estimate.
β̂1 is called the ordinary least squares (OLS) slope estimate.
It can be computed whenever the sample variance of the xi is
not zero, which only rules out the case where each xi has the
same value.
The intuition is that the variation in x is what permits us to
identify its impact on y .
Once we have β̂1 , we compute β̂0 = y − β̂1 x. This is the OLS
intercept estimate.
These days, we let the computer do the calculations, which
are tedious even if n is small.
For any candidates β̂0 and β̂1 , define a fitted value for each i
as

ŷi = β̂0 + β̂1 xi (28)

We have n of these. This is the value we predict for yi given
that x = xi .
The “mistake” we make is the residual:

ûi = yi − ŷi = yi − β̂0 − β̂1 xi (29)

Suppose we measure the size of the mistake, for each i, by

squaring it. Then we add them all up:
n
X n
X
ûi2 = (yi − β̂0 − β̂1 xi )2 (30)
i=1 i=1

This is called the sum of squared residuals. Finally, choose

β̂0 and β̂1 to minimize the sum of squared residuals. Using
calculus (or other arguments), it can be shown that the
solutions are the same we obtained before.
Algebraic Properties of OLS Statistics
Remeber how we obtained β̂0 and β̂1 . When an intercept is
included, we will have:
n
X
(yi − β̂0 − β̂1 xi ) = 0 (31)
i=1

The OLS residuals always add up to zero, by construction,

n
X
ûi = 0 (32)
i=1

Because yi = ŷi + ûi by definition,

n
X n
X n
X
−1 −1 −1
n yi = n ŷi + n ûi (33)
i=1 i=1 i=1

and so y = ŷ .
Similarly the way we obtained our estimates,
n
X
n−1 xi (yi − β̂0 − β̂1 xi ) = 0 (34)
i=1

The sample covariance (and therefore the sample correlation)

between the explanatory variables and the residuals is always zero:
n
X
n−1 xi ûi = 0 (35)
i=1

Because the ŷi are linear functions of the xi , the fitted values and
residuals are uncorrelated, too:
n
X
n−1 ŷi ûi = 0 (36)
i=1
Both properties hold by construction. β̂0 and β̂1 were chosen to
make them true.
A third property is that the point (x, y ) is always on the OLS
regression line. That is, if we plug in the average for x, we predict
the sample average for y :

y = β̂0 + β̂1 x (37)

Again, we chose the estimates to make this true.
 Expected Value of OLS

Mathematical statistics: How do our estimators behave across

different samples of data? On average, would we get the right
answer if we could repeatedly sample?
We need to find the expected value of the OLS estimators –
in effect, the average outcome across all possible random
samples – and determine if we are right on average.
Leads to the notion of unbiasedness, which is a “desirable”
characteristic for estimators.

E (β̂) = β (38)
Remember, our objective is to estimate β1 , the slope
population parameter that describes the relationship between
y and x.
β̂1 is an estimator of that parameter obtained for a specific
sample.
Different samples will generate different estimates (β̂1 ) for the
“true” β1 , i.e. (β̂1 ) is a random variable.
Unbiasedness is the idea that if we could take as many
random samples on Y as we want from the population, and
compute an estimate each time, the average of these
estimates would be equal to β1 .
Assumption SLR.1 (Linear in Parameters)
The population model can be written as

y = β0 + β1 x + u (39)
where β0 and β1 are the (unknown) population parameters.
We view x and u as outcomes of random variables; thus, y is
random.
Stating this assumption formally shows that our goal is to
estimate β0 and β1 .
Assumption SLR.2 (Random Sampling)
We have a random sample of size n, {(xi , yi ) : i = 1, ..., n},
following the population model.
We know how to use this data to estimate β0 and β1 by OLS.
Because each i is a draw from the population, we can write,
for each i,

yi = β0 + β1 xi + ui (40)
Notice that ui here is the unobserved error for observation i.
It is not the residual that we compute from the data!
Assumption SLR.3 (Sample Variation in the Explanatory
Variable)
The sample outcomes on xi are not all the same value.
This is the same as saying the sample variance of
{xi : i = 1, ..., n} is not zero.
In practice, this is no assumption at all. If the xi are all the
same value, we cannot learn how x affects y in the population.
Assumption SLR.4 (Zero Conditional Mean)
In the population, the error term has zero mean given any
value of the explanatory variable:

E (u|x) = E (u) = 0. (41)

This is the key assumption for showing that OLS is unbiased,
with the zero value not being important once we assume
E (u|x) does not change with x.
Note that we can compute the OLS estimates whether or not
this assumption holds, or even if there is an underlying
population model.
 Showing OLS is unbiased

How do we show β̂1 is unbiased for β1 ? What we need to show is

E (β̂1 ) = β1 (42)
where the expected value means averaging across random samples.
Step 1: Write down a formula for β̂1 . It is convenient to use
Pn
(xi − x)yi
β̂1 = Pi=1
n 2
(43)
i=1 (xi − x)
which is one of several equivalent forms.
It is convenient to define SSTx = ni=1 (xi − x)2 , to total variation
P
in the xi , and write
Pn
(xi − x)yi
β̂1 = i=1 (44)
SSTx
Remember, SSTx is just some positive number. The existence of
β̂1 is guaranteed by SLR.3.

Step 2: Replace each yi with yi = β0 + β1 xi + ui (which uses

SLR.1 and the fact that we have data from SLR.2).
The numerator becomes
n
X n
X
(xi − x)yi = (xi − x)(β0 + β1 xi + ui ) (45)
i=1 i=1
n
X n
X n
X
= β0 (xi − x) + β1 (xi − x)xi + (xi − x)ui (46)
i=1 i=1 i=1
n
X n
X
= 0 + β1 (xi − x)2 + (xi − x)ui (47)
i=1 i=1
n
X
= β1 SSTx + (xi − x)ui (48)
i=1
Pn Pn Pn
We used i=1 (xi − x) = 0 and i=1 (xi − x)xi = i=1 (xi − x)2 .
We have shown

Pn Pn
β1 SSTx + i=1 (xi − x)ui i=1 (xi− x)ui
β̂1 = = β1 + (49)
SSTx SSTx
Note how the last piece is the slope coefficient from the OLS
regression of ui on xi , i = 1, ..., n. We cannot do this regression
because the ui are not observed.
Now define

(xi − x)
wi = (50)
SSTx
so we have
n
X
β̂1 = β1 + w i ui (51)
i=1

β̂1 is a linear function of the unobserved errors, ui . The wi are

all functions of {x1 , x2 , ..., xn }.
The (random) difference between β̂1 and β1 is due to this
linear function of the unobservables.
Step 3: Find E (β̂1 ).
Under Assumptions SLR.2 and SLR.4, E (ui |x1 , x2 , ..., xn ) = 0.
That means, conditional on {x1 , x2 , ..., xn },

E (wi ui |x1 , x2 , ..., xn ) = wi E (ui |x1 , x2 , ..., xn ) = 0

because wi is a function of {x1 , x2 , ..., xn }. (In the next slides I
omit the conditioning in the expectations)
This would not be true if, in the population, u and x are
correlated.
Now we can complete the proof: conditional on {x1 , x2 , ..., xn },
n
!
X
E (β̂1 ) = E β1 + w i ui (52)
i=1
n
X n
X
= β1 + E (wi ui ) = β1 + wi E (ui ) (53)
i=1 i=1

= β1 (54)

Remember, β1 is the fixed constant in the population. The

estimator, β̂1 , varies across samples and is the random outcome:
before we collect our data, we do not know what β̂1 will be.
THEOREM (Unbiasedness of OLS)
Under Assumptions SLR.1 through SLR.4

E (β̂0 ) = β0 and E (β̂1 ) = β1 . (55)

Omit the proof for β̂0 .

Each sample leads to a different estimate, β̂0 and β̂1 . Some
will be very close to the true values β0 = 3 and β1 = 2.
Nevertheless, some could be very far from those values.
If we repeat the experiment again and again, and average the
estimates, we would get very close to 2.
The problem is, we do not know which kind of sample we
have. We can never know whether we are close to the
population value.
We hope that our sample is ”typical” and produces a slope
estimate close to β1 but we can never know.
 Reminder

Errors are the vertical distances between observations and the

unknown Conditional Expectation Function. Therefore, they
are unknown.
Residuals are the vertical distances between observations and
the estimated regression function. Therefore, they are known.
 SE and the data

The correct SE estimation procedure is given by the underlying

structure of the data
It is very unlikely that all observations in a dataset are
unrelated, but drawn from identical distributions
(homoskedasticity)
For instance, the variance of income is often greater in
families belonging to top deciles than among poorer families
(heteroskedasticity)
Some phenomena do not affect observations individually, but
they do affect groups of observations uniformly within each
group (clustered data)
 Variance of the OLS Estimators

Under SLR.1 to SLR.4, the OLS estimators are unbiased. This

tells us that, on average, the estimates will equal the
population values.
But we need a measure of dispersion (spread) in the sampling
distribution of the estimators. We use the variance (and,
ultimately, the standard deviation).
We could characterize the variance of the OLS estimators
under SLR.1 to SLR.4 (and we will later). For now, it is
easiest to introduce an assumption that simplifies the
calculations.
Assumption SLR.5 (Homoskedasticity, or Constant Variance)
The error has the same variance given any value of the explanatory
variable x:

Var (u|x) = σ 2 > 0 (56)

where σ 2 is (virtually always) unknown.

Because we assume SLR.4, that is, E (u|x) = 0 whenever we

assume SLR.5, we can also write

E (u 2 |x) = σ 2 = E (u 2 ) (57)
Under the population Assumptions SLR.1 (y = β0 + β1 x + u),
SRL.4 (E (u|x) = 0) and SLR.5 (Var (u|x) = σ 2 ),

E (y |x) = β0 + β1 x (58)
2
Var (y |x) = σ (59)

So the average or expected value of y is allowed to change with x

– in fact, this is what interests us – but the variance does not
change with x. (See Graphs on next two slides)
THEOREM (Sampling Variances of OLS)
Under Assumptions SLR.1 to SLR.2,

σ2 σ2
Var (β̂1 |x) = Pn 2
= (60)
i=1 (xi − x) SSTx
2 −1
Pn 2


σ n i=1 xi
Var (β̂0 |x) = (61)
SSTx
(conditional on the outcomes {x1 , x2 , ..., xn }).
To show this, write, as before,
n
X
β̂1 = β1 + w i ui (62)
i=1

where wi = (xi − x)/SSTx . We are treating this as nonrandom in

the derivation. Because β1 is a constant, it does not affect
Var (β̂1 ). Now, we need to use the fact that, for uncorrelated
random variables, the variance of the sum is the sum of the
variances.
The {ui : i = 1, 2, ..., n} are actually independent across i, and so
they are uncorrelated. So (remember that if we know x, we know
w)

n
!
X
Var (β̂1 |x) = Var wi ui |x (63)
i=1
n
X n
X
= Var (wi ui |x) = wi2 Var (ui |x) (64)
i=1 i=1
n
X n
X
= wi2 σ 2 =σ 2
wi2 (65)
i=1 i=1

where the second-to-last equality uses Assumption SLR.5, so that

the variance of ui does not depend on xi .
Now we have

n n Pn 2
X X (xi − x)2 i=1 (xi − x)
wi2 = = (66)
(SSTx )2 (SSTx )2
i=1 i=1
SSTx 1
= 2
= (67)
(SSTx ) SSTx

We have shown

σ2
Var (β̂1 ) = (68)
SSTx
Usually we are interested in β1 . We can easily study the two
factors that affect its variance.

σ2
Var (β̂1 ) = (69)
SSTx

1 As the error variance increases, i.e, as σ 2 increases, so does

Var (β̂1 ). The more “noise” in the relationship between y and
x – that is, the larger variability in u – the harder it is to learn
about β1 .
2 By contrast, more variation in {xi } is a good thing:

SSTx ↑ implies Var (β̂1 ) ↓ (70)

Notice that SSTx /n is the sample variance in x. We can think of
this as getting close to the population variance of x, σx2 , as n gets
large. This means

SSTx ≈ nσx2 (71)

which means, as n grows, Var (β̂1 ) shrinks at the rate 1/n. This is
why more data is a good thing: it shrinks the sampling variance of
our estimators.
The standard deviation of β̂1 is the square root of the variance. So
σ
sd(β̂1 ) = √ (72)
SSTx
This turns out to be the measure of variation that appears in
confidence intervals and test statistics.
Estimating the Error Variance
In the formula

σ2
Var (β̂1 ) = (73)
SSTx
we can compute SSTx from {xi : i = 1, ..., n}. But we need to
estimate σ 2 .
Recall that

σ 2 = E (u 2 ). (74)
Therefore, if we could observe a sample on the errors,
{ui : i = 1, 2, ..., n}, an unbiased estimator of σ 2 would be the
sample average
n
X
n−1 ui2 (75)
i=1

But this not an estimator because we cannot compute it from the

data we observe, since ui are unobserved.
How about replacing each ui with its “estimate”, the OLS residual
ûi ?

ui = yi − β0 − β1 xi (76)
ûi = yi − β̂0 − β̂1 xi (77)
ûi can be computed from the data because it depends on the
estimators β̂0 and β̂1 . Except by fluke,

ûi 6= ui (78)
for any i.

ûi = yi − β̂0 − β̂1 xi = (β0 + β1 xi + ui ) − β̂0 − β̂1 xi (79)

= ui − (β̂0 − β0 ) − (β̂1 − β1 )xi (80)

E (β̂0 ) = β0 and E (β̂1 ) = β1 , but the estimators almost always

differ from the population values in a sample.
Now, what about this as an estimator of σ 2 ?
n
X
n−1 ûi2 = SSR/n (81)
i=1

It is a true estimator and easily computed from the data after OLS.
As it turns out, this estimator is slightly biased: its expected value
is a little less than σ 2 .
The estimator does not account for the two restrictions on the
residuals, used to obtain β̂0 and β̂1 :

n
X
ûi = 0 (82)
i=1
n
X
xi ûi = 0 (83)
i=1

There is no such restriction on the unobserved errors.

The unbiased estimator of σ 2 uses a degrees-of-freedom
adjustment. The residuals have only n − 2 degrees-of-freedom, not
n.

SSR
σ̂ 2 = (84)
(n − 2)
THEOREM: Unbiased Estimator of σ 2
Under Assumptions SLR.1 to SLR.5,

E (σ̂ 2 ) = σ 2 (85)
In regression output, it is
s
√ SSR
σ̂ = σ̂ 2 = (86)
(n − 2)
that is usually reported. This is an estimator of sd(u), the
standard deviation of the population error. And SSR = ni=1 ub2 .
P
σ̂ is called the standard error of the regression, which
means it is an estimate of the standard deviation of the error
in the regression. Stata calls it the root mean squared error.
Given σ̂, we can now estimate sd(β̂1 ) and sd(β̂0 ). The
estimates of these are called the standard errors of the β̂j .
We just plug σ̂ in for σ:
σ̂
se(β̂1 ) = √ (87)
SSTx
where both the numerator and denominator are computed
from the data.
For reasons we will see, it is useful to report the standard
errors below the corresponding coefficient, usually in
parentheses.
OLS inference is generally faulty in the presence of
heteroskedasticity
Fortunately, OLS is still useful
Assume SLR.1-4 hold, but not SLR.5. Therefore

Var (ui |xi ) = σi2

The variance of our estimator, βb1 equals:

Pn
(xi − x)2 σi2
Var (βb1 ) = i=1
SSTx2

When σi2 = σ 2 for all i, this formula reduces to the usual

σ2
form, SST 2
x
A valid estimator of Var(βb1 ) for heteroskedasticity of any form
(including homoskedasticity) is

− x)2 ubi 2
Pn
i=1 (xi
Var (βb1 ) =
SSTx2

which is easily computed from the data after the OLS

regression
As a rule, you should always use the , robust command in
STATA.
 Clustered data

Observations are related to each other within certain groups

You want to regress kids’ grades on class size to determine the
effect of class size on grades
The unobservables of kids belonging to the same classroom
will be correlated (e.g., teacher quality, recess routines) while
will not be correlated with kids in far away classrooms
This means we are assuming independence across “clusters”
but correlation within clusters.
 Cluster robust standard errors

Using cluster-robust standard errors changes somewhat

Instead of summing over each individual, we first sum over
groups
 Clustered data

Let’s stack the observations and use matrix notation. Stack

observations by cluster

yg = xg β + ug

The OLS estimator of β is:

βb = [X 0 X ]−1 X 0 y

we just stacked the data is all

The variance is given by:

Var (β) = E [[X 0 X ]−1 X 0 ΩX [X 0 X ]−1 ]

 Clustered data

With this in mind, we can now write the variance-covariance

matrix for clustered data
G
X
b = [X 0 X ]−1 [
Var (β) xg0 ubg ubg0 xg ][X 0 X ]−1
i=1

In STATA: vce(cluster clustervar). Where clustervar

is a variable that identifies the groups in which unobservables
are allowed to correlate
 The importance of knowing your data

In real world you should never go with the “independent and

identically distributed” (i.e., homoskedasticity) case. Life is
not that simple.
You need to know your data in order to choose the correct
error structure and then infer the required SE calculation
If you have aggregate variables, like class size, clustering at
that level is required
 Outline: CEF and Linear regression

Properties of the conditional expectation function (CEF)

Reasons for using linear regression
Regression anatomy theorem
Omitted variable bias
 Properties of the conditional expectation function

Now we relax the assumption that x is fixed across samples

and instead assume it is a random variable
Assume we are interested in the returns to schooling in a
wage regression. We can summarize the predictive power of
schooling’s effect on wages with the conditional expectation
function
E (yi |xi ) (88)
The CEF for a dependent variable, yi , given covariates Xi , is
the expectation, or population average, of yi with xi held
constant.
E (yi |xi ) gives the expected value of y for given values of x
It provides a reasonable representation of how y changes with
x
If x is random, then E (yi |xi ) is a random function
We write the CEF as E (yi |xi ) and note that it’s explicitly a
function of xi , and because xi is a random variable, so is the
CEF
We’re often interested in CEFs that are functions of many
variables, conveniently subsumed in the vector xi , and for a
specific value of xi , we will write

E (yi |xi = x)
 Helpful result: Law of Iterated Expectations

One result we will use in the following theorems is called law of

iterated expectations. It’s fairly intuitive.
Definition of Law of Iterated Expectations (LIE)
The unconditional expectation of a random variable is equal to the
expectation of the conditional expectation of the random variable
conditional on some other random variable

E (Y ) = E (E [Y |X ])

.
Say that the population is divided by gender. We could take
conditional expectations by gender and combine them
(properly weighted) to get the unconditional expectation

E [IQ] = E (E [IQ|Sex])
X
= Pr (Sexi ) · E [IQ|Sexi ]
Sexi
= Pr (Male) · E [IQ|Male]
+Pr (Female) · E [IQ|Female]

In other words, the average of the conditional averages is the

unconditional average.
Proof.
For the continuous case:
Z
E [E (Y |X )] = E (Y |X = u)gx (u)du
Z Z 
= tfy |x (t|X = u)dt gx (u)du
Z Z
= tfy |x (t|X = u)gx (u)dudt
Z Z 
= t fy |x (t|X = u)gx (u)du dt
Z
= t [fx,y du] dt
Z
= tgy (t)dt
= E (y )
Proof.
For the discrete case,
X
E (E [Y |X ]) = E [Y |X = x]p(x)
x
!
X X
= yp(y |x) p(x)
x y
XX
= yp(x, y )
x y
X X
= y p(x, y )
y x
X
= yp(y )
y
= E (Y )
 Property 1: CEF Decomposition Property

Theorem 3.1.1: The CEF Decomposition Property

yi = E (yi |xi ) + ui
where
1 ui is mean independent of xi ; that is

E (ui |xi ) = 0

2 ui is uncorrelated with any function of xi

In words: The theorem says that any random variable, yi , can be
decomposed into two parts: the part that can be explained by xi
and the part left over that can’t be which is by definition
unexplained by xi . Proof is in Angrist and Pischke (ch. 3)
 Property 2: CEF Prediction Property

Theorem 3.1.2: The CEF Prediction Property

Let m(xi ) be any function of xi . The CEF solves

E (yi |xi ) = arg minm(xi ) E [(yi − m(xi ))2 ].

In words: The CEF is the minimum mean squared error predictor

of yi given xi .

Proof is in Angrist and Pischke (ch. 3)

 Property 3: ANOVA Theorem

Theorem 3.1.3: The ANOVA (Analysis of variance) Theorem

The final property of the CEF is the ANOVA theorem which states

V (yi ) = V [E (yi |xi )] + E [V (yi |xi )]

where V (·) is the variance and V (yi |xi ) is the conditional variance
of yi given xi .

Proof is in Angrist and Pischke (ch. 3).

 3 reasons why linear regression may be of interest

Angrist and Pischke argue that the linear regression may be of

interest to you – even if the underlying CEF is not linear! We
review some of the linear theorems now. These are merely to
justify the use of linear models to approximate the CEF.
Theorem 3.1.4 - The Linear CEF Theorem
Suppose the CEF is linear. Then the population regression is it.

Comment: Trivial theorem imho because if the population CEF is

linear, then of course you should linear regression to estimate it.
Duh.
Proof in Angrist and Pischke (ch. 3). Proof uses the CEF
Decomposition Property from earlier.
Theorem 3.1.5 - The Best Linear Predictor Theorem
1 The CEF, E (y |x ), is the minimum mean squared error
i i
(MMSE) predictor of yi given xi in the class of all functions xi
by the CEF prediction property
2 The population regression function, E (xi yi )E (xi xi0 )−1 , is the
best we can do in the class of all linear functions
Proof is in Angrist and Pischke (ch. 3).
Theorem 3.1.6: The Regression CEF Theorem
The function xi β provides the minimum mean squared error
(MMSE) linear approximation to E (yi |xi ), that is

β = arg minb E {(E (yi |xi ) − xi0 b)2 }

Again, proof in Angrist and Pischke (ch. 3).

 Theorem 3.1.7: Regression anatomy theorem

Regression anatomy theorem is maybe more intuitive with an

example and some data visualization. It concerns multiple
linear regression.
Can we estimate the causal effect of family size on labor
supply by regressing labor supply (workforpay) on family size
(numkids)?

workforpayi = β0 + β1 numkidsi + ui
. regress workforpay numkids

where the first line is the causal / econometric model, and the
second line is the regression command in STATA
If family size is random, then number of kids is uncorrelated
with the unobserved error term, which means we can interpret
βb1 as the causal effect.
Example: if Melissa has no children in reality (i.e.,
numkids= 0) and we wanted to know what the effect on labor
supply will be if we surgically manipulated her family size (i.e.,
numkids = 1) then βb1 would be our answer
Visual: Even better, we could just plot the regression
coefficient in a scatter plot showing all i (workforpay,
numkids) pairs and the slope coefficient would be the best fit
of the data through these points, as well as tell us the average
causal effect of family size on labor supply
But how do we interpret βb1 if numkids is non-random?
Assume that family size is random once we condition on race,
age, marital status and employment. Then the model is:

Workforpayi = β0 + β1 Numkidsi + γ1 Whitei + γ2 Marriedi

+γ3 Agei + γ4 Employedi + ui

If we want to estimate average causal effect of family size on

labor supply, we will need two things:
1 a data set with all 6 variables;
2 numkids must be randomly assigned conditional on the other
4 variables
Now how do we interpret βb1 ? And can we visualize βb1 when
there’s multiple dimensions to the data? Yes, using the
regression anatomy theorem, we can.
Theorem 3.1.7: Regression Anatomy Theorem
Assume your main multiple regression model of interest:

yi = β0 + β1 x1i + · · · + βk xki + · · · + βK xKi + ei

and an auxiliary regression in which the variable x1i is regressed on

all the remaining independent variables

x1i = γ0 + γk−1 xk−1i + γk+1 xk+1i + · · · + γK xKi + fi

and x̃1i = x1i − xb1i being the residual from the auxiliary regression.
The parameter β1 can be rewritten as:

Cov (yi , x̃1i )

β1 =
Var (x̃1i )

In words: The regression anatomy theorem is about interpretation.

It says that βb1 is simply a scaled covariance with the x˜1 residual
used instead of the actual data x.
I think a more detailed proof could be helpful, so I’m leaving it in the slides for now.
Regression Anatomy Proof
To prove the theorem, note E [x̃ki ] = E [xki ] − E [b
xki ] = E [fi ], and plug yi and
residual x̃ki from xki auxiliary regression into the covariance cov (yi , x̃ki )

cov (yi , x̃ki )

βk =
var (x̃ki )
cov (β0 + β1 x1i + · · · + βk xki + · · · + βK xKi + ei , x̃ki )
=
var (x̃ki )
cov (β0 + β1 x1i + · · · + βk xki + · · · + βK xKi + ei , fi )
=
var (fi )

1 Since by construction E [fi ] = 0, it follows that the term β0 E [fi ] = 0.

2 Since fi is a linear combination of all the independent variables with the
exception of xki , it must be that

β1 E [fi x1i ] = · · · = βk−1 E [fi xk−1i ] = βk+1 E [fi xk+1i ] = · · · = βK E [fi xKI ] = 0
Regression Anatomy Proof (cont.)
3 Consider now the term E [ei fi ]. This can be written as:

E [ei fi ] = E [ei fi ]
= E [ei x̃ki ]
= E [ei (xki − xbki )]
= E [ei xki ] − E [ei x̃ki ]

Since ei is uncorrelated with any independent variable, it is also uncorrelated

with xki : accordingly, we have E [ei xki ] = 0. With regard to the second term of
the subtraction, substituting the predicted value from the xki auxiliary
regression, we get

E [ei x̃ki ] = E [ei (γb0 + γb1 x1i + · · · + γ bk+1 xk+1i + · · · + γ

bk−1 xk−1 i + γ bK xKi )]

Once again, since ei is uncorrelated with any independent variable, the

expected value of the terms is equal to zero. Then, it follows E [ei fi ] = 0.
Regression Anatomy Proof (cont.)
4 The only remaining term is E [βk xki fi ] which equals E [βk xki x̃ki ] since fi = x̃ki . The
term xki can be substituted using a rewriting of the auxiliary regression model, xki ,
such that
xki = E [xki |X−k ] + x̃ki
This gives

E [βk xki x̃ki ] = E [βk E [x̃ki (E [xki |X−k ] + x̃ki )]]

= βk E [x̃ki (E [xki |X−k ] + x̃ki )]
= βk {E [x̃ki2 ] + E [(E [xki |X−k ]x̃ki )]}
= βk var (x̃ki )

which follows directly from the orthogonoality between E [xki |X−k ] and x̃ki . From
previous derivations we finally get

cov (yi , x̃ki ) = βk var (x̃ki )

which completes the proof.

 STATA command: reganat (i.e., regression anatomy)

. ssc install reganat, replace

. sysuse auto
. regress price length weight headroom mpg
. reganat price length weight headroom mpg, dis(weight length) biline
 Big picture

1 Regression provides the best linear predictor for the dependent

variable in the same way that the CEF is the best unrestricted
predictor of the dependent variable
2 If we prefer to think of approximating E (yi |xi ) as opposed to
predicting yi , the regression CEF theorem tells us that even if
the CEF is nonlinear, regression provides the best linear
approximation to it.
3 Regression anatomy theorem helps us interpret a single slope
coefficient in a multiple regression model by the
aforementioned decomposition.
 Omitted Variable Bias

A typical problem is when a key variable is omitted. Assume

schooling causes earnings to rise:

Yi = β0 + β1 Si + β2 Ai + ui

Yi = log of earnings
Si = schooling measured in years
Ai = individual ability

Typically the econometrician cannot observe Ai ; for instance, the

Current Population Survey doesn’t present adult respondents’ family
background, intelligence, or motivation.
What are the consequences of leaving ability out of the
regression? Suppose you estimated this short regression
instead:
Yi = β0 + β1 Si + ηi
where ηi = β2 Ai + ui ; β0 , β1 , and β2 are population
regression coefficients; Si is correlated with ηi through Ai
only; and ui is a regression residual uncorrelated with all
regressors by definition.
 Derivation of Ability Bias

Suppressing the i subscripts, the OLS estimator for β1 is:

Cov (Y , S) E [YS] − E [Y ]E [S]

βb1 = =
Var (S) Var (S)

Plugging in the true model for Y , we get:

Cov [(β0 + β1 S + β2 A + u), S]

βb1 =
Var (S)
E [(β0 S + β1 S 2 + β2 SA + uS)] − E (S)E [β0 + β1 S + β2 A + u]
=
Var (S)
β1 E (S 2 ) − β1 E (S)2 + β2 E (AS) − β2 E (S)E (A) + E (uS) − E (S)E (u)
=
Var (S)
Cov (A, S)
= β1 + β2
Var (S)

If β2 > 0 and Cov(A, S)> 0 the coefficient on schooling in the shortened

regression (without controlling for A) would be upward biased
 Summary

When Cov (A, S) > 0 then ability and schooling are correlated.
When ability is unobserved, then not even multiple regression
will identify the causal effect of schooling on wages.
Here we see one of the main justifications for this class – what
will we do when the treatment variable is endogenous?
Because endogeneity means the causal effect has not been
identified.
 Overview
Part 1: The Core
Part 2: Selection on observables Regression fundamentals
Part 3: Selection on unobservables Randomized Experiments
Part 4: Advanced material Directed Acyclical Graphs
Conclusion
References

Introduction to Selection and Randomization

Potential outcomes concepts and definitions

Random assignment and selection bias
Krueger (1999)

Cunningham Causal Inference

 Introduction to the Selection Problem

A lot of the language we use in the class is from the medical

literature (e.g., “treatment”, “control”)
Simple example introducing potential outcomes notation and
the selection problem: “Do hospitals make people healthier?”
National Health Interview Survey (NHIS) 2005
Health status measured from 1 (poor health) to 5 (excellent
health)
Group Sample Size Mean Health Status Std. Error
Hospital 7,774 3.21 0.014
No hospital 90,049 3.93 0.003
 Introductory Notation

Think of hospitalization as a “treatment” and health status as

an “outcome”.
Let the treatment (hospitalization) be a binary variable:
(
1 if hospitalized
Di =
0 if not hospitalized

where i indexes an individual observation, such as a person

Observed outcomes (health status) is Yi
Causal question: D → Y , “Does hospitalization cause health
to improve?”
Note: we did not ask “is hospitalization correlated with
(D,Y )
health?” n1 √Cov √
VarD VarY
What’s the difference between the two questions? What does
cause mean you think? What does correlation mean?
 Introduction to Rubin Causal Model

Observed variables:
Treatment, Di , is observed as either 0 or 1 for each i unit.
Actual outcomes, Y , are observed for each unit i
Unobserved counterfactual variables:
Each individual i has one counterfactual (potential) outcome
that in principle exists but is not observed:
(
Yi1 if Di = 1
Potential outcome =
Yi0 if Di = 0

Yi1 is the health of individual i (e.g. Jill) in the hospital

Yi0 is the health of the same individual i (e.g. also Jill) not in
the hospital
 Important definitions

Definition 1: Individual treatment Definition 2: Average treatment effect

effect (ATE)
The individual treatment effect, δi , The average treatment effect is the
equals Yi1 − Yi0 population average of all i individual
treatment effects

Definition 3: Switching equation E [δi ] = E [Yi1 − Yi0 ]

An individual’s observed health = E [Yi1 ] − E [Yi0 ]
outcomes, Y , is determined by
treatment assignment, Di , and
corresponding potential outcomes: Definition 4: Fundamental problem of
causal inference
Yi = Di Yi1 + (1 − Di )Yi0
It is impossible to observe both Yi1
and Yi0 for the same individual and so
(
Yi1 if Di = 1
Yi = individual causal effects, δi , are
Yi0 if Di = 0
unknowable.
 Counterfactuals

Consider the following two situations:

1 Jack is in the hospital (DJack = 1) and his health is a 2
(YJack = 2).
2 Jill is not in the hospital (DJill = 0) and her health is a 4
(YJill = 4)
According to definition 3 (switching equation):
1 Jack’s observed health outcome, YJack = 2 equals his potential
1
health outcome under treatment, YJack =2
2 Jill’s observed health outcome, YJill = 4 equals her potential
0
health outcome under control, YJill =4
According to definitions 1, 2 and 4:
1 We do not know Jack’s potential health outcome under control
0
(i.e., YJack ) so therefore we do not know his treatment effect
1 0
since δJack = YJack − YJack
2 We do not know Jill’s potential health outcome under
1
treatment (i.e., YJill ) so therefore we do not know her
1 0
treatment effect since δJill = YJill − YJill
3 Therefore we do not know the average treatment effect (i.e.,
1 0
E [δ]) because we are missing YJill and YJack – the missing
counterfactuals (definition 4) – which are needed to calculate
ATE
We cannot calculate the average treatment effect because we
are missing individual treatment effects, and we are missing
individual treatment effects because we are missing
counterfactuals for each unit i
 Conditional Average Treatment Effects

Definition 5: Average Treatment Effect on the Treated (ATT)

The average treatment effect on the treatment group is equal to
the average treatment effect conditional on being a treatment
group member:

E [δ|D = 1] = E [Y 1 − Y 0 |D = 1]
= E [Y 1 |D = 1] − E [Y 0 |D = 1]

Definition 6: Average Treatment Effect on the Untreated (ATU)

The average treatment effect on the untreated group is equal to
the average treatment effect conditional on being untreated:

E [δ|D = 0] = E [Y 1 − Y 0 |D = 0]
= E [Y 1 |D = 0] − E [Y 0 |D = 0]
 Simple difference in means

The fundamental problem of causal inference states that since

we cannot observe both YJack1 0
and YJack (or any i for that
matter) at the same moment in time, then we cannot
calculate the causal effect of hospitalization on health
outcomes.
But we have to make due with observable health outcomes,
Y , since that’s all we got
If we subtract Jill’s observed health outcome (YJill = 4) from
Jack’s observed health outcome, (YJack = 2), we get:

(YJack |DJack = 1) − (YJill |DJill = 0) = 2 − 4 = −2

which implies that hospitalization caused health to go from

good to poor which is probably ridiculous.
Do hospitalizations make people sick? Or do sick people go to
hospitals? This is the “selection problem” – without the full
potential outcomes, the simple difference between treatment
and control group won’t tell us the causal effect of
hospitalization on health outcomes.
Same problem carries over to the simple difference in means,
E [Yi |Di = 1] − E [Yi |Di = 0], too
So what are we actually measuring if we compare average
health status for the hospitalized with that of the
non-hospitalized?
Definition 7: Simple difference in mean outcomes (SDO)
A simple difference in mean outcomes (SDO) is the difference
between the population average outcome for the treatment and
control groups, and can be approximated by the sample averages:

E [Y 1 |D = 1] − E [Y 0 |D = 0] = EN [yi |di = 1] − EN [yi |di = 0]

in large samples. It is called the naive average treatment effect in

Morgan and Winship.

Notice that . . . :
All individuals in the population contribute twice to ATE,
whereas a sampled individual is used only once to estimate
SDO by contributing to either EN [yi |di = 1] or EN [yi |di = 0].
Statistical models, such as SDO, are valuable insofar as they
can provide unbiased and/or consistent estimates of the
parameter of interest (i.e., ATE). But notice the subtle
difference between the LHS and RHS:
<
SDO > ATE
E [y |d = 1] − E [y |d = 0] <
> E [Y 1 ] − E [Y 0 ]

The LHS term is the estimator of the RHS term which is a

parameter, and estimators can be biased.
 Biased simple difference in mean outcomes

Decomposition of the simple difference in mean outcomes (SDO)

The simple difference in mean outcomes can be decomposed into three parts
(ignoring sample average notation):

E [Y 1 |D = 1] − E [Y 0 |D = 0] = ATE
+E [Y 0 |D = 1] − E [Y 0 |D = 0]
+(1 − π)(ATT − ATU) (89)

How do we interpret this?

First line is the parameter of interest: the average treatment effect, E [Y 1 − Y 0 ]
Second line is one of the most important sources of bias: the “selection bias”
which is the difference in potential health outcome for the treatment and control
group had neither received any hospitalization. In other words, how did Jack and
Jill’s health status compare to one another had Jack not gone to the hospital?
Was he sicker than her already?
Third line notes that part of the bias is due to heterogeneous treatment effects
weighted by the share of the population in the control group
Proof from Morgan and Winship, p. 26 on following slides
 Decomposition of SDO

ATE is equal to sum of conditional average expectations by LIE

ATE = E [Y 1 ] − E [Y 0 ]
= {πE [Y 1 |D = 1] + (1 − π)E [Y 1 |D = 0]}
−{πE [Y 0 |D = 1] + (1 − π)E [Y 0 |D = 0]}

Use simplified notations

E [Y 1 |D = 1] = a
E [Y 1 |D = 0] = b
E [Y 0 |D = 1] = c
E [Y 0 |D = 0] = d
ATE = e

Rewrite ATE

e = {πa + (1 − π)b} − {πc + (1 − π)d}

Move SDO terms to LHS

e = {πa + (1 − π)b} − {πc + (1 − π)d}

e = πa + b − πb − πc − d + πd
e = πa + b − πb − πc − d + πd + (a − a) + (c − c) + (d − d)
= e − πa − b + πb + πc + d − πd − a + a − c + c − d + d
a − d = e − πa − b + πb + πc + d − πd + a − c + c − d
a − d = e + (c − d) + a − πa − b + πb − c + πc + d − πd
a − d = e + (c − d) + (1 − π)a − (1 − π)b + (1 − π)d − (1 − π)c
a − d = e + (c − d) + (1 − π)(a − c) − (1 − π)(b − d)

Substitute conditional means

E [Y 1 |D = 1] − E [Y 0 |D = 0] = ATE
+(E [Y 0 |D = 1] − E [Y 0 |D = 0])
+(1 − π)({E [Y 1 |D = 1] − E [Y 0 |D = 1]}
−(1 − π){E [Y 1 |D = 0] − E [Y 0 |D = 0]})
1 0
E [Y |D = 1] − E [Y |D = 0] = ATE
+(E [Y 0 |D = 1] − E [Y 0 |D = 0])
+(1 − π)(ATT − ATU)
Decomposition of difference in means

EN [yi |di = 1] − EN [yi |di = 0] = E [Y 1 ] − E [Y 0 ]

| {z } | {z }
SDO Average Treatment Effect

+ E [Y |D = 1] − E [Y 0 |D = 0]
0
| {z }
Selection bias
+ (1 − π)(ATT − ATU)
| {z }
Heterogenous treatment effect bias

where EN [yi |di = 1] → E [Y 1 |D = 1],

EN [yi |di = 0] → E [Y 0 |D = 0] and (1 − π) is the share of the
population in the control group.
 Independence assumption

Independence assumption
Treatment is independent of potential outcomes

(Y 0 , Y 1 ) ⊥
⊥D

In words: Random assignment means that the treatment has been

assigned to units independent of their potential outcomes. Thus,
mean potential outcomes for the treatment group and control
group are the same for a given state of the world

E [Y 1 |D = 1] = E [Y 1 |D = 0]
E [Y 0 |D = 1] = E [Y 0 |D = 0]
 Random Assignment Solves the Selection Problem

EN [yi |di = 1] − EN [yi |di = 0] = E [Y 1 ] − E [Y 0 ]

| {z } | {z }
SDO Average Treatment Effect

+ E [Y |D = 1] − E [Y 0 |D = 0]
0
| {z }
Selection bias
+ (1 − π)(ATT − ATU)
| {z }
Heterogenous treatment effect bias

Notice that the selection bias from the second line of the
decomposition of SDO was:
E [Y 0 |D = 1] − E [Y 0 |D = 0]
If treatment is independent of potential outcomes, then swap
out equations and selection bias zeroes out:
E [Y 0 |D = 1] − E [Y 0 |D = 0] = E [Y 0 |D = 0] − E [Y 0 |D = 0]
= 0
 Random Assignment Solves the Heterogenous Treatment Effects

How does randomization affect heterogeneity treatment effects bias from the
third line? Rewrite definitions for ATT and ATU:

ATT = E [Y 1 |D = 1] − E [Y 0 |D = 1]
ATU = E [Y 1 |D = 0] − E [Y 0 |D = 0]

Rewrite the third row bias after 1 − π:

ATT − ATU = E[Y1 | D=1] − E [Y 0 |D = 1]

−E[Y1 | D=0] + E [Y 0 |D = 0]
= 0

If treatment is independent of potential outcomes, then:

EN [yi |di = 1] − EN [yi |di = 0] = E [Y 1 ] − E [Y 0 ]

SDO = ATE
 What independence does not mean

Notice – independent treatment assignment means that

E [Y 1 |D = 1] = E [Y 1 |D = 0]

and the equivalent for Y 0 .

But that does not in any way imply that there is no causal
effect. Independence does not imply, in other words, that
E [Y 1 |D = 1] is equal to E [Y 0 |D = 0].
Independence only implies that the the average values for a
given potential outcome (i.e., Y 1 or Y 0 ) are the same for the
groups who did receive the treatment as those who did not
 SUTVA

The potential outcomes model presupposes a set of bundled

assumptions called the “stable unit-treatment value
assumption”
1 S: stable
2 U: across all units, or the population
3 TV: treatment-value (“treatment effect”, “causal effect”)
4 A: assumption
SUTVA means that average treatment effects are parameters
that assume (1) homogenous dosage and (2) potential
outcomes are invariant to who else (and how many) is treated.
 SUTVA: Homogenous dose

SUTVA constrains what the treatment can be.

Individuals are receiving the same treatment – i.e., the “dose”
of the treatment to each member of the treatment group is
the same. That’s the “stable unit” part.
If we are estimating the effect of hospitalization on health
status, we assume everyone is getting the same dose of the
hospitalization treatment.
Easy to imagine violations if hospital quality varies, though,
across individuals. But, that just means we have to be careful
what we are and are not defining as the treatment
 SUTVA: No externalities

What if hospitalizing Jack (hospitalized, D = 1) is actually

about vaccinating Jack from small pox?
If Jack is vaccinated for small pox, then Jill’s potential health
status (without vaccination) may be higher than when he isn’t
vaccinated.
0 , may vary with what Jack does regardless
In other words, YJill
of whether she herself receives treatment.
SUTVA means that you don’t have a problem like this.
If there are no externalities from treatment, then δi is stable
for each i unit regardless of whether someone else receives the
treatment too.
 SUTVA: Partial equilibrium only

Easier to imagine this with a different example.

Let’s say we are estimating the effect of some technological
innovation that lowers the cost functions to firms in
competitive markets.
A decrease in cost raises profits in the short-run, but positive
profits leads to firm entry in the long-run.
Firm entry in the long-run causes the supply curve to shift
right, pushing market prices down until price equals average
total cost.
The first effect – short-run responses to decreases in cost –
are the only things we can estimate with potential outcomes.
 Example 1: Krueger (1999)

Krueger (1999) econometrically re-analyzes a randomized

experiment to determine the causal effect of class size on
student achievement
The project is the Tennessee Student/Teacher Achievement
Ratio (STAR) experiment from the 1980s
11,600 students and their teachers were randomly assigned to
one of the following three groups:
1 Small class of 13-17 students
2 Regular class of 22-25 students
3 Regular class of 22-25 students with a full-time teacher’s aide
After the assignment, the design called for students to remain
in the same class type for four years
Randomization occurred within schools
 Regression analysis of experiments

With randomization one could simply calculate SDO (simple difference in mean
outcomes) for the treatment and control group and know that SDO=ATE
because of independence
Nonetheless, it is often useful to analyze experimental data with regression
analysis (see MW section 3.2.2; MHE ch. 2)
Assume that treatment effects are constant – i.e., Yi1 − Yi0 = δ ∀i
Substitute into a rearranged switching equation (Definition 2):

Yi = Di Yi1 + (1 − Di )Yi0
Yi = Yi0 + (Yi1 − Yi0 )Di
Yi = Yi0 + δDi
Yi = E [Yi0 ] + δDi + Yi0 − E [Yi0 ]
Yi = α + δDi + ηi

where ηi is the random part of Yi0

This is a regression equation that could be used to estimate the causal effect of
D on Y
 Regression analysis of experiments (cont.)

The conditional expectation, E [Yi |Di ], with treatment status

switched on and off gives:

E [Yi |Di = 1] = α + δ + E [ηi |Di = 1]

E [Yi |Di = 0] = α + E [ηi |Di = 0]

Subtracting the latter from the former, we get:

E [Yi |Di = 1] − E [Yi |Di = 0] = |{z}

δ
| {z }
SDO Treatment Effect

+ E [ηi |Di = 1] − E [ηi |Di = 0]

| {z }
Selection bias
We can estimate SDO using least squares but there’s other
options as well
In the STAR experiment, Di , equalled one if the student was
enrolled in a small class and had been randomly assigned
Recall that randomization implies that treatment is
independent of potential outcomes, and therefore the selection
bias vanishes
 Why Include Control Variables?

To evaluate experimental data, one may want to add

additional controls in the multivariate regression model. So,
instead of estimating the prior equation, we might estimate:

Yi = α + δDi + Xi0 γ + ηi

There are 2 main reasons for including additional controls in

the regression models:
1 Conditional random assignment. Sometimes randomization is
done conditional on some observable. (here that’s the school).
We’ll discuss “conditional independence assumption” when we
cover matching.
2 Additional controls increase precision. Although control
variables Xi are uncorrelated with Di , they may have
substantial explanatory power for Yi . Including controls thus
reduces variance in the residuals which lowers the standard
errors of the regression estimates.
 Regression in Krueger (1999)

Krueger estimates the following econometric model

Yics = β0 + β1 SMALLcs + β2 REG/Acs + αs + εics

i indexes a student, c indexes a class, and s indexes a school

Yics is the student’s percentile score
SMALLcs is a dummy equalling 1 if she is assigned to a small
class.
REG /Acs is a dummy equalling 1 if she was assigned to a
regular class with an aide
α is a “school fixed effect” which is a vector of school-specific
dummy variables. He conditions on school fixed effects
because randomized classroom assignment occurred within
schools.
 Regression results Kindergarten
TABLE V
OLS AND REDUCED-F ORM ESTIMATES OF EFFECT OF CLASS-SIZE ASSIGNME
AVERAGE PERCENTILE OF STANFORD ACHIEVEMENT TEST

OLS: actual class size Reduced form: initial cl

Explanatory
variable (1) (2) (3) (4) (5) (6) (7)

A. Kindergarten
Small class 4.82 5.37 5.36 5.37 4.82 5.37 5.36
(2.19) (1.26) (1.21) (1.19) (2.19) (1.25) (1.21)
Regular/aide class .12 .29 .53 .31 .12 .29 .53
(2.23) (1.13) (1.09) (1.07) (2.23) (1.13) (1.09)
White/Asian (1 5 — — 8.35 8.44 — — 8.35
yes (1.35) (1.36) (1.35)
Girl (1 5 yes) — — 4.48 4.39 — — 4.48
(.63) (.63) (.63)
Free lunch (1 5 — — 213.15 213.07 — — 213.15
yes) (.77) (.77) (.77)
White teacher — — — 2.57 — — —
(2.10)
Teacher experience — — — .26 — — —
(.10)
Master’s degree — — — 2.51 — — —
(1.06)
School éxed effects No Yes Yes Yes No Yes Yes
R2 .01 .25 .31 .31 .01 .25 .31
B. First grade
Regression results 1st grade
 Problem 1: Attrition

A common problem in randomized experiments on humans is

attrition – i.e., people leaving the experiment
If attrition is random, then attrition affects the treatment and
control groups in the same way and our estimates remain
unbiased
But in this application, attrition is probably non-random:
especially good students from large classes may have enrolled
in private schools creating a selection bias problem
Krueger addresses this concern by imputing the test scores
(from their earlier test scores) for all children who leave the
sample and then re-estimates the model including students
with imputed test scores.
 Problem 1: Attrition
EXPERIMENTAL ESTIMATES 515

TABLE VI
EXPLORATION OF EFFECT OF ATTRITION DEPENDENT VARIABLE: AVERAGE
PERCENTILE SCORE ON SAT

Actual and imputed

Actual test data test data

Coefficient Coefficient
on small Sample on small Sample
Grade class dum. size class dum. size

K 5.32 5900 5.32 5900

(.76) (.76)
1 6.95 6632 6.30 8328
(.74) (.68)
2 5.59 6282 5.64 9773
(.76) (.65)
3 5.58 6339 5.49 10919
(.79) (.63)

Estimates of reduced-form models are presented. Each regression includes the following explanatory
variables: a dummy variable indicatin g initial assignment to a small class; a dummy variable indicating initial
assignment to a regular/aide class, unrestricted school effects; a dummy variable for student gender; and a
dummy variable for student race. The reported coefficien t on small class dummy is relative to regular classes.
Standard errors are in parentheses.

had higher test scores, on average, than students assigned to

Non-random attrition hardly biases the results.
small classes who also left the sample, then the small class effects
will be biased upwards. One reason why this pattern of attrition
 N UMBER OF STUDENTS IN EACH TYPE OF CLASS

Problem 2:A. Switch Classrooms

Kindergarten to érst grade after Random Assignment
First grade

“It is virtuallyKindergarten
impossible to Small
Small
prevent
1292
some
Regular
60
students
Reg/aide
48
from switching
All
1400
Regular 126 737 663 1526
between class Aide
types over time.”
122 (Krueger
761 1999,706 p. 506)
1589
All 1540 1558 1417 4515
B. First grade to second grade
Second grade

First grade Small Regular Reg/aide All

Small 1435 23 24 1482
Regular 152 1498 202 1852
Aide 40 115 1560 1715
All 1627 1636 1786 5049
C. Second grade to third grade
Third grade
Interpreting
Second Krueger’s“transition
grade Small matrix”
Regular Reg/aide(above)
All
Small 1564 37 35 1636
If students
Regular remained in 167 their same
1485 class152type 1804 over time, all the
Aide 40 76 1857 1973
off-diagonal
All elements would
1771 be
1598zero 2044 5413

Interpretation: Of the 1,482 first graders assigned to small

classrooms, 1,435
To address remained
this potential in and
problem, small classes;
the variability 23 and 24
of class
size for into
switched a givenregular
type of assignment,
and in somewith
regular of the aide
analysis that
classes in the
follows initial random assignment is used as an instrumental
second grade
variable for actual class size.

B. Data
If students withand stronger
Standardized Tests
expected academic potential were
Students were tested at the end of March or beginning of
more likely
Aprilto move
of each year. into theconsisted
The tests smallof the
classes,
Stanford then
Achieve-these
ment Test (SAT), which measured achievement in reading, word
transitions wouldandbias
recognition, mathainsimple comparison
grades K–3, ofBasic
and the Tennessee outcomes across
Skills First (BSF) test, which measured achievement in reading
class types.
and math in grades 1–3. The tests were tailored to each grade
Problem 2: Switch Classrooms after Random Assignment

Subjects moved between treatment and control groups. How

to address this?
A common solution to this problem is to use initial classroom
assignment (i.e., small or regular classes) as an instrument for
actual assignment. We will discuss instrumental variables
later, so I will hold off on that now.
Krueger reports regression results where instead of the
student’s actual status as the treatment variable, he regresses
performance against their randomly assigned class size. This
is called the “reduced form” model, and we learn more about
this when we cover IV.
In Kindergarten, OLS and reduced form are the same because
students remained in their initial class for at least one year.
From grade 1 onwards, OLS and reduced form results differ.
Problem 2: Switch Classrooms after Random Assignment
 Problem 3: Heterogeneous Treatment Effects

Heterogeneous treatment effect bias occurs if treatment

effects differ across the population (i.e., ATT 6= ATU)
If people selecting to take part of the randomized trial have
different returns compared to the population average, then
the experiment will only identify a localized average treatment
effect for the sub-population participants in the experiment.
 Problem 4 and 5

Supply Side Changes

If programs are scaled up, the supply side implementing the
treatment may be different
In the trial phase, the supply side may be more motivated than
during the large scale roll-out of a program
Attrition bias
Attrition rates (i.e., leaving the sample between the baseline
and the follow-up surveys) may be different in treatment and
control groups
The estimated treatment effect may therefore be biased
 Problem 6 and 7

Hawthorne effects
People behave differently if they are being observed in an
experiment. Similar to “placebo effects” in that this is a false
positive result.
If they operate differently on treatment and control groups,
then they may introduce biases
If people from the control group behave differently, these
effects are sometimes called “John Henry” effects
Substitution bias
Control group members may seek substitutes for treatment
This would bias the estimated treatment effects downward.
Can you see why?
Can also occur if the experiment frees up resources that can
now be concentrated on the control group.
 Overview
Part 1: The Core
Part 2: Selection on observables Regression fundamentals
Part 3: Selection on unobservables Randomized Experiments
Part 4: Advanced material Directed Acyclical Graphs
Conclusion
References

History of graphical causal modeling in science

Path diagrams were developed by Sewell Wright, early 20th

century geneticist, for causal inference
Sewell Wright’s father, Phillip Wright, used them to prove the
existence of the “instrumental variable” estimator (see Stock
and Trebbi 2003)
Judea Pearl and colleagues in Artificial Intelligence at UCLA
developed DAG modeling to create a formalized causal
inference methodology

Cunningham Causal Inference

Three biases in observational studies DAGs help illustrate

1 Omitted variable bias

Something causes the outcome which is correlated with the
treatment
2 Reverse causality
Your logic about cause and effect is backwards
3 Conditioning on collider
Conditioning on some variables can introduce spurious
correlations
 Simple DAG

A Directed Acyclic Graph (DAG) is a set of nodes and arrows

with no directed cycles

Z X Y

Nodes represent variables

Arrows represent direct causal effects (“direct” means not
mediated by other variables in the graph)
 Complete DAGs

A DAG is a complete set of all causal relationships, but we

emphasize the causal relationships that surround D and Y
All direct causal effects among the variables in the graph
All common causes of any pair of variables in the graph
Do not leave out anything, even if you can’t measure it or
don’t have data on it.
DAGs are based on theory, a model, observation, experience,
prior studies, intuition
Try to get into the habit of accompanying every regression
you will ever run with a DAG of your own creation to help
guide your identification strategy
 Some DAG concepts

Z X Y

U is a parent of X and Y
X and Y are descendants of Z
There is a directed path from Z to Y
There are two paths from Z to U (but no directed path)
X is a collider of the path Z → X ← U
X is a noncollider of the path Z → X → Y
 Confounding

Confounding occurs when when the treatment and the

outcomes have a common cause or parent which creates
spurious correlation between D and Y
D Y

The correlation between D and Y no longer reflects the

causal effect of D on Y
 Backdoor Paths

Confounding creates backdoor paths between treatment and

outcome (D ← X → Y )
However, we can “block” the backdoor path by conditioning
on the common cause X
Once we condition on X , the correlation between D and Y
estimates the causal effect of D on Y
Conditioning means calculating
E [Y |D = 1, X ] − E [Y |D = 0, X ] for each value of X

D Y

X
 Blocked backdoor paths

A backdoor path is blocked if and only if:

It contains a noncollider that has been conditioned on
Or it contains a collider that has not been conditioned on and
has no descendents that have been conditioned on
 Blocked paths

Examples:
1 Conditioning on a noncollider blocks a path:

X Z Y
2 Conditioning on a collider opens a path:
Z X Y
3 Not conditioning on a collider blocks a path:
Z X Y
 Backdoor criterion

Backdoor criterion
Conditioning on X satisfies the backdoor criterion with respect to
(D, Y ) directed path if:
1 All backdoor paths are blocked by X
2 No element of X is a collider
In words: If X satisfies the backdoor criterion with respect to
(D, Y ), then matching on X identifies the causal effect of D on Y
Matching on all common causes is sufficient
There are two backdoor paths from D to Y

X1 D Y

X2

Conditioning on X1 and X2 blocks both backdoor paths

Matching may work even if not all common causes are
observed
U and X1 are common causes
X1

U X2 D Y

Conditioning on X1 and X2 is enough

There may be more than one set of conditioning
variables that satisfy the backdoor criterion
X1

X3 D Y

X2
X1

X3 D Y

X2

Conditioning on the common causes, X1 and X2 , is sufficient

. . . but so is conditioning on X3
 Collider bias

Matching on an outcome may create bias

There is only one backdoor path from D to Y

X1 D Y

X2

Conditioning on X1 blocks the backdoor path

But what if we also condition on X2 ?

X1 D Y

X2

Conditioning on X2 opens up a new path!

Matching on all pretreatment covariates can create bias
There is one backdoor path and it is closed
U1

X D Y

U2

No confounding – D is identified
But what if we condition on X ? Now a backdoor path opens.
U1

X D Y

U2
Ultimately, we can’t know if we have a collider bias problem,
or whether we’ve satisfied the backdoor criterion, without a
model
There’s no getting around it – all empirical work requires
theory to guide the work. Otherwise how do you know if
you’ve conditioning on a noncollider or a collider?
Put differently, you cannot identify treatment effects without
making assumptions about the process that generated your
data
 Simple example of collider bias

Important: Since unconditioned colliders block back-door paths,

what exactly does conditioning on a collider do? Let’s illustrate
with a fun example and some made-up data
CNN.com headline: Megan Fox voted worst – but sexiest –
actress of 2009 (link)
Assume talent and beauty are independent, but each causes
someone to become a movie star. What’s the correlation
between talent and beauty for a sample of movie stars
compared to the population as a whole (stars and non-stars)?
What if the sample consists only of movie stars?

Movie Star

Talent Beauty
 STATA code
clear all
set seed 3444

* 2500 independent draws from standard normal distribution

set obs 2500
generate beauty=rnormal()
generate talent=rnormal()

* Creating the collider variable (star)

gen score=(beauty+talent)
egen c85=pctile(score), p(85)
gen star=(score>=c85)
label variable star "Movie star"

* Conditioning on the top 15%

twoway (scatter beauty talent, mcolor(black) msize(small) msymbol(smx)),
ytitle(Beauty) xtitle(Talent) subtitle(Aspiring actors and actresses)
by(star, total)
 Aspiring actors and actresses
0 1

4
2
0
-2
-4
Beauty

-4 -2 0 2 4

Total
4
2
0
-2
-4

-4 -2 0 2 4
Talent
Graphs by Movie star

Figure: Top left figure: Non-star sample scatter plot of beauty (vertical axis) and talent
(horizontal axis). Top right right figure: Star sample scatter plot of beauty and talent.
Bottom left figure: Entire (stars and non-stars combined) sample scatter plot of beauty and
talent.
 Final Remarks

The backdoor criterion provides a useful graphical guide that

can be used to select matching variables
The backdoor criterion is only a set of sufficient conditions,
but covers most of the interesting cases
Applying the backdoor criterion requires knowledge of the
DAG
Suppose that there is a set of observed pretreatment covariates
and that we know if they are causes of the treatment and/or
the outcomes
Suppose that there exists a subset of the observed covariates
such that matching on them is enough to control for
confounding
Then, matching on the subset of the observed covariates that
are either a cause of the treatment or a cause of the outcome
or both is also enough to control for confounding
What if you can’t conduct randomized experiment?

Problems with the experimental design itself:

non-compliance by administrators
non-compliance by members of the treatment group
non-compliance by members of the control group
Experiments may be impractical due to:
Too expensive
Unethical
Not feasible for some other reason
 Observational alternatives to experiments

1 Selection on observables: Treatment and control groups differ

from each other on observable characteristics only
Stratification/subclassification
Matching
Propensity score matching
2 Selection on unobservables: treatment and control groups
differ from each other on unobservable characteristics
exogenous variable induces variation in treatment –
instrumental variables
selection mechanism is known – regression discontinuity
designs
treatment and controls are observed before and after
treatment – panel estimators, differences-in-differences,
synthetic control, even studies
 Readings

Textbook readings
Read MW chapter 3-4
Read MHE chapter 3
Article readings
See website under Matching estimation (Job Trainings Papers)
 GERRY B. HILL, WAYNE MILLAR and JAMES CONNELLY

Figure 1
Lung Cancer at Autopsy: Combined Results from 18 Studies

Per cent of autopsies

8
*

1860 1870 1880 1890 1900 l910 1920 1930 1940 1950
Year
Observed +fitted

Mortality Statistics
"The
TheGreat Debate"
Registrar General of England and Wales began publishing the num- 371
bers of deaths for specific cancer sites in 1911.W The death rates for can-
cer of the lung from 1911 to 1955 were
Figure 2(a)published by Percy Stocks.26The
rates increased exponentially
Mortality overof
from Cancer thetheperiod:
Lung in10% per year in males
Males
and 6% per year in females. Canadian rates for the period 1931-52 were
Rate per 100,000
published
120 by A. J. Phillips.27 The rates were consistently lower in Canada
than in- England and Wales, but also increased exponentially at 8% per
l00
year in males and 4% per year in females.
The
80 -British and Canadian rates are shown in Figure 2. The rates (a) for
males, and (b) for females have been age-standardized,28 and the trends
6 0 - to 1990, using data published by Richard Peto and colleagues, 29
extended
and40by- Statistics Canada.30In British males the rates reached a maxi-
I mum in the mid-1970's and then declined. In Canadian males the initial
rise 20
was- more prolonged, reaching a maximum in 1990.Among females
the age-standardized rates continue to climb in both countries, the rise
0
beingl910
steeper in Canada
1920 1930 1940than in Britain.
1960 1960 1970 l980 l990 2000
The fact that mortality was lower Yearat first in Canada than in Britain
may be explained by the difference in smoking in the two countries.
-England
Percy Stocks31 cited data on the+
& Wales annual
Canadaconsumption
+ United per adult of ciga-
Kingdom
rettes in various countries between 1939 and 1957. In 1939 the con-
 increases with the amount smoked.

Figure 4
Smoking and Lung Cancer Case-control Studies
376 Odds Ratio* GERRY B. HILL, WAYNE MILLAR and JAMES CONNELLY
'0 1
Cohort Studies 60
Cohort studies, though less prone to bias, are much more difficult to
perform than case-control studies, since it is necessary to assemble many
thousands of individuals, determine their smoking status, and follow
them up for several years to determine how many develop lung cancer.
Four such studies were mounted in the 1950s. The subjects used were
British doctors,61United States veterans,62 Canadian veterans,63 and vol-
unteers assembled by the American Cancer Society.@All four used mor-
tality as"the end-point.
Lees than 20 20 or more All
Figure 5 shows the combined mortality ratios for cancer of the lung in
males by level of cigarette smoking. Two of the studies involved females,
deaths m
but the numbers of lung cancerMale8 wereFemales
too small to provide precise
estimates. Since all causes of death were recorded in the cohort studies it
was Weighted
possiblemean
to determine the relationship between smoking and dis-
'
eases other than lung cancer. Sigruficant associations were found in rela-
tion to several types of cancer (e.g. mouth, pharynx, larynx, esophagus,
bladder) and with chronic respiratory disease and cardiovascular disease.
Figure 5
Smoking and Lung cancer Cohort Studies in Males

Mortality Ratio*
25
-1

Less than 10 10 to 19 , 20 or more

welghied mean of 4 etudles

 The Great Debate

Smoking, S, causes lung cancer, C (S → C ) versus spurious correlation due to

backdoor path:
Z

S C
Criticisms from Joseph Berkson, Jerzy Neyman and Ronald Fisher: (Hill, Millar
and Connelly 2003)
1 Correlation b/w smoking and lung cancer is spurious due to biased selection of
subjects (e.g., conditioning on collider problem)
2 Functional form complaints about using “risk ratios” and “odds ratios”
3 Confounder, Z , creates backdoor path between smoking and cancer
4 Implausible magnitudes
5 No experimental evidence to incriminate smoking as a cause of lung cancer
Fisher’s confounding theory
Fisher, equally famous as a geneticist, argued from logic, statistics and genetic
evidence for a hypothetical confounding genome, Z , and therefore smokers and
non-smokers were not exchangeable (violation of independence assumption)
Other studies showed that cigarette smokers and non-smokers were different on
observables – more extraverted than non-smokers and pipe smokers, differed in
age, differed in income, differed in education, etc.
 Uh, I thought you told me Fisher was smart?

Fisher’s arguments were actually based on sound science.

It may be too easy for us to criticize Fisher for his stance
because we look back to a different time when the
smoking/lung cancer link was not universally accepted, and
evidence for the causal link was shallow:
“the [the epidemiologists] turned out to be right,
but only because bad logic does not necessarily lead
to wrong conclusions.” Robert Hooke’s (1983), How
to Tell the Liars from the Statisticians.
But FWIW, Fisher was a chain smoking pipe smoker, he died
of lung cancer, and he was a paid expert witness for the
tobacco industry.
 Motivation: Smoking and Mortality

Table: Death rates per 1,000 person-years (Cochran 1968)

Smoking group Canada U.K. U.S.

Non-smokers 20.2 11.3 13.5
Cigarettes 20.5 14.1 13.5
Cigars/pipes 35.5 20.7 17.4

Are cigars dangerous?

Non-smokers and smokers differ in mortality and age

Table: Mean ages, years (Cochran 1968)

Smoking group Canada U.K. U.S.

Non-smokers 54.9 49.1 57.0
Cigarettes 50.5 49.8 53.2
Cigars/pipes 65.9 55.7 59.7

Older people die at a higher rate, and for reasons other than
just smoking cigars
Maybe cigar smokers higher observed death rates is because
they’re older on average
 Subclassification

One way to think about the problem is that the covariates are
not balanced – their mean values differ for treatment and
control group. So let’s try to balance them.
Subclassification (also called stratification): Compare
mortality rates across the different smoking groups within age
groups so as to neutralize covariate imbalances in the
observed sample
 Subclassification

Divide the smoking group samples into age groups

For each of the smoking group samples, calculate the
mortality rates for the age group
Construct probability weights for each age group as the
proportion of the sample with a given age
Compute the weighted averages of the age groups mortality
rates for each smoking group using the probability weights
 Subclassification: example

Death rates Number of

Pipe-smokers Pipe-smokers Non-smokers
Age 20-50 15 11 29
Age 50-70 35 13 9
Age +70 50 16 2
Total 40 40

Question: What is the average death rate for pipe smokers?

 Subclassification: example

Death rates Number of

Pipe-smokers Pipe-smokers Non-smokers
Age 20-50 15 11 29
Age 50-70 35 13 9
Age +70 50 16 2
Total 40 40

Question: What is the average death rate for pipe smokers?

     
11 13 16
15 · + 35 · + 50 · = 35.5
40 40 40
 Subclassification: example

Death rates Number of

Pipe-smokers Pipe-smokers Non-smokers
Age 20-50 15 11 29
Age 50-70 35 13 9
Age +70 50 16 2
Total 40 40

Question: What would the average mortality rate be for pipe smokers
if they had the same age distribution as the non-smokers?
 Subclassification: example

Death rates Number of

Pipe-smokers Pipe-smokers Non-smokers
Age 20-50 15 11 29
Age 50-70 35 13 9
Age +70 50 16 2
Total 40 40

Question: What would the average mortality rate be for pipe smokers
if they had the same age distribution as the non-smokers?
     
29 9 2
15 · + 35 · + 50 · = 21.2
40 40 40
Table: Adjusted death rates using 3 age groups (Cochran 1968)

Smoking group Canada U.K. U.S.

Non-smokers 20.2 11.3 13.5
Cigarettes 28.3 12.8 17.7
Cigars/pipes 21.2 12.0 14.2
 Covariates

Definition: Predetermined Covariates

Variable X is predetermined with respect to the treatment D (also
called “pretreatment”) if for each individual i, Xi0 = Xi1 , i.e., the
value of Xi does not depend on the value of Di . Such
characteristics are called covariates.
Comment I: Does not imply X and D are independent
Comment II: Predetermined variables are often time invariant (e.g.,
sex, race), but time invariance is not a necessary condition
 Outcomes

Definition: Outcomes
Those variables, Y , that are (possibly) not predetermined are
called outcomes (for some individual i, Yi0 6= Yi1 )

Comment: Recall the “collider bias”; in general, one shouldn’t

condition on outcomes because it may induce bias
 Adjustment for Observables

Subclassification
Matching
Propensity score methods
Regression
 Identification under independence

Recall that randomization implies

(Y 0 , Y 1 ) ⊥
⊥D

and therefore:

E [Y |D = 1] − E [Y |D = 0] = E [Y 1 |D = 1] − E [Y 0 |D = 0]
| {z }
by the switching equation
1 0
= E [Y ] − E [Y ]
| {z }
by independence
1 0
= E [Y − Y ]
| {z }
ATE

As well as that ATT = ATE :

E [Y 1 − Y 0 ] = E [Y 1 − Y 0 |D = 1]
 Identification under conditional independence

Identification assumptions:
1 (Y 1 , Y 0 ) ⊥
⊥ D|X (conditional independence)
2 0 < Pr (D = 1|X ) < 1 with probability one (common support)
Identification result:
Given assumption 1:

E [Y 1 − Y 0 |X ] = E [Y 1 − Y 0 |X , D = 1]
= E [Y |X , D = 1] − E [Y |X , D = 0]

Given assumption 2:

δATE = E [Y 1 − Y 0 ]
Z
= E [Y 1 − Y 0 |X , D = 1]dPr (X )
Z
= (E [Y |X , D = 1] − E [Y |X , D = 0]) dPr (X )
 Identification under conditional independence

Identification assumptions:
1 (Y 1 , Y 0 ) ⊥
⊥ D|X (conditional independence)
2 0 < Pr (D = 1|X ) < 1 with probability one (common support)
Identification result:
Similarly

δATT = E [Y 1 − Y 0 |D = 1]
Z
= (E [Y |X , D = 1] − E [Y |X , D = 0]) dPr (X |D = 1)

To identify δATT the conditional independence and common

support assumptions can be relaxed to:
1 Y0 ⊥⊥ D|X
2 Pr (D = 1|X ) < 1 (with Pr (D = 1) > 0)
 Subclassification estimator

The identification result is:

Z
δATE = (E [Y |X , D = 1] − E [Y |X , D = 0]) dPr (X )
Z
δATT = (E [Y |X , D = 1] − E [Y |X , D = 0]) dPr (X |D = 1)

Assume X takes on K different cells {X 1 , . . . , X k , . . . , X K }.

Then the analogy principle suggests the following estimators:
K  k
X 1,k 0,k N
δATE =
b (Y − Y ) ·
N
k=1
K  k
X 1,k 0,k NT
δATT =
b (Y − Y ) ·
NT
k=1

where Nk is the number of obs. and NTK is the number of

1,k
treatment observations in cell k; Y is the mean outcome
0,k
for the treated in cell k; Y is the mean outcome for the
control in cell k
 Subclassification by Age (K = 2)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old 28 24 4 3 10
Young 22 16 6 7 10
Total 10 20

K  k
1,k 0,k N
Question: What is δ[
X
ATE = (Y − Y ) · ?
N
k=1
 Subclassification by Age (K = 2)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old 28 24 4 3 10
Young 22 16 6 7 10
Total 10 20

K  k
1,k 0,k N
Question: What is δ[
X
ATE = (Y − Y ) · ?
N
k=1
   
13 17
4· +6· = 5.13
30 30
 Subclassification by Age (K = 2)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old 28 24 4 3 10
Young 22 16 6 7 10
Total 10 20

K
NTk
 
1,k 0,k
Question: What is δ[
X
ATT = (Y −Y )· ?
NT
k=1
 Subclassification by Age (K = 2)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old 28 24 4 3 10
Young 22 16 6 7 10
Total 10 20

K
NTk
 
1,k 0,k
Question: What is δ[
X
ATT = (Y −Y )· ?
NT
k=1
   
3 7
4· +6· = 5.4
10 10
 Subclassification by Age and Gender (K = 4)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old Males 28 22 4 3 7
Old Females 24 3
Young Males 21 16 5 3 4
Young Females 23 17 6 4 6
Total 10 20

K  k
1,k 0,k N
Problem: What is δ[
X
ATE = (Y − Y ) · ?
N
k=1
 Subclassification by Age and Gender (K = 4)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old Males 28 22 4 3 7
Old Females 24 3
Young Males 21 16 5 3 4
Young Females 23 17 6 4 6
Total 10 20

K  k
1,k 0,k N
Problem: What is δ[
X
ATE = (Y − Y ) · ?
N
k=1
Not identified!
 Subclassification by Age and Gender (K = 4)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old Males 28 22 4 3 7
Old Females 24 3
Young Males 21 16 5 3 4
Young Females 23 17 6 4 6
Total 10 20

K  k
1,k 0,k NT
Question: What is δ[
X
ATT = (Y − Y ) · ?
NT
k=1
 Subclassification by Age and Gender (K = 4)

Death Rate Number of

Xk Smokers Non-smokers Diff. Smokers Non-smokers
Old Males 28 22 4 3 7
Old Females 24 3
Young Males 21 16 5 3 4
Young Females 23 17 6 4 6
Total 10 20

K  k
1,k 0,k NT
Question: What is δ[
X
(Y − Y ) ·
ATT = ?
NT
k=1
     
3 3 4
4· +5· +6· = 5.1
10 10 10
 Curse of Dimensionality

Subclassification may become less feasible in finite samples as

the number of covariates grows (e.g., K = 4 was too many for
this sample)
Assume we have k covariates and we divide each into 3 coarse
categories (e.g., age: young, middle age, old; income: low,
medium, high, etc.)
The number of sub classification cells (or “strata”) is 3k . For
k = 10, then it’s 310 = 59, 049
 Curse of Dimensionality

If sparseness occurs, it means many cells may contain either

only treatment units or only control units but not both. If so,
we cannot use sub classification.
Subclassification is also a problem if the cells are “too
coarse”. We can always use “finer” classifications, but finer
cells worsens the dimensional problem, so we don’t gain much
from that. ex: using 10 variables and 5 categories for each, we
get 510 = 9, 765, 625.
 Matching

Alternatively, we could estimate δATT by imputing the missing

potential outcome of each treatment unit using the observed
outcome from that outcome’s “closest” control unit
1 X
δATT = (Yi − Yj(i) )
NT
Di =1

where Yj(i) is the observed outcome of a control unit such

that Xj(i) is the closest value to Xi among all of the control
observations
 Matching

We could also use the average observed outcome over M

closest matches:
M
" #!
1 X 1 X
δATT = Yi − Yjm (1)
NT M
Di =1 m=1

Works well when we can find good matches for each

treatment group unit, so M is usually defined to be small (i.e.,
M = 1 or M = 2)
 Matching

We can also use matching to estimate δATE . In that case, we

match in both directions:
1 If observation i is treated, we impute Yi0 using the control
matches, {Yj1 (i) , . . . , YjM (i) }
2 If observation i is control, we impute Yi1 using the treatment
matches, {Yj1 (i) , . . . , YjM (i) }
The estimator is:
N M
" !#
1 X 1 X
δATE =
b (2Di − 1) Yi − Yjm (i)
N M
i=1 m=1
 Matching example with single covariate

Potential Outcome
unit under Treatment under Control
i Yi1 Yi0 DI Xi
1 6 ? 1 3
2 1 ? 1 1
3 0 ? 1 10
4 0 0 2
5 9 0 3
6 1 0 -2
7 1 0 -4

1 X
Question: What is δ[
ATT = (Yi − Yj(i) )?
NT
Di =1
 Matching example with single covariate

Potential Outcome
unit under Treatment under Control
i Yi1 Yi0 DI Xi
1 6 ? 1 3
2 1 ? 1 1
3 0 ? 1 10
4 0 0 2
5 9 0 3
6 1 0 -2
7 1 0 -4

1 X
Question: What is δ[
ATT = (Yi − Yj(i) )?
NT
Di =1
Match and plug in!
 Matching example with single covariate

Potential Outcome
unit under Treatment under Control
i Yi1 Yi0 DI Xi
1 6 9 1 3
2 1 0 1 1
3 0 9 1 10
4 0 0 2
5 9 0 3
6 1 0 -2
7 1 0 -4

1 X
Question: What is δ[
ATT = (Yi − Yj(i) )?
NT
Di =1
1 1 1
δbATT = · (6 − 9) + · (1 − 0) + · (0 − 9) = −3.7
3 3 3
A Training Example
Trainees Non-Trainees
unit age earnings unit age earnings
1 28 17700 1 43 20900
2 34 10200 2 50 31000
3 29 14400 3 30 21000
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900
2 34 10200 2 50 31000
3 29 14400 3 30 21000
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900
2 34 10200 2 50 31000
3 29 14400 3 30 21000
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900
2 34 10200 2 50 31000
3 29 14400 3 30 21000
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000
3 29 14400 3 30 21000
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300 15 25 23300
16 43 10700 16 24 9700
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300 15 25 23300
16 43 10700 16 24 9700 1 43 20900
17 28 11500 17 29 6200
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300 15 25 23300
16 43 10700 16 24 9700 1 43 20900
17 28 11500 17 29 6200 8 28 8800
18 27 10700 18 35 30200
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300 15 25 23300
16 43 10700 16 24 9700 1 43 20900
17 28 11500 17 29 6200 8 28 8800
18 27 10700 18 35 30200 4 27 9300
19 28 16300 19 32 17800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300 15 25 23300
16 43 10700 16 24 9700 1 43 20900
17 28 11500 17 29 6200 8 28 8800
18 27 10700 18 35 30200 4 27 9300
19 28 16300 19 32 17800 8 28 8800
Average: 28.5 16426 20 23 9500 Average:
21 32 25900
Average: 33 20724
A Training Example
Trainees Non-Trainees Matched Sample
unit age earnings unit age earnings unit age earnings
1 28 17700 1 43 20900 8 28 8800
2 34 10200 2 50 31000 14 34 24200
3 29 14400 3 30 21000 17 29 6200
4 25 20800 4 27 9300 15 25 23300
5 29 6100 5 54 41100 17 29 6200
6 23 28600 6 48 29800 20 23 9500
7 33 21900 7 39 42000 10 33 15500
8 27 28800 8 28 8800 4 27 9300
9 31 20300 9 24 25500 12 31 26600
10 26 28100 10 33 15500 11,13 26 8450
11 25 9400 11 26 400 15 25 23300
12 27 14300 12 31 26600 4 27 9300
13 29 12500 13 26 16500 17 29 6200
14 24 19700 14 34 24200 9,16 24 17700
15 25 10100 15 25 23300 15 25 23300
16 43 10700 16 24 9700 1 43 20900
17 28 11500 17 29 6200 8 28 8800
18 27 10700 18 35 30200 4 27 9300
19 28 16300 19 32 17800 8 28 8800
Average: 28.5 16426 20 23 9500 Average: 28.5 13982
21 32 25900
Average: 33 20724
Age Distribution: Before Matching

A: Trainees
3
2
1
frequency
0

B: Non−Trainees
3
2
1
0

20 30 40 50 60
age
Graphs by group
Age Distribution: After Matching

A: Trainees
3
2
1
frequency
0

B: Non−Trainees
3
2
1
0

20 30 40 50 60
age
Graphs by group
Training E↵ect Estimates

Di↵erence in average earnings between trainees and non-trainees

Before matching

16426 20724 = 4298

After matching:

16426 13982 = 2444

 Alternative distance metric: Euclidean distance
 
X1
X2 
When the vector of matching covariates, X =  .  has more
 
 .. 
Xk
than one dimension (k > 1) we will need a new definition of
distance to measure “closeness”.
Definition: Euclidean distance
q
||Xi − Xj || = (Xi − Xj )0 (Xi − Xj )
v
u k
uX
= t (Xni − Xnj )2
n=1

Comment: The Euclidean distance is not invariant to changes in

the scale of the X ’s. For this reason, alternative distance metrics
that are invariant to changes in scale are used
 Normalized Euclidean distance

Definition: Normalized Euclidean distance

A commonly used distance is the normalized Euclidean distance:
q
||Xi − Xj ||= (Xi − Xj )0 Vb −1 (Xi − Xj )

where
b12 0 . . .
 
σ 0
0 σ b22 . . . 0
Vb −1 = . .. . . .. 
 
. . . . .
0 0 ... bk2
σ

Notice that the normalized Euclidean distance is equal to:

v
u k
uX (Xni − Xnj )
||Xi − Xj ||= t
bn2
σ
n=1

bn2 , and
Thus, if there are changes in the scale of Xni , these changes also affect σ
the normalized Euclidean distance does not change
 Mahalanobis distance

Definition: Mahalanobis distance

The Mahalanobis distance is the scale-invariant distance metric:
q
b −1 (Xi − Xj )
||Xi − Xj ||= (Xi − Xj )0 Σ X

where Σ
b X is the sample variance-covariance matrix of X .

Comments: Related, you can create arbitrary distances like:

v
u k
uX
||Xi − Xj ||= t ωn · (Xni − Xnj )2
n=1

(with all ωn ≥ 0) so that we assign large ωn ’s to those covariates

that we want to match particularly well.
 Matching and the Curse of Dimensionality

Dimensionality creates headaches for us in matching.

Bad news: Matching discrepancies ||Xi − Xj(i) || tend to
increase with k, the dimension of X
Good news: Matching discrepancies converge to zero . . .
Bad news: . . . but they converge very slow if k is large
Good news?: Mathematically, it can be shown that
1
||Xi − Xj(i) || converges to zero at the same rate as 1
Nk
Bad news: It’s hard to find good matches when X has a large
dimension: you need many observations if k is big.
 Deriving the matching bias
Derive the matching bias by first writing out the sample ATT
estimate:
1 X
δbATT = (Yi − Yj(i) ),
NT
Di =1

where each i and j(i) units are matched, Xi ≈ Xj(i) and Dj(i) = 0.
Define potential outcomes and switching eq.
µ0 (x) = E [Y |X = x, D = 0] = E [Y 0 |X = x],
µ1 (x) = E [Y |X = x, D = 1] = E [Y 1 |X = x],
Yi = µDi (Xi ) + εi
Substitute and distribute terms
1 X 1
(µ (Xi ) + εi ) − (µ0 (Xj(i) ) + εj(i) )

δbATT =
NT
Di =1
1 X 1 1 X
= (µ (Xi ) − µ0 (Xj(i) )) + (εi − εj(i) )
NT NT
Di =1 Di =1
 Deriving the matching bias

Difference between sample estimate and population parameter is:

1 X 1
µ (Xi ) − µ0 (Xj(i) ) − δATT


δbATT − δATT =
NT
Di =1
1 X
+ (εi − εj(i) )
NT
Di =1

Algebraic manipulation and simplification:

1 X 1
µ (Xi ) − µ0 (Xi ) − δATT


δbATT − δATT =
NT
Di =1
1 X
+ (εi − εj(i) )
NT
Di =1
1 X 0
µ (Xi ) − µ0 (Xj(i) ) .


+
NT
Di =1
 Deriving the matching bias
Apply the Central Limit Theorem and the difference,
r
1 b
(δATT − δATT ),
N
converges to a Normal distribution with zero mean. But, however,
r r
1 b 1 0
E[ (δATT − δATT )] = E [ (µ (Xi ) − µ0 (Xj(i) ))|D = 1].
N N
Now consider the implications if k is large:
The difference between Xi and Xj(i) converges to zero very
slowly
The difference µ0 (Xi ) − µ0 (Xj(i) ) converges to zero very slowly
q
E [ N1 (µ0 (Xi ) − µ0 (Xj(i) ))|D = 1] may not converge to zero
andqan be very large!
E [ 1 (δbATT − δATT )] may not converge to zero because the
N
bias of the matching discrepancy is dominating the matching
estimator!
Bias is often an issue when we match in many dimensions
 Solutions to matching bias problem

The bias of the matching estimator is caused by large matching

discrepancies ||Xi − Xj(i) ||. The curse of dimensionality virtually
guarantees this. However:
1 But the matching discrepancies are observed. We can always
check in the data how well we’re matching the covariates.
2 For δbATT we can always make the matching discrepancies
small by using a large reservoir of untreated units to select the
matches (that is, by making NC large).
3 If the matching discrepancies are large, so we are worried
about potential biases, we can apply bias correction techniques
4 Partial solution: propensity score methods (coming soon. . . )
 Matching with bias correction

Each treated observation contributes

µ0 (Xi ) − µ0 (Xj(i) )

to the bias.
Bias-corrected (BC) matching:

BC 1 Xh c0 (X ))
c0 (Xi ) − µ
i
δbATT = (Yi − Yj(i) ) − (µ j(i)
NT
Di =1

c0 (X ) is an estimate of E [Y |X = x, D = 0]. For

where µ
example using OLS.
Under some conditions, the bias correction eliminates the bias
of the matching estimator without affecting the variance.
 Bias adjustment in matched data

Potential Outcome
unit under Treatment under Control
i Yi1 Yi0 Di Xi
1 10 8 1 3
2 4 1 1 1
3 10 9 1 10
4 8 0 4
5 1 0 0
6 9 0 8

10 − 8 4 − 1 10 − 9
δbATT = + + =2
3 3 3
 Bias adjustment in matched data

Potential Outcome
unit under Treatment under Control
i Yi1 Yi0 Di Xi
1 10 8 1 3
2 4 1 1 1
3 10 9 1 10
4 8 0 4
5 1 0 0
6 9 0 8

10 − 8 4 − 1 10 − 9
δbATT = + + =2
3 3 3
c0 (X ) = βb0 + βb1 X = 2 + X
For the bias correction, estimate µ
 Bias adjustment in matched data

Potential Outcome
unit under Treatment under Control
i Yi1 Yi0 Di Xi
1 10 8 1 3
2 4 1 1 1
3 10 9 1 10
4 8 0 4
5 1 0 0
6 9 0 8

10 − 8 4 − 1 10 − 9
δbATT = + + =2
3 3 3
For the bias correction, estimate µ c0 (X ) = βb0 + βb1 X = 2 + X

(10 − 8) − (µc0 (3) − µc0 (4)) (4 − 1) − (µ c0 (1) − µ

c0 (0))
δbATT = +
3 3
c0 (10) − µ
(10 − 9) − (µ c0 (8))
+ = 1.33
3
 Matching bias: Implications for practice

Bias arises because of the effect of large matching discrepancies on

µ0 (Xi ) − µ0 (Xj(i) ). To minimize matching discrepancies:
1 Use a small M (e.g., M = 1). Larger values of M produce
large matching discrepancies.
2 Use matching with replacement. Because matching with
replacement can use untreated units as a match more than
once, matching with replacement produces smaller matching
discrepancies than matching without replacement.
3 Try to match covariates with a large effect on µ0 (·)
particularly well.
 Large sample distribution for matching estimators

Matching estimators have a Normal distribution in large

samples (provided the bias is small):
p d 2
NT (δbATT − δATT ) −
→ N(0, σATT )

For matching without replacement, the “usual” variance

estimator:
M
!2
2 1 X 1 X
bATT =
σ Yi − Yjm (i) − δbATT ,
NT M
Di =1 m=1

is valid.
 Large sample distribution for matching estimators

For matching with replacement:

M
!2
2 1 X 1 X
σ
bATT = Yi − Yjm (i) − δbATT
NT M
Di =1 m=1
1 X Ki (Ki − 1)
 
+ var
c (ε|Xi , Di = 0)
NT M2
Di =0

where Ki is the number of times observation i is used as a

match.
c (Yi |Xi , Di = 0) can be estimated also by matching. For
var
example, take two observations with Di = Dj = 0 and
Xi ≈ Xj , then
(Yi − Yj )2
c (Yi |Xi , Di = 0) =
var
2
c (εi |Xi , Di = 0))
is an unbiased estimator of var
The bootstrap doesn’t work!
 Propensity score

Overview
What do we use a propensity score for?
How do we construct the propensity score?
How do we implement propensity score estimation in STATA?
Discuss several articles using propensity score matching
 Joke (sort of. . . )

Two heart surgeons walk into a bar.

Jack: “I just finished my 100th heart surgery!”
Jill: “I finished my 100th heart surgery last week. So I have
more experience than you, and am therefore probably a better
surgeon. How many of your patients died within 3 months of
surgery? I’ve only had 10 die.”
Jack: “Five. Which means I’m probably the better surgery.”
Jill: “Or maybe my patients are older and have a higher risk of
mortality than your patients.”
I didn’t say it was a good joke.
They are debating about who is the better surgeon – the
treatment of interest. But if the patients’ characteristics differ
between Jack and Jill, and those characteristics are correlated
with mortality, then it confounds the inference
To really judge who is the better surgeon, we want them to
each see the same groups of patients of equivalent
characteristics
 Purpose of propensity scores

When treatment is non-random, you can use propensity score

matching to compare a treatment and control group who are
equivalent on observable characteristics
Propensity score matching also provides a way to summarize
covariate information about treatment selection into a single
scalar
Can be used to adjust for differences via study design, or
matching, or during estimation of the treatment effect (e.g.,
subclassification or regression)
 Some caveats

This is only relevant for selection on observables

If you cannot write down a DAG such that conditioning on
some covariate(s) meets the backdoor criterion, then
propensity score matching is not the appropriate methodology
You need to exhaustively identify the variables, X , that will
block all back door paths between the treatment, D, and the
outcome, Y using your knowledge of theory and the
institutional details
 Motivation

If the treatment is randomly assigned, then we use the simple

difference in outcomes (SDO) estimator

1 X 1 X
(Y |D = 1) − (Y |D = 0)
NT NC
This is called “ignobility” when we have randomized
treatment assignment
But if what if ignobility is violated?
Propensity score matching has a weaker assumption called
conditional independence which we will discuss later
 OLS

In principle you could estimate the treatment effect using

regression and control for X

Y = δD + βX + ε

where X is a matrix of covariates that you think affect the

probability of receiving a scholarship
OLS consistently estimates the conditional mean, but if
probability of treatment is nonlinear, this conditional mean
may be less informative
Usually, we won’t know how selection depended on X – only
that it did (e.g., DAG and the backdoor criterion)
 Propensity score

The idea with matching on X ’s was to compare units who

were ”close” to one another based on some distance to the
nearest X neighbor. But we had some issue with matching
discrepancies and the the curse of dimensionality.
The idea with propensity score matching is to compare units
who, based solely on their observables, had very similar
probabilities of being placed into treatment
If conditional on X , two units have a similar probability of
treatment, then we say they have similar propensity scores
If two units have equivalent propensity scores, then differences
between their outcomes is attributable to the treatment
If we compare a unit in the treatment group to a control
group unit with two similar propensity scores, then conditional
on the propensity score, all remaining variation between these
two is randomness—if selection on observables is a correct
assumption.
 First: How do we practically do this?

Estimation using propensity score matching is a two-step

procedure
1 First step: estimate the propensity score
2 Second step: calculate the average causal effect of interest by
averaging differences in outcomes over units with similar
propensity scores
Estimating the propensity score
Estimate the following equation with binary treatment D on
the left-hand-side and observables X related to selection
determination (e.g., backdoor criterion) on the right-hand-side
using probit or logit model
Prob(D = 1|X ) = γX + ω
Second using the estimated coefficients, calculate the
predicted probability of treatment
ρb = γ
bX
The propensity score is just the predicted conditional
probability of treatment, or the fitted value for each unit
 Definition

Definition of Propensity score

Propensity score is defined as the selection probability conditional
on the confounding variables: p(X ) = Pr (D = 1|X )

Identification Assumptions:
1 (Y 0 , Y 1 ) ⊥
⊥ D|X (conditional independence)
2 0 < Pr (D = 1|X ) < 1 (common support)
Identifying assumption I: Conditional independence (Rosenbaum and Rubin 1983)
(Yi0 , Yi1 ) ⊥
⊥ D|Xi . There exists a set X of observable covariates such that after
controlling for these covariates, treatment assignment is independent of potential
outcomes.

Interpretation: After controlling for X , the assignment of units to the

treatment is ‘as good as random’.
Why?: Conditional independence allows us the ability to construct an unbiased
counterfactual for the treatment group using the non-experimental control
group units.
Also called ‘ignorable treatment assignment’ or ‘unconfoundedness’ (statistics),
backdoor criterion (Pearl), ‘selection on observables’, or ‘exogeneity’
(economics). Exogeneity in traditional econometric pedagogy

Yi0 = α + β 0 Xi + ε, and
Yi1 = Yi0 + δ

then we can write:

Yi = α + δDi + β 0 Xi + εi
and conditional independence ⇐⇒ εi ⊥ ⊥ Di |Xi (exogeneity)
Conditional independence is not testable
Identifying assumption II: Common support
For each value of X , there is a positive probability of being both
treated and untreated

0 < Pr (Di = 1|Xi ) < 1

Interpretation: Implies that the probability of receiving

treatment for every value of the vector X is strictly within the
unit interval: as is the probability of not receiving treatment.
Why?: Common support ensures there is sufficient overlap in
the characteristics of treated and untreated units to find
adequate matches
Common support is testable
 Identification given strong ignobility

When both assumptions are satisfied, the treatment

assignment is said to be strongly ignorable in the
terminology of Rosenbaum and Rubin (1983).
By definition

δi (Xi ) = E [Yi1 − Yi0 |Xi = x]

= E [Yi1 |Xi = x] − E [Yi0 |Xi = x]

Unconfoundedness allow us to substitute

E [Yi1 |Di = 1, Xi = x] = E [Yi |Di = 1, Xi = x]

and similar for other term.

Common support allows us to estimate both terms
Then, δ = E [δ(Xi )]
Propensity score theorem (Rosenbaum and Rubin 1983)
If (Y 1 , Y 0 ) ⊥
⊥ D|X (conditional independence), then
(Y 1 , Y 0 ) ⊥
⊥ D|p(X ) where p(X ) = Pr (D = 1|X ), the propensity
score
Sufficient conditions: Conditioning on the propensity score is
enough to have independence between the treatment indicator, D,
and the potential outcomes, (Y 1 , Y 0 )
Dimension reduction: Extremely valuable theorem. Here’s why
Stratifying on X tends to run into sparseness-related problems
(i.e., empty cells) in finite samples for even moderate number
of covariates
But the propensity score, p(X ), is just a scalar, and therefore
stratifying across that probability is going to be a lot easier.
Proof: Straightforward application of the law of iterated
expectations (LIE) with nested conditioning from MHE (pp. 80-81)
 Proof

If we can show that the probability an individual receives

treatment conditional on potential outcomes and the
propensity score is not a function of potential outcomes

Pr (D = 1|Y 0 , Y 1 , p(X )) 6= f (Y 0 , Y 1 )

then we will have proven that there is an independence

between Y 0 , Y 1 and D conditional on X
Before diving into the proof, first recognize that

Pr (D = 1|Y 0 , Y 1 , p(X )) = E [D|Y 0 , Y 1 , p(X )]

because

E [D|Y 0 , Y 1 , p(X )] = 1 · Pr (D = 1|Y 0 , Y 1 , p(X ))

+0 · Pr (D = 0|Y 0 , Y 1 , p(X ))

and the second term cancels out.

Proof.
Assume that (Y 1 , Y 0 ) ⊥
⊥ D|X . Then:

Pr (D = 1|Y 1 , Y 0 , p(X )) = E [D|Y 1 , Y 0 , p(X )]

| {z }
See previous slide

= E [E [D|Y 1 , Y 0 , p(X ), X ]|Y 1 , Y 0 , p(X )]

| {z }
by LIE

= E [E [D|Y 1 , Y 0 , X ]|Y 1 , Y 0 , p(X )]

| {z }
Given X , we know p(X )

= E [E [D|X ]|Y 1 , Y 0 , p(X )]

| {z }
by conditional independence

= E [p(X )|Y 1 , Y 0 , p(X )]

| {z }
propensity score definition
= p(X )

Using a similar argument, we obtain:

Pr (D = 1|p(X )) = E [D|p(X )] = E [E [D|X ]|p(X )]

| {z } | {z }
Previous slide LIE
= E [p(X )|p(X )] = p(X )
| {z }
definition

and Pr (D = 1|Y 1 , Y 0 , p(X )) = Pr (D = 1|p(X )) by conditional independence

 Propensity score theorem

Like the omitted variable bias formula for regression, the

propensity score theorem says that you need only control for
covariates that affect the probability of treatment
But it also says something more
The only covariate you really need to control for is the
probability of treatment itself
Direct propensity score matching works in the same way as
covariate matching (e.g., nearest neighbor matching) except
that we match on the score instead of the covariates directly
Corollary
If (Y 1 , Y 0 ) ⊥
⊥ D|X , we can estimate average treatment effects:

E [Y 1 − Y 0 |p(X )] = E [Y |D = 1, p(X )] − E [Y |D = 0, p(X )]

Three-step procedure:
1 Estimate the conditional probability of treatment, or
propensity score, p(X ) = Pr (D = 1|X ), using any standard
probability model (logit or probit)
2 Do matching, sub classification (stratification), inverse
probability weighting or some other algorithmic procedures to
estimate the average causal effect conditional on the
estimated propensity score
3 Compute standard errors
 Balancing property

Because the propensity score is a function of X, we know:

Pr (D = 1|X , p(X )) = Pr (D = 1|X )
= p(X )
∴ conditional on p(X ), the probability that D = 1 does not
depend on X .
D and X are independent conditional on p(X ):
D⊥
⊥ X |p(X )
So we obtain the balancing property of the propensity score:
Pr (X |D = 1, p(X )) = Pr (X |D = 0, p(X ))
conditional on the property score, the distribution of the
covariates is the same for treatment and control group units
We can use this to check if our estimated propensity score
actually produces balance:
Pr (X |D = 1, pb(X )) = Pr (X |D = 0, pb(X ))
 Balancing Property

D Y

p(X )

X
 Inverse Probability Weighting

Proposition
If Y 1 , Y 0 ⊥
⊥ D|X , then

D − p(X )
 
δATE = E [Y 1 − Y 0 ] = E Y ·
p(X ) · (1 − p(X ))
D − p(X )
 
1
δATT = E [Y 1 − Y 0 |D = 1] = ·E Y ·
Pr (D = 1) 1 − p(X )

Proof.

D − p(X ) 
   
Y 
E Y = E X , D = 1 p(X )
p(X )(1 − p(X )) p(X )
X

−Y 
 
+E X , D = 0 (1 − p(X ))
1 − p(X )
= E [Y |X , D = 1] − E [Y |X , D = 0]

and the results follow from integrating over P(X ) and P(X |D = 1).
 Weighting on the propensity score

ATE and ATT parameters (previous slide):

D − p(X )
 
δATE = E [Y 1 − Y 0 ] = E Y ·
p(X ) · (1 − p(X ))
D − p(X )
 
1
δATT = E [Y 1 − Y 0 |D = 1] = ·E Y ·
Pr (D = 1) 1 − p(X )

The analogy principle suggests a two-step estimator:

1 Estimate the propensity score: pb(X )
2 Use estimated score to produce analog estimators. Let δbATE and δbATT be an
estimate of the ATE and ATT parameter:
N
1 X Di − pb(Xi )
δbATE = Yi ·
N pb(Xi ) · (1 − pb(Xi ))
i=1
N
1 X Di − pb(Xi )
δbATT = Yi ·
NT 1 − pb(Xi )
i=1
 Weighting on the propensity score

Sample analogs of the population parameters (previous slide):

N
1 X Di − pb(Xi )
δbATE = Yi ·
N pb(Xi ) · (1 − pb(Xi ))
i=1
N
1 X Di − pb(Xi )
δbATT = Yi ·
NT 1 − pb(Xi )
i=1

Standard errors:
We need to adjust the standard errors for first-step estimation
of p(X )
Parameteric first step: Newey and McFadden (1994)
Non-parametric first step: Newey (1994)
Or bootstrap the entire two-step procedure.
 Other algorithmic methods

1 Estimate propensity score

2 Choose algorithm
3 Check overlap/common support
4 Matching Quality/Effect estimation
5 Sensitivity Analysis
 Estimation

Once you have determined the propensity score with the

procedure above, there are several ways to use it to estimate
average treatment effects
1 Stratification. Run the second step within each group.
Calculate the weighted mean of the within-group estimates to
get the average treatment effect
2 Matching. Match each treatment observation to its nearest
neighbor(s) based on similar propensity scores.
 Stratification: Achieving Balance

Stratification requires imposing balance by grouping the data

and testing for differences in covariate means
1 Sort the data by propensity score and divide into groups of
observations with similar propensity scores (e.g., percentiles)
2 Within each group, test (using a t-test) whether the means of
the covariates (X ) are equal between treatment and control
3 If so, then stop. If not, it means the covariates aren’t balanced
within that group. Divide the group in half and repeat
4 If a particular covariate is unbalanced for multiple groups,
modify the initial logit or probit equation by including higher
order terms and/or interactions with that covariate and repeat
This is done to satisfy the balancing property.
Notice: we never even look at outcomes in this step
 Matching strategy and ATT estimation

The standard matching strategy is the following:

Pair each treatment unit i with one or more comparable
control group unit j, where comparability is in terms of
proximity to the estimated propensity score
Associate to the treatment unit’s outcome Yi a matched
outcome of Yi(j) given by the weighted outcomes of its
neighbors in the control group
X
Yi(j) = wij Yj
j∈C (i)

where
C (i) is the set of neighbors with W = 0 of the treatment unit i
and
P wij is the weight of control group units j with
j∈C (i) wij = 1
 Estimation (cont.)

The ATT equals

E [Yi1 |Di = 1] − E [Yi0 |Di = 1]

We estimate it as follows
1 X  
[=
ATT = Yi − Yi(j)
NT
i:Wi =1

where NT is the number of matched treatment units in the sample.

 Matching methods

An estimate of the propensity score is not enough to estimate

the ATT
In fact, the probability of observing two units with exactly the
same propensity score is in principle zero because p(x) is
continuous
Several matching methods have been proposed in the
literature, but the most widely used are:
Nearest-neighbor matching (with or without caliper)
Radius matching
Kernel matching
Stratification matching
Typically, one treatment unit i is matched to several control
units j, but sometimes one-to-one matching is used
 STATA

The STATA command psmatch2 will perform propensity

score matching
Rich suite of options: nearest neighbor (with or without
caliper, with or without replacement), k-nearest neighbors,
radius, kernel, local linear regression, and Mahalanobis
matching
It includes routes for common support graphing (psgraph)
and covariate imbalance testing (pstest)
But standard errors are incorrect. They use bootstrapping
methods or variance approximation.
Has an excellent help file
Type in ssc install psmatch2
 STATA

Another useful command in STATA is pscore

Various matching methods for estimating ATT
Standard errors are still incorrect: bootstrapping, and variance
approximation
Balancing tests based on stratification
Have to use net search pscore to install it.
 STATA

As of STATA 14, built-in suite of options called teffects

Fewer options. No sampling without replacement
Standard errors are correct
I’m less familiar with it
 Nearest Neighbor

Pretty similar to covariate matching. Formula is

 
NN 1 X X
ATT = Yi − wij Yj
NT
i:Wi =1 j∈C (i)M

NT is the number of Treatment units i

wij is equal to N1C if j is a control unit and zero otherwise; NC
is number of control units j
 NN Matching: Bias vs. Variance

How many nearest neighbors should we use?

Matching just one nearest neighbor minimizes bias at the cost
of larger variance
Matching using additional nearest neighbors increases the bias
but decreases the variance
Matching with or without replacement
with replacement keeps bias low at the cost of larger variance
without replacement keeps variance low but at the cost of
potential bias
 Caliper matching

Caliper matching is a variation on NN matching that tries to

build into the algorithm stops against bad matches
It does this by imposing a tolerable maximum distance (e.g.,
0.2 units in the propensity score away from a treatment unit
i’s propensity score)
If there doesn’t exist any control group unit j within that
“caliper”, then treatment unit i is discarded
It’s difficult to know what this caliper should be ex ante
 Radius matching

Each treatment unit i is matched only with the control group unit
j whose propensity score falls into a predefined neighborhood of
the propensity score of the treatment unit.
All the control units with ρbj falling within a radius r from ρbi
are matched to the treatment unit i
The smaller the radius, the better the quality of the matches,
but the higher the possibility some treatment units are not
matched because the neighborhood does not contain control
group units j
 Checking the common support assumption

We can summarize the propensity scores in the treatment and

control group and count how many units are off-support
For instance, let’s say that the treatment group propensity
scores range from [0.1, 0.85]. That means the min and max
scores are in that range.
We could check what the distribution is for the control group,
and drop all the units before 0.1 and above 0.85. In other
words, we’re throwing away observations that don’t satisfy the
common support assumption.
A histogram of propensity scores by treatment and control
group also highlights the overlap problem
 Overview
Part 1: The Core
Stratification
Part 2: Selection on observables
Matching
Part 3: Selection on unobservables
Propensity score matching
Part 4: Advanced material
Regression discontinuity designs
Conclusion
References

Examples of propensity score matching

Example: Job Trainings Program (NSW) (LaLonde 1986,

Dehejia and Wahba 1999; 2002)

Cunningham Causal Inference

 Description of NSW Job Trainings Program

The National Supported Work Demonstration (NSW), operated by

Manpower Demonstration Research Corp in the mid-1970s:
was a temporary employment program designed to help
disadvantaged workers lacking basic job skills move into the
labor market by giving them work experience and counseling
in a sheltered environment
was also unique in that it randomly assigned qualified
applicants to training positions:
Treatment group: received all the benefits of NSW program
Control group: left to fend for themselves
admitted AFDC females, ex-drug addicts, ex-criminal
offenders, and high school dropouts of both sexes
 NSW Program

Treatment group members were:

guaranteed a job for 9-18 months depending on the target
group and site
divided into crews of 3-5 participants who worked together and
met frequently with an NSW counselor to discuss grievances
and performance
paid for their work
Other details about the NSW program:
Wages: NSW offered the trainees lower wage rates than they
would’ve received on a regular job, but allowed their earnings
to increase for satisfactory performance and attendance
Post-treatment: after their term expired, they were forced to
find regular employment
Job types: varied within sites – gas station attendant, working
at a printer shop – and males and females were frequently
performing different kinds of work
 NSW Data

NSW data collection:

MDRC collected earnings and demographic information from
both treatment and control at baseline and every 9 months
thereafter
Conducted up to 4 post-baseline interviews
Different sample sizes from study to study can be confusing,
but has simple explanations
Estimation:
NSW was a randomized job trainings program; therefore
estimating the average treatment effect is straightforward:
1 X 1 X
Yi − Yi ≈ E [Y 1 − Y 0 ]
Nt Nc
Di =1 Di =0

in large samples assuming treatment selection is independent

of potential outcomes (randomization) – i.e., (Y 0 , Y 1 ) ⊥
⊥ D.
NSW worked: Treatment group participants’ real earnings
post-treatment (1978) was positive and economically
meaningful – ≈ $900 (LaLonde 1986) to $1,800 (Dehejia and
Wahba 2002) depending on the sample used
 LaLonde, Robert J. (1986). “Evaluating the Econometric
Evaluations of Training Programs with Experimental Data”.
American Economic Review.

LaLonde’s study: was not an evaluation of the NSW program,

but rather an evaluation of econometric models done by:
replacing the experimental NSW control group with
non-experimental control group drawn from two public use
datasets: Current Population Survey (CPS) and Panel Study
of Income Dynamics (PSID)
estimating the average effect using non-experimental workers
as controls for the NSW trainees
comparing his non-experimental estimates to the experimental
estimates of $900
LaLonde’s conclusion: available econometric approaches were
biased and inconsistent
His estimates were way off and usually the wrong sign
Conclusion was influential in policy circles and led to greater
push for more experimental evaluations
 Imbalanced covariates for experimental and non-experimental
samples

CPS NSW
All Controls Trainees
Nc = 15, 992 Nt = 297
covariate mean (s.d.) mean mean t-stat diff
Black 0.09 0.28 0.07 0.80 47.04 -0.73
Hispanic 0.07 0.26 0.07 0.94 1.47 -0.02
Age 33.07 11.04 33.2 24.63 13.37 8.6
Married 0.70 0.46 0.71 0.17 20.54 0.54
No degree 0.30 0.46 0.30 0.73 16.27 -0.43
Education 12.0 2.86 12.03 10.38 9.85 1.65
1975 Earnings 13.51 9.31 13.65 3.1 19.63 10.6
1975 Unemp 0.11 0.32 0.11 0.37 14.29 -0.26
 Dehija and Wahba (1999)

Dehejia and Wahba (DW) update LaLonde’s original study

using propensity score matching
1 Dehejia, Rajeev H. and Sadek Wahba (1999). “Causal Effects
in Nonexperimental Studies: Reevaluating the Evaluation of
Training Programs”. Journal of the American Statistical
Association, vol. 94(448): 1053-1062 (pdf)
Can propensity score matching improve over the estimators
that LaLonde examined?
 Proposition 2

X ⊥
⊥ D|p(X )

Conditional on the propensity score, the covariates are

independent of the treatment, suggesting that the distribution
of covariate values should be the same for both treatment and
control groups
 Trimming the data

“Trimming” throws away units from control which do not

overlap with the treatment in terms of estimated propensity
score
In STATA:
. su pscore if treat==1
. drop if pscore<r(mean) & treat==0
Overlap
Overlap
Results
Covariate balance
 Estimation in STATA

We will replicate some of DW (1999) using the experimental

treatment and non-experimental PSID sample
You will need to download the following zipped dataset from
Barbara Sianesi’s presentation given to the 2010 German
STATA Users Group in Berlin.
Note you will need to install psmatch2

ssc install psmatch2, replace

. use ./nsw_psid, clear
(NSW treated and PSID non-treated)

. qui probit treated age black hispanic married educ nodegree re75

. margins, dydx(_all)

Average marginal effects Number of obs = 2787

Model VCE : OIM

Expression : Pr(treated), predict()

dy/dx w.r.t. : age black hispanic married education nodegree re75

------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
age | -.0035844 .000462 -7.76 0.000 -.0044899 -.002679
black | .0766501 .0088228 8.69 0.000 .0593577 .0939426
hispanic | .0831734 .0157648 5.28 0.000 .0522751 .1140718
married | -.0850743 .0070274 -12.11 0.000 -.0988478 -.0713009
education | .0003458 .0023048 0.15 0.881 -.0041716 .0048633
nodegree | .0418875 .0108642 3.86 0.000 .0205942 .0631809
re75 | -6.89e-06 5.89e-07 -11.71 0.000 -8.04e-06 -5.74e-06
------------------------------------------------------------------------------

. // compute the propensity score

. predict double score
(option pr assumed; Pr(treated))
. // compare the densities of the estimated propensity score over groups
. density2 score, group(treated) saving(./psm2a, replace)
(file ./psm2a saved)

. graph export psm2a.pdf, replace

(file psm2a.pdf written in PDF format)

. psgraph, treated(treated) pscore(score) bin(50) saving(psm2b, replace)

(file psm2b.gph saved)

. graph export psm2b.pdf, replace

(file psm2b.pdf written in PDF format)
6
4
2
0

0 .2 .4 .6 .8 1
Pr(treated)

treated=1 treated=0
0 .2 .4 .6 .8 1
Propensity Score

Untreated Treated
. // compute nearest neighbor matching with caliper and replacement
. psmatch2 treated, pscore(score) outcome(re78) caliper(0.01)
There are observations with identical propensity score values.
The sort order of the data could affect your results.
Make sure that the sort order is random before calling psmatch2.
----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat
----------------------------+-----------------------------------------------------------
re78 Unmatched | 5976.35202 21553.9209 -15577.5689 913.328457 -17.06
ATT | 6067.8117 5758.47686 309.334834 1080.935 0.29
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

psmatch2: | psmatch2: Common

Treatment | support
assignment | Off suppo On suppor | Total
-----------+----------------------+----------
Untreated | 0 2,490 | 2,490
Treated | 26 271 | 297
-----------+----------------------+----------
Total | 26 2,761 | 2,787

. //evaluate common support

. summarize _support if treated

Variable | Obs Mean Std. Dev. Min Max

-------------+--------------------------------------------------------
_support | 297 .9124579 .2831048 0 1
. //evaluate quality of matching
. pstest2 age black hispanic married educ nodegree re75, sum graph

----------------------------------------------------------------------------
| Mean %reduct | t-test
Variable Sample | Treated Control %bias |bias| | t p>|t|
------------------------+----------------------------------+----------------
age Unmatched | 24.626 34.851 -116.6 | -16.48 0.000
Matched | 25.052 25.443 -4.5 96.2 | -0.61 0.545
| |
black Unmatched | .80135 .2506 132.1 | 20.86 0.000
Matched | .78967 .78967 0.0 100.0 | -0.00 1.000
| |
hispanic Unmatched | .09428 .03253 25.5 | 5.21 0.000
Matched | .09594 .08856 3.0 88.0 | 0.30 0.767
| |
married Unmatched | .16835 .86627 -194.9 | -33.02 0.000
Matched | .1845 .14022 12.4 93.7 | 1.40 0.163
| |
education Unmatched | 10.38 12.117 -68.6 | -9.51 0.000
Matched | 10.465 10.166 11.8 82.8 | 1.54 0.125
| |
nodegree Unmatched | .73064 .30522 94.0 | 15.10 0.000
Matched | .71587 .69373 4.9 94.8 | 0.56 0.573
| |
re75 Unmatched | 3066.1 19063 -156.6 | -20.12 0.000
Matched | 3197.4 3307.8 -1.1 99.3 | -0.28 0.778
| |
----------------------------------------------------------------------------
 black

nodegree

hispanic

education

age

re75

married

-200 -100 0 100 200

Unmatched Matched
 Kernel matching

Alternatively we can perform propensity score matching with a

kernel-based method.
Notice on the next slide that the estimate of the ATT switches
sign relative to that produced by the NN matching algorithm
. // compute kernel-based matching with normal kernel
. psmatch2 treated, pscore(score) outcome(re78) kernel k(normal) bw(0.01)
----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat
----------------------------+-----------------------------------------------------------
re78 Unmatched | 5976.35202 21553.9209 -15577.5689 913.328457 -17.06
ATT | 5976.35202 6882.18396 -905.831935 2151.26377 -0.42
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

| psmatch2:
psmatch2: | Common
Treatment | support
assignment | On support | Total
-----------+-----------+----------
Untreated | 2,490 | 2,490
Treated | 297 | 297
-----------+-----------+----------
Total | 2,787 | 2,787
. //evaluate quality of matching
. pstest2 age black hispanic married educ nodegree re75, sum graph

----------------------------------------------------------------------------
| Mean %reduct | t-test
Variable Sample | Treated Control %bias |bias| | t p>|t|
------------------------+----------------------------------+----------------
age Unmatched | 24.626 34.851 -116.6 | -16.48 0.000
Matched | 24.626 24.572 0.6 99.5 | 0.09 0.926
| |
black Unmatched | .80135 .2506 132.1 | 20.86 0.000
Matched | .80135 .81763 -3.9 97.0 | -0.50 0.614
| |
hispanic Unmatched | .09428 .03253 25.5 | 5.21 0.000
Matched | .09428 .08306 4.6 81.8 | 0.48 0.631
| |
married Unmatched | .16835 .86627 -194.9 | -33.02 0.000
Matched | .16835 .1439 6.8 96.5 | 0.82 0.413
| |
education Unmatched | 10.38 12.117 -68.6 | -9.51 0.000
Matched | 10.38 10.238 5.6 91.8 | 0.81 0.415
| |
nodegree Unmatched | .73064 .30522 94.0 | 15.10 0.000
Matched | .73064 .72101 2.1 97.7 | 0.26 0.793
| |
re75 Unmatched | 3066.1 19063 -156.6 | -20.12 0.000
Matched | 3066.1 3905.8 -8.2 94.8 | -1.99 0.047
| |
----------------------------------------------------------------------------
 black

nodegree

hispanic

education

age

re75

married

-200 -100 0 100 200

Unmatched Matched
 0 1 on the estimated propensity score. The covariate matching es-
The normalized difference provides a scale-free measure of the timators use the matrix Ane (the diagional matrix with inverse
difference in the location of the two distributions, and is useful sample variances on the diagonal) as the distance measure. Be-
for assessing the degree of difficulty in adjusting for differences cause we are focused on the average effect for the treated, the
in covariates. bias correction only requires an estimate of µ0 (Xi ). We esti-
Matchings vs. Propensity score
Panel A contains the results for pretreatment variables and mate this regression function using linear regression on all nine
Panel B for outcomes. Notice the large differences in back- pretreatment covariates in Table 1, panel A, but do not include
ground characteristics between the program participants and the any higher order terms or interactions, with only the control
PSID sample. This is what makes drawing causal inferences units that are used as a match [the units j such that Wj = 0 and

Table 2. Experimental and nonexperimental estimates for the NSW data

M=1 M=4 M = 16 M = 64 M = 2490

Est. (SE) Est. (SE) Est. (SE) Est. (SE) Est. (SE)
Panel A:
Experimental estimates
Covariate matching 1.22 (0.84) 1.99 (0.74) 1.75 (0.74) 2.20 (0.70) 1.79 (0.67)
Bias-adjusted cov matching 1.16 (0.84) 1.84 (0.74) 1.54 (0.75) 1.74 (0.71) 1.72 (0.68)
Pscore matching 1.43 (0.81) 1.95 (0.69) 1.85 (0.69) 1.85 (0.68) 1.79 (0.67)
Bias-adjusted pscore matching 1.22 (0.81) 1.89 (0.71) 1.78 (0.70) 1.67 (0.69) 1.72 (0.68)
Regression estimates
Mean difference 1.79 (0.67)
Linear 1.72 (0.68)
Quadratic 2.27 (0.80)
Weighting on pscore 1.79 (0.67)
Weighting and linear regression 1.69 (0.66)
Panel B:
Nonexperimental estimates
Simple matching 2.07 (1.13) 1.62 (0.91) 0.47 (0.85) −0.11 (0.75) −15.20 (0.61)
Bias-adjusted matching 2.42 (1.13) 2.51 (0.90) 2.48 (0.83) 2.26 (0.71) 0.84 (0.63)
Pscore matching 2.32 (1.21) 2.06 (1.01) 0.79 (1.25) −0.18 (0.92) −1.55 (0.80)
Bias-adjusted pscore matching 3.10 (1.21) 2.61 (1.03) 2.37 (1.28) 2.32 (0.94) 2.00 (0.84)
Regression estimates
Mean difference −15.20 (0.66)
Linear 0.84 (0.88)
Quadratic 3.26 (1.04)
Weighting on pscore 1.77 (0.67)
Weighting and linear regression 1.65 (0.66)
NOTE: The outcome is earnings in 1978 in thousands of dollars.
 Conclusions

Propensity scores are awesome to check the balance and

overlap of covariates – an under appreciated diagnostic, and
one that you won’t probably do if you only run regressions
There are extensions for more than two treatments (Cattaneo
2010)
The propensity score can make groups comparable but only
on the variables used to estimate the propensity score in the
first place. There is NO guarantee you are balancing on
unobserved covariates.
If you know that there are important unobservable variables,
you may need another tool.
Randomization ensure that both observable and unobservable
variables are balanced
 Consider this

One thing to ruminate over is that in invoking CIA, the researcher

is basically saying that there exists some unknown randomness
determining treatment once she conditions on X . But if there is
some randomization,
1 how does she know that exactly, and
2 why not use that randomization itself as the treatment, such
as one often does in a reduced form instrumental variables
approach (more to come on that)? This is basically what
Krueger (1999) does when he regresses test scores onto
original room assignment (rather than actual room
assignment).
HT to Rodney Andrews for making this point to me in a
conversation
 Conclusions

Think hard about this question: Do you really need a

propensity score analysis? Why do you believe that CIA holds?
Why not use the conditional randomization process as the
explanatory variable? If you can’t do that, how confident are
you in CIA?
 Overview
Part 1: The Core
Stratification
Part 2: Selection on observables
Matching
Part 3: Selection on unobservables
Propensity score matching
Part 4: Advanced material
Regression discontinuity designs
Conclusion
References

What is regression discontinuity design?

Recently there has been an increase in a particular type of research design known as
regression discontinuity design (RDD). Cook (2008) has a fascinating history of
thought on how and why.
Donald Campbell is the originator of regression discontinuity design. First
study is Thistlethwaite and Campbell (1960). Merit awards were given to
students whose test scores were over some cutoff point. They compared award
winners to non-winners just around the threshold to identify the causal effect of
merit awards on future academic outcomes.
Pictures are helpful for understanding the RDD research design. Tell me what
you think these are saying.

Cunningham Causal Inference

 RDD Visual Example
THE EFFECT OF ATTENDING THE FLAGSHIP STATE UNIVERSITY ON EARNINGS

FIGURE 1.—FRACTION ENROLLED AT THE FLAGSHIP STATE UNIVERSITY

.1 .2 .3 .4 .5 .6 .7 .8 .9 1
0

-300 -250 -200 -150 -100 -50 0 50 100 150 200 250 300 350

Local Average
 V. Results discontinuity in earnings at the admission cutoff. This
A. Earnings Discontinuities at the Admission Cutoff
RDD Visual isExample
shown for white men in figure 2, which shows a
regression of residual earnings on a cubic polynomial of
To the extent that there are economic returns to attend- adjusted SAT score. Table 1 shows the discontinuity
ing the flagship state university, one should observe a estimates that result from varying functional form

FIGURE 2.—NATURAL LOG OF ANNUAL EARNINGS FORWHITE MEN TEN TO FIFTEEN YEARS AFTER HIGH SCHOOL GRADUATION (FIT WITH A CUBIC
POLYNOMIAL OF ADJUSTED SAT SCORE)

Estimated Discontinuity = 0.095 (z = 3.01)

.2
(Residual) Natural Log of Earnings
-.3 -.2 -.1-.4 0 .1

-300 -250 -200 -150 -100 -50 0 50 100 150 200 250 300 350
SAT Points Above the Admission Cutoff

Predicted Earnings Local Average

 What is a regression discontinuity design?

RDD is based on a simple idea: the assignment of some

treatment to observational units is completely known.
We know that the probability of treatment “jumps” when an
applicant’s test scores meet university’s admission rule
If the students accepted and the students rejected are similar
near the cutoff point, then data can be analyzed as though it
were a conditionally randomized experiment .
RDD is appropriate in situations where treatment assignment
is sharply discontinuous in the values of a variable
It’s best understood with an example. We’ll look at Angrist
and Lavy (1999) which was one of a few papers around the
same time to bring this methodology into economics
Angrist, Joshua D. and Victor Lavy (1999). “Using Maimonides’
Rule to Estimate the Effect of Class Size on Scholastic
Achievement”. Quarterly Journal of Economics.
 Summary

Angrist and Lavy (1999) are interested in estimating the

causal effect of class size on educational outcomes
So like Krueger (1999), Angrist and Lavy (1999) are
interested in the effect of class size on pupil achievement.
 Maimonedes Rule and class size

One of the earliest references to class size is the Babylonian

Talmud, completed around 6th century, which discusses rules
for the determination of class size and pupil-teacher ratios in
bible study.
Maimonides was a 12th century Rabbinic scholar who
interpreted the Talmud’s discussion of class size as follows
“Twenty-five children may be put in charge of one teacher. If
the number in the class exceeds twenty-five but is not more
than forty, two teachers must be appointed.”
 Maimonedes Rule and Class size

“The importance of Maimonedes’ rule for our purposes is that

since 1969, it has been used to determine the division of
enrollment cohorts into classes in Israeli public schools.”
(Angrist and Lavy 1999)
 Identification

Class size in most places and most times is strongly associated

with unobservable differences in variables that also predict
educational performance
Poverty, affluence, enthusiasm/skepticism about the value of
education, special needs of students for remedial or advanced
instruction, obscure and barely intelligible obsessions of
bureaucracies
Each of these determines (i) class size and (ii) clouds the
effect of class size on academic performance because (iii) each
is also correlated independently with academic performance
However—if adherence to Maimonides’ rule is perfectly rigid,
then what would separate a school with a single class of size
40 from the same school with two classes whose average size
is 20.5 ( 41
2 = 20.5) is the enrollment of a single student
 Exogenous variation in class size

Maimonides’ rule has the largest impact on a school with

about 40 students in a grade cohort
With cohorts of size 40, 80 and 120 students, the steps down
in average class size required by Maimonides’ rule when an
additional student enrolls are from 40 to 20.5 ( 41
2 ), 40 to 27
81 121
( 3 ), and 40 to 30.25 ( 4 )
School also use the percent disadvantaged in a school “to
allocate supplementary hours of instruction and other school
resources” which is why Angrist and Lavy control for it
 Data

Test data came from short-lived national testing program in

Israeli elementary schools
June 1991 and 1992 near end of school year all 4-5th graders
were given achievement tests to measure math and (Hebrew)
reading skills
Average math and reading test scores (scaled from 1-100)
linked with data on school characteristics and class size from
other sources
Beginning of year (BOY) enrollment data was collected and
linked
Class size was collected around the time of the testing and
obtained from administrative source
 Data

The unit of observation in the linked data sets is the class

(remember the example comparing micro and aggregate data
regressions)
linked class-level data includes average test scores in each class,
spring class size, BOY enrollment for each school and grade, a
town identifier, school-level index of student socioeconomic
status called “percent disadvantaged” (PD) and variables
identifying the ethnic and religious composition of the school
Study was limited to Jewish public schools, both secular and
religious schools, which account for the vast majority of
school children in Israel
Arab schools weren’t given the tests; ultra-orthodox Jewish
schools were omitted because they use a curriculum that
differs considerably from public schools
 US IN G M AIM ON IDE S ’ R UL E 539

TABLE I
U NWE IGH TE D D E SCRIP TIVE S TATISTICS

Qu a n t iles

Va r ia ble Mea n S.D. 0.10 0.25 0.50 0.75 0.90

A. F u ll sa m ple
5t h gr a de (2019 cla sses, 1002 sch ools, t est ed in 1991)
Cla ss size 29.9 6.5 21 26 31 35 38
E n r ollm en t 77.7 38.8 31 50 72 100 128
P er cen t disa dva n t a ged 14.1 13.5 2 4 10 20 35
Rea din g size 27.3 6.6 19 23 28 32 36
Ma t h size 27.7 6.6 19 23 28 33 36
Aver a ge ver ba l 74.4 7.7 64.2 69.9 75.4 79.8 83.3
Aver a ge m a t h 67.3 9.6 54.8 61.1 67.8 74.1 79.4
4t h gr a de (2049 cla sses, 1013 sch ools, t est ed in 1991)
Cla ss size 30.3 6.3 22 26 31 35 38
E n r ollm en t 78.3 37.7 30 51 74 101 127
P er cen t disa dva n t a ged 13.8 13.4 2 4 9 19 35
Rea din g size 27.7 6.5 19 24 28 32 36
Ma t h size 28.1 6.5 19 24 29 33 36
Aver a ge ver ba l 72.5 8.0 62.1 67.7 73.3 78.2 82.0
Aver a ge m a t h 68.9 8.8 57.5 63.6 69.3 75.0 79.4
 Discontinuity sample

Maimonides’ rule can be used to identify the effects of class

size because the rule induces a discontinuity in the
relationship between enrollment and class size at enrollment
multiples of 40
Angrist and Lavy (1999) present descriptive statistics for one
such “discontinuity sample” defined as only schools with
enrollments ± 5 students: 36,45; 76,85; 116,125
Slightly fewer than 25% of classes come from schools with
enrollments of this range
Average class size is a bit larger in this ± discontinuity sample
than in the overall sample
But otherwise remarkably similar to the full sample
Cla ss size 30.5 6.2 22 26 31 35 38
E n r ollm en t 79.6 37.3 34 52 74 104 129
P er cen t disa dva n t a ged 13.8 13.4 2 4 9 19 35
Rea din g size 24.5 5.4 17 21 25 29 31
Ma t h size 24.7 5.4 18 21 25 29 31
Aver a ge ver ba l 86.3 6.1 78.4 83.0 87.2 90.7 93.1
Aver a ge m a t h 84.1 6.8 75.0 80.2 84.7 89.0 91.9
B. !/" 5 Discon t in u it y sa m ple (en r ollm en t 36–45, 76–85, 116–124)

5t h gr a de 4t h gr a de 3r d gr a de

Mea n S.D. Mea n S.D. Mea n S.D.

(471 cla sses, (415 cla sses, (441 cla sses,

224 sch ools) 195 sch ools 206 sch ools)

Cla ss size 30.8 7.4 31.1 7.2 30.6 7.4

E n r ollm en t 76.4 29.5 78.5 30.0 75.7 28.2
P er cen t disa dva n t a ged 13.6 13.2 12.9 12.3 14.5 14.6
Rea din g size 28.1 7.3 28.3 7.7 24.6 6.2
Ma t h size 28.5 7.4 28.7 7.7 24.8 6.3
Aver a ge ver ba l 74.5 8.2 72.5 7.8 86.2 6.3
Aver a ge m a t h 67.0 10.2 68.7 9.1 84.2 7.0

Va r ia ble defin it ion s a r e a s follows: Cla ss size # n u m ber of st u den t s in cla ss in t h e spr in g, E n r ollm en t #
Sept em ber gr a de en r ollm en t , P er cen t disa dva n t a ged # per cen t of st u den t s in t h e sch ool fr om ‘‘disa dva n t a ged
ba ckgr ou n ds,’’Rea din g size # n u m ber of st u den t s wh o t ook t h e r ea din g t est , Ma t h size # n u m ber of st u den t s
wh o t ook t h e m a t h t est , Aver a ge ver ba l # a ver a ge com posit e r ea din g scor e in t h e cla ss, Aver a ge m a t h #
a ver a ge com posit e m a t h scor e in t h e cla ss.
 Class size function and enrollment size

es
fsc = es −1
(90)
int 40 +1
where es is the beginning-of-year enrollment in school s in a given grade
(e.g., 5th grade); fsc is class size assigned to class c in school s for that
grade; int(n) is the largest integer less than or equal to n
This equation captures the fact that Maimonides’ rule allows
enrollment cohorts of 1-40 to be grouped in a single class, but
enrollment cohorts of 41-80 are split into two classes of average size
20.5-40, enrollment cohorts of 81-120 are split into three classes of
average size 27-40, and so on.
 Class size function and enrollment size

Although fsc is fixed within schools, in practice enrollment

cohorts are not necessarily divided into classes of equal size
But, even though the actual relationship between class size
and enrollment size involves many factors, in Israel it clearly
has a lot to do with fsc as we’ll see
US IN G M AIM ON IDE S ’ R UL E 541
 Class size function and test scores

The class size function is correlated with average class size as

well as average test scores of fourth and fifth graders
The following picture plots average reading test scores and
average values of fsc by enrollment size in enrollment intervals
of ten for fifth graders
The figure shows that test scores are generally higher in
schools with larger enrollments and larger predicted class sizes
But, it also shows an up-and-down pattern in which average
scores by enrollment size mirror the class-size function
US IN G M AIM ON IDE S ’ R UL E 543
 Class-size function and test scores

The overall positive correlation between test scores and enrollment is partly
attributed to larger schools in Israel being geographically concentrated in
larger, more affluent cities (smaller schools in poorer “developmental towns”
outside the major urban centers)
They note that the enrollment size and the PD index measuring proportion of
students from disadvantaged background is negatively correlated
They control for the “trend” association between test scores and enrollment
size and plot the residuals from regressions of average scores and the average
of fsc on average enrollment and PD index for each interval
The estimates for fifth graders imply a reduction in predicted class size of ten
students is associated with a 2.2 point increase in average reading scores – a
little more than one-quarter of a standard deviation in the distribution of class
averages
US IN G M AIM ON IDE S ’ R UL E 545
 F IGURE III
Aver a ge Test (Rea din g/Ma t h ) Scor es a n d P r edict ed Cla ss Size by E n r ollm en t ,
Residu a ls fr om Regr ession s on P er cen t Disa dva n t a ged a n d E n r ollm en t

do n ot pr ovide a fr a m ewor k for for m a l st a t ist ica l in fer en ce.

Alt h ou gh t h e m icr o da t a for fou r t h a n d fift h gr a der s a r e u n -
a va ila ble, a m odel for in dividu a l pu pils’ t est scor es is u sed t o
 OLS model

yisc = Xs β + nsc δ + µc ηs + εsc (91)

where yisc is pupil i’s score, Xs is a vector of school characteristics,
sometimes including functions of enrollment and nsc is the size of
class c in school s.
The term µc is an i.i.d. random class component, and the
term ηs is an i.i.d. random school component
The class-size coefficient, δ is the primary parameter of
interest.
 OLS model

This equation describes the average potential outcomes of

students under alternative assignments of nsc controlling for
any effects of Xs
If nsc was randomly assigned conditional on Xs , then δ would
be the weighted average response to random variation in class
size along the length of the individual causal response
functions connecting class size and pupil scores
Since nsc is not randomly assigned, in practice it is likely that
it is correlated with potential outcomes – in this case, the
error components in the equation
But while OLS may not have a causal interpretation, using
regression discontinuity design might
 Grouped regression

ȳsc = Xs β + nsc δ + ηs + [µc + ε̄sc ] (92)

where overbars denote averages. The bracketed term is the
class-level error term.

Step 1: “first stage”

nsc = Xs π0 + fsc π1 + ψsc (93)

where πj are parameter and the error term is defined as the
residual from the population regression of ηs onto Xs and fsc and
captures other things associated with enrollment
 TABLE II
OLS E STIMATE S F OR 1991

5t h Gr a de

Rea din g com pr eh en sion Ma t h Rea din g

(1) (2) (3) (4) (5) (6) (7)

M ean score 74.3 67.3

(s.d .) (8.1) (9.9)
R egressors
Cla ss size .221 !.031 !.025 .322 .076 .019 0.141
(.031) (.026) (.031) (.039) (.036) (.044) (.033)
P er cen t disa dva n t a ged !.350 !.351 !.340 !.332
(.012) (.013) (.018) (.018)
E n r ollm en t !.002 .017
(.006) (.009)
Root MSE 7.54 6.10 6.10 9.36 8.32 8.30 7.94
R2 .036 .369 .369 .048 .249 .252 .013
N 2,019 2,018

Th e u n it of obser va t ion is t h e a ver a ge scor e in t h e cla ss. St a n da r d er r or s a r e r epor t ed in pa r en t h eses. St a n da r d er r or s w

 TABLE III
R E DUCE D -F ORM E STIMATE S F OR 1991

5t h Gr a der s 4t h

Rea din g R
Cla ss size com pr eh en sion Ma t h Cla ss size com

(1) (2) (3) (4) (5) (6) (7) (8) (9)

A. F u ll sa m ple

M ean s 29.9 74.4 67.3 30.3

(s.d .) (6.5) (7.7) (9.6) (6.3)
R egressors
f sc .704 .542 !.111 !.149 !.009 !.124 .772 .670 !.085
(.022) (.027) (.028) (.035) (.039) (.049) (.020) (.025) (.031
P er cen t disa dva n t a ged !.076 !.053 !.360 !.355 !.354 !.338 !.054 !.039 !.340
(.010) (.009) (.012) (.013) (.017) (.018) (.008) (.009) (.013
E n r ollm en t .043 .010 .031 .027
(.005) (.006) (.009) (.005)
Root MSE 4.56 4.38 6.07 6.07 8.33 8.28 4.20 4.13 6.64
R2 .516 .553 .375 .377 .247 .255 .561 .575 .311
N 2,019 2,019 2,018 2,049

B. Discon t in u it y sa m ple

M ean s 30.8 74.5 67.0 31.1

(s.d .) (7.4) (8.2) (10.2) (7.2)
R egressors
f sc .481 .346 !.197 !.202 !.089 !.154 .625 .503 !.061
(.053) (.052) (.050) (.054) (.071) (.077) (.050) (.053) (.056
P er cen t disa dva n t a ged !.130 !.067 !.424 !.422 !.435 !.405 !.068 !.029 !.348
(.029) (.028) (.027) (.029) (.039) (.042) (.029) (.028) (.032
E n r ollm en t .086 .003 .041 .063
(.015) (.015) (.022) (.014)
Root MSE 5.95 5.58 6.24 6.24 8.58 8.53 5.49 5.26 6.57
R2 .360 .437 .421 .421 .296 .305 .428 .475 .299
N 471 471 471 415

Th e fu n ct ion f sc is equ a l t o en r ollm en t /[in t ((en r ollm en t ! 1)/40) " 1]. St a n da r d er r or s a r e r epor t ed in pa r en t h eses. St a n da r d er r or s wer
bet ween cla sses. Th e u n it of obser va t ion is t h e a ver a ge scor e in t h e cla ss.
 Second step: use fitted values in grouped regression

ȳsc = Xs β + nc
sc δ + ηs + [µc + ε̄sc ] (94)
where nc
sc is the predicted class size from the previous regression
 TABLE IV
2SLS E STIMATE S F OR 1991 (F IF TH G RADE RS )

Rea din g com pr eh en sion Ma t h

!/" 5
Discon t in u it y
F u ll sa m ple sa m ple F u ll sa m ple

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

M ean score 74.4 74.5 67.3

(s.d .) (7.7) (8.2) (9.6)
R egressors
Cla ss size ".158 ".275 ".260 ".186 ".410 ".582 ".013 ".230 ".261 ".202
(.040) (.066) (.081) (.104) (.113) (.181) (.056) (.092) (.113) (.131)
P er cen t disa dva n t a ged ".372 ".369 ".369 ".477 ".461 ".355 ".350 ".350
(.014) (.014) (.013) (.037) (.037) (.019) (.019) (.019)
E n r ollm en t .022 .012 .053 .041 .062
(.009) (.026) (.028) (.012) (.037)
E n r ollm en t squ a r ed/100 .005 ".010
(.011) (.016)
P iecewise lin ea r t r en d .136 .193
(.032) (.040)
Root MSE 6.15 6.23 6.22 7.71 6.79 7.15 8.34 8.40 8.42 9.49
N 2019 1961 471 2018 1960

Th e u n it of obser va t ion is t h e a ver a ge scor e in t h e cla ss. St a n da r d er r or s a r e r epor t ed in pa r en t h eses. St a n da r d er r or s wer e cor r ect ed for wit h in -sch ool cor r
All est im a t es u se f sc a s a n in st r u m en t for cla ss size.
 How large is their effects?

The Tennessee STAR experiment yielded effect sizes of about 0.13-0.27

standard deviation among pupils and about 0.32 - 0.66 standard deviation in
the distribution of class means
Angrist and Lavy (1999) compare their results by calculating their effect size
associated with reducing class size by eight pupils (same as STAR)
They then multiply this times their second step estimate for reading scores for
fifth graders (-0.275) gives an effect size of around 2.2 points or 0.29 standard
deviation
Their estimates of effect size for fifth graders are at the low end of the range of
those found in the Tennessee experiment
Observational studies are often confounded by a failure to isolate a credible
source of exogenous variation in school inputs, leading one researcher
(Hanushek 1997) to conclude that school inputs don’t matter in academic
performance
RDD overcomes problems of confounding by exploiting exogenous variation
that originates in administrative rules
As with STAR experiment, their study shows that smaller classes appear
beneficial to student academic achievement
 RDD Data Requirements

Question: So, where can I find these “jumps”? Answer: Humans are
embedding “jumps” in their rules all the time. Dumb rules – while usually bad
policy – are great for research.
Validity doesn’t require the assignment rule be arbitrary, only that it is known,
precise and free of manipulation. The most effective RDD studies involve
programs where X has a “hair trigger” that is not tightly related to the
outcome being studied. Examples:
Probability of being tried as an adult (higher penalties for a given crime) “jumps” at age 18
Probability of being arrested for DWI “jumps” at blood alcohol content >0.08
Probability of receiving universal healthcare insurance “jumps” at age 65
Probability of receiving medical attention “jumps” when birthweight falls below 1,500 grams
Probability of having to attend summer school “jumps” when grade falls below 60
Data requirements can be substantial. Large sample sizes are characteristic
features of the RDD
If there are strong trends, one typically needs a lot of data
Researchers are typically using administrative data or settings such as birth
records where there are many observations
 More recent examples

Almond et al. (2010): Infants with low birthweight have both

higher mortality and receive more medical treatment. Medical
treatment becomes more likely when infant weight falls below
1,500 grams (or 3̃ pounds). What’s the effect of medical
treatment on infant mortality?
Anderson and Magruder (2012): At Yelp.com, individuals
reviewed restaurants on a scale of 1 to 5. The aggregate score
was a weighted average of all reviews. Yelp assigned an
integer stars based on which quintile a restaurant was in the
distribution of reviews. What’s the effect of a star on
reservations and revenue?
 Sharp vs. Fuzzy RDD
RDD estimates the causal effect by distinguishing the discontinuous
function, 1(X ≥ X0 ), from the smooth selection function, f (X )
(though Van der Klaauw COLLEGE
uses AID
S instead of X notation) 1261
AND ENROLLMENT

i-
II
H

Q)
o

I,
v)

0)
o

Selection variable S
2
FIGURE
ASSIGNMENT IN THE SHARP (DASHED) AND FUZZY (SOLID) RD DESIGN
 Sharp vs. Fuzzy RDD

1 Sharp RDD: Treatment is a deterministic function of running

variable, X . Example: Medicare benefits.
2 Fuzzy RDD: Discontinuous “jump” in the probability of
treatment when X > X0 . Cutoff is used as an instrumental
variable for treatment. Example: Maimonides Rule (Angrist
and Lavy 1999)
 Treatment assignment in the sharp RDD

Deterministic treatment assignment (“sharp RDD”)

In Sharp RDD, treatment status is a deterministic and
discontinuous function of a covariate, Xi :
(
1 if Xi ≥ X0
Di =
0 if Xi < X0

where X0 is a known threshold or cutoff. In other words, if you

know the value of Xi for a unit i, you know treatment assignment
for unit i with certainty.

Example: Let X be age. Americans aged 64 are not eligible for

Medicare, but Americans aged 65 (X ≥ X65 ) are eligible for
Medicare (ignoring disability exemptions)
 Treatment effect: definition and estimation

Definition of treatment effect

Assume constant treatment effects potential outcomes model linear in X :

Yi0 = α + βXi
Yi1 = Yi0 + δ

Use the switching equation and write in terms of Yi :

Yi = Yi0 + (Yi1 − Yi0 )Di

Yi = α + βXi + δDi + εi

The treatment effect parameter, δ, is the discontinuity in the conditional

expectation function:

δ = limXi →X0 E [Yi1 |Xi = X0 ] − limX0 ←Xi E [Yi0 |Xi = X0 ]

= limXi →X0 E [Yi |Xi = X0 ] − limX0 ←Xi E [Yi |Xi = X0 ]

The sharp RDD estimation is interpreted as an average causal effect of the

treatment at the discontinuity

δSRD = E [Yi1 − Yi0 |Xi = X0 ]

Notice the role of extrapolation in estimating treatment effects
when sharp RDD
Left of cutoff, only non-treated observations, Di = 0 for
X < X0
Right of cutoff, only treated observations, Di = 1 for X ≥ X0

Figure: Dashed lines are extrapolations

 Key identifying assumption

Continuity of conditional regression functions (Hahn, Todd and Van der Klaauw
2001; Lee 2008)
E [Yi0 |X = X0 ] and E [Yi1 |X = X0 ] are continuous (smooth) in X at X0 .

Remark Alfred Marshall quotes Darwin in Principles of Economics (1890), “Natura

non fact saltum”, which means“nature does not make jumps”. Jumps are
unnatural, so when they occur, they require an explanation
Meaning If population average potential outcomes, Y 1 and Y 0 , are continuous
functions of X at the cutoff, X0 , then potential average outcomes do not jump at
X0 . All other unobserved determinants of Y are continuously related to the running
variable, X
Implication This assumption allows us to use average outcome of units right below
the cutoff as a valid counterfactual for units right above the cutoff. The causal
effect of the treatment will be based on extrapolation from the trend,
E [Yi0 |X < X0 ], to those values of X > X0 for the E [Yi0 |X > X0 ].
/// --- Examples using simulated data
set obs 1000
set seed 1234567

* Generate running variable

gen x = rnormal(100, 50)
replace x=0 if x < 0
drop if x > 280
sum x, det

* Set the cutoff at X=140. Treated if X > 140

gen D = 0
replace D = 1 if x > 140
 Graphical example of continuous assumption

gen y1 = 100 + 0*D + 2*x + rnormal(0, 20)

scatter y1 x if D==0, msize(vsmall) || scatter y1 x if D==1, msize(vsmall)

legend(off) xline(140, lstyle(foreground)) || lfit y1 x if D ==0, color(red
|| lfit y1 x if D ==1, color(red) ytitle("Outcome (Y)") xtitle("Test
Score (X)")
 Graphical example with treatment effect, cutoff=140

gen y = 100 + 80*D + 2*x + rnormal(0, 20)

scatter y x if D==0, msize(vsmall) || scatter y x if D==1, ///

msize(vsmall) legend(off) xline(140, lstyle(foreground)) || /// lfit
y x if D ==0, color(red) || lfit y x if D ==1, color(red) /// ytitle("Out
(Y)") xtitle("Test Score (X)")
800
600

Vertical distance is the treatment effect

Outcome (Y)
400
200
0

0 50 100 150 200 250

Test Score (X)
 Tangent: centering at the cutoff point

It is common for authors to transform X by “centering” at X0 :

Yi = α + β(Xi − X0 ) + δDi + εi

This doesn’t change the interpretation of the treatment effect – only the
interpretation of the intercept.
Example: Medicare and age 65. Center the running variable (age) by
subtracting 65:

Y = β0 + β1 (Age − 65) + β2 Edu

= β0 + β1 Age − β1 65 + β2 Edu
= α + β1 Age + β2 Edu

where α = β0 − β1 65. All other coefficients, notice, have the same

interpretation, except for the intercept.
 Nonlinearity bias

Smoothness in E [Yi0 |Xi ] and linearity are different things. What if the trend relation
E [Yi0 |Xi ] does not jump at X0 but rather is simply nonlinear?
gen x2 = x*x
gen x3 = x*x*x
gen y = 10000 + 0*D - 100*x +x2 + rnormal(0, 1000)

scatter y x if D==0, msize(vsmall) || scatter y x if D==1, msize(vsmall) legend(off) xline(140,

lstyle(foreground)) ylabel(none) || lfit y x if D ==0, color(red) || lfit y x if D ==1, color(red)
xtitle("Test Score (X)") ytitle("Outcome (Y)")
 Sharp RDD: Nonlinear Case

Suppose the nonlinear relationship is E [Yi0 |Xi ] = f (Xi ) for

some reasonably smooth function f (Xi ). In that case we’d fit
the regression model:

Yi = f (Xi ) + δDi + ηi

Since f (Xi ) is counterfactual for values of Xi > X0 , how will

we model the nonlinearity? There are 2 ways of approximating
f (Xi ):
1 Let f (Xi ) equal a p th order polynomial:

Yi = α + β1 xi + β2 xi2 + · · · + βp xip + δDi + ηi

2 Use a nonparametric kernel method (later)

 Different polynomials on the 2 sides of the discontinuity

We can generalize the function, f (xi ), by allowing the xi terms to differ on

both sides of the cutoff by including them both individually and interacting
them with Di . In that case we have:

E [Yi0 |Xi ] = α + β01 X̃i + β02 X̃i2 + · · · + β0p X̃ip

E [Yi1 |Xi ] = α + δ + β11 X̃i + β12 X̃i2 + · · · + β1p X̃ip

where X̃i is the centered running variable (i.e., Xi − X0 ). Centering at X0

ensures that the treatment effect at Xi = X0 is the coefficient on Di in a
regression model with interaction terms
As Lee and Lemieux (2010) note, allowing different functions on both sides of
the discontinuity should be the main results in an RDD paper as otherwise we
use values from both sides of the cutoff to estimation the function on each side
 Different polynomials on the 2 sides of the discontinuity

To derive a regression model, first note that the observed values must be used
in place of the potential outcomes:

E [Y |X ] = E [Y 0 |X ] + E [Y 1 |X ] − E [Y 0 |X ] D

Regression model you estimate is:

Yi = α + β01 x̃i + β02 x̃i2 + · · · + β0p x̃ip

+δDi + β1∗ Di x̃i + β2∗ Di x̃i2 + · · · + βp∗ Di x̃ip + εi

where β1∗ = β11 − β01 , β2∗ = β21 − β21 and βp∗ = β1p − β0p
The equation we looked at earlier a few slides back was just a special case of
the above equation with β1∗ = β2∗ = βp∗ = 0
The treatment effect at x0 is δ
The treatment effect at Xi − X0 = c > 0 is: δ + β1∗ c + β2∗ c 2 + · · · + βp∗ c p
 Polynomial simulation example

capture drop y x2 x3
gen x2 = x*x
gen x3 = x*x*x
gen y = 10000 + 0*D - 100*x +x2 + rnormal(0, 1000)

reg y D x x2 x3

predict yhat

scatter y x if D==0, msize(vsmall) || scatter y x if D==1, msize(vsmall)

legend(off) xline(140, lstyle(foreground)) ylabel(none) || line yhat
x if D ==0, color(red) sort || line yhat x if D ==1, sort color(red)
xtitle("Test Score (X)") ytitle("Outcome (Y)")
Outcome (Y)

0 50 100 150 200 250

Test Score (X)

Figure: Third degree polynomial. Actual model second degree polynomial.

Figure: P-order polynomial regressions with and without centering. Only intercept changes.
 Kernel regression

In addition to using p th order polynomials to model the nonlinearities, we can

Thekernel
use Kernelregression.
Method Neither is right or wrong – they have advantages and
disadvantages.
The nonparametric
The nonparametric kernel method
kernel methodhashas
problems because
its problems youcase
in this are trying to
estimate regressions
because at the to
you are trying cutoff pointregressions
estimate which results in cuto§
at the a “boundary
point. problem”
(Hahn, Todd
This andinVan
results der Klaauw
a "boundary 2001)
problem".

WhileWhile the "true"

the “true” e§ect
effect is AB,
is AB, withwith a certain
a certain bandwidtha arectangular
bandwidth rectangularkernel
wouldkernel wouldthe
estimate estimate theAe§ect
effect as 0 B 0 as A’B’.

ThereThere is therefore
is therefore systematic
systematic biasbias
withwith
thethe kernel
kernel method
method if ifthe f (X
thef (X ) )is
is upwards or downwards sloping.
upwards or downwards sloping
Waldinger (Warwick) 21 / 48
 Kernel Method - Local linear regression

The standard solution to this problem is to run local linear nonparametric

regression (Hahn, Todd and Van der Klaauw 2001). In the case described by
the previous slide, this would substantially reduce the bias
Think of it as a weighted regression restricted to a window (hence “local”).
The kernel provides the weights to that regression. STATA’s poly command
estimates kernel-weighted local polynomial regression.
A rectangular kernel would give the same result as taking E [Y ] at a given bin
on X . The triangular kernel gives more importance to the observations closer
to the center.
While estimating this in a given window of width h around the cutoff is
straightforward, it’s more difficult to choose this bandwidth (or window), and
the method is sensitive to the choice of bandwidth. There is essentially a
tradeoff between bias and efficiency. See Lee and Lemieux (2010) for two
methods to choose on the bandwidth. This is an active area of research in
econometrics.
Example of sharp RDD: David Card, Carlos Dobkin and Nicole
Maestas (2008), “The Impact of Nearly Universal Insurance
Coverage on Health Care Utilization: Evidence from Medicare”,
American Economic Review.
 Non-elderly health insurance information

20% of non elderly adults in US lacked health insurance

coverage in 2005
Most were from lower-income families, nearly one-half were
African American or Hispanic
Many analysts have argued that unequal insurance coverage
contributes to disparities in health care utilization and health
outcomes across SES
Even among the insured, there are differences: copayments,
deductibles, other features that affect use
Evidence that better insurance causes better health outcomes
is limited
Both supply and demand for insurance depend on health
status, confounding observational comparisons between people
with different insurance characteristics
 Elderly health insurance information

< 1% of the elderly population are uninsured

Most have fee-for-service Medicare coverage
The transition occurs abruptly at age 65, which is the
threshold for Medicare eligibility
 Estimation

Reduced form measure of causal effect of health insurance

status on health care use, y :
X
k k
yija = Xija α + fj (α; β) + Cija δ + uija
k

where i indexes individuals, j indexes a socioeconomic group,

a indexes an age, uija the unobserved error component, Xija a
set of covariates (e.g., gender and region), fj (α; β) a smooth
function representing the age profile of outcome y for group j,
k (k = 1, 2, . . . , K ) are characteristics of the insurance
and Cija
coverage held by the individual such as copayment rates
Problem: Insurance coverage is endogenous
→ cov (uija , Cija ) 6= 0.
Identification: age threshold for Medicare eligibility at 65 used
as credibly exogenous variation in insurance status.
 Insurance and age
2244 THE AMERICAN ECONOMIC REVIEW DECEMBER 2008




'SBDUJPO
XJUI
DPWFSBHF



"OZ
DPWFSBHF
IJHIFE
XIJUFT

 "OZ
DPWFSBHF
BMM
HSPVQT
"OZ
DPWFSBHF
MPXFE
NJOPSJUJFT
5XP
QPMJDJFT
IJHIFE
XIJUFT
5XP
QPMJDJFT
BMM
HSPVQT
 5XP
QPMJDJFT
MPXFE
NJOPSJUJFT


                   
"HF
JO
RVBSUFST

Figure 1. Coverage by Any Insurance and by Two or More Policies, by Age and Demographic Group

where hj 1a 2 5 fj 1a 2 1 d1g1j 1a 2 1 d2g2j 1a 2 represents the reduced-form age profile for group j, p yj
5 p1j d1 1 p2j d2, and vijay
5 uija 1 v1ija d1 1 v2ija d2 is an error term. Assuming that the profiles
 Estimation

Suppose health insurance coverage can be summarized by two dummy variables

1 (any coverage) and C 2 (generous insurance). Linear probability model:
– Cija ija

1
Cija = Xija βj1 + gj1 (a) + Da πj1 + vija
1

2
Cija = Xija βj2 + gj2 (a) + Da πj2 + vija
2

where βj1 and βj2 are group-specific coefficients, gj1 (a) and gj2 (a) are smooth
age profiles for group j, and Da is a dummy for being age ≥65.
Reduced form from couple slides back
X
k k
yija = Xija α + fj (α; β) + Cija δ + uija
k

Combining Cija equations, and rewrite reduced for model

yija = Xija (αj + βj1 δj1 + βj2 δj2 ) + hj (a) + Da πjy + vija
y

where h(a) = fj (a) + δ 1 gj1 (a) + δ 2 gj2 (a) is reduced form age profile for group j,
πjy = πj1 δ 1 + πj2 δ 2 and vija
y 1 δ 1 + v 2 δ 2 is the error term.
= uija + vija ija
Assuming that the profiles fj (a), gj (a) and gj2 (a) are continuous at age 65
1

(i.e., continuous assumption), then any discontinuity in y is due to insurance

and magnitudes depend on the size of the insurance changes at 65 (πj1 and πj2 )
and on the associated causal effects (δ 1 and δ 2 )
 Estimation

For some basic health care services (e.g., routine doctor visits), it may be the
only thing that matters is insurance.
But, in those situations, the implied discontinuity in y at 65 for group j will be
proportional to the change in insurance status experienced by that group
For more expensive (or elective) services, the generosity of the coverage may
matter (for instance, if patients are unwilling to cover the required copay or if
the managed care program won’t cover the service).
This creates a potential identification problem in interpreting the discontinuity
in y for any one group
Since πjy is a linear combination of the discontinuities in coverage and
generosity, δ 1 and δ 2 can be estimated by a regression across groups:

πjy = δ 0 + δ 1 πj1 + δj2 + ej

where ej is an error term reflecting a combination of the sampling errors in πjy ,

πj1 and πj2
RDD models like the ones on the previous slide are fit with OLS by
demographic subgroup to individual data
Estimates are combined across groups to estimate the above model
 Data

1992-2003 National Health Interview Survey (NHIS)

NHIS reports respondents’ birth year and birth month, and the calendar quarter
of the interview. Authors construct an estimate of age in quarters at date of
interview.
A person reaching 65 in the interview quarter is age 65 and 0 quarters
Assuming a uniform distribution of interview dates, one-half of these people will
be 0-6 weeks younger than 65 and one-half will be 0-6 weeks older
Analysis is limited to people >55 and <75. Final sample size is n =160,821
(although some outcomes are not available for earlier years)
1992-2003 hospital discharge records for California, Florida and New York
Discharge files represent a complete census of discharges from all hospitals in the
three states (except federally regulated institutions).
Data files include information on age in months at the time of admission
Sample selection criteria: drop records for people admitted as transfers from
other institutions, and limit people between 60 and 70 years of age at admission
Sample sizes are 4,017,325 (California), 2,793,547 (Florida), 3,121,721 (New
York)
 Medicare description

Description of Medicare eligibility

Medicare is available to people who are at least 65 and have
worked 40 quarters or more in covered employment or have a
spouse who did.
Coverage is available to younger people with severe kidney
disease and recipients of Social Security Disability Insurance
Eligible individuals can obtain Medicare hospital insurance
(Part A) free of charge, and medical insurance (Part B) for a
modest monthly premium
Individuals receive notice of their impending eligibility for
Medicare shortly before 65th birthday and are informed they
have to enroll in it and choose whether to accept Part B
coverage.
Coverage begins on the first day of the month in which they
turn 65.
 Medicare description

Five insurance-related variables: probability of Medicare

coverage, any health insurance coverage, private coverage,
two or more forms of coverage, and individual’s primary health
insurance is managed care program.
Data are drawn from 1999-2003 NHIS and for each
characteristic, authors show the incidence rate at ages 63-64
and the change at age 65 based on a version of the CK
equations that include a quadratic in age, fully interacted with
a post-65 dummy as well as controls for gender, education,
race/ethnicity, region and sample year
Alternative specifications were also used – a parametric model
fit to a narrower age window (ages 63-67) and a local linear
regression specification using a chosen bandwidth. Both show
show similar estimates of the change at age 65.
 Changes in insurance coverage at age 65
2246 THE AMERICAN ECONOMIC REVIEW DECEMBER 2008

Table 1—Insurance Characteristics Just before Age 65 and Estimated Discontinuities at Age 65
On Medicare Any insurance Private coverage 21 Forms coverage Managed care
Age RD Age RD Age RD Age RD Age RD
63–4 at 65 63–4 at 65 63–4 at 65 63–4 at 65 63–4 at 65
112 122 132 142 152 162 172 182 192 1102

Overall sample 12.3 59.7 87.9 9.5 71.8 22.9 10.8 44.1 59.4 228.4
14.12 10.62 11.12 12.82 12.12
Classified by ethnicity and education:
White non-Hispanic:
High school dropout 21.1 58.5 84.1 13.0 63.5 26.2 15.0 44.5 48.1 225.0
14.62 12.72 13.32 14.02 14.52
High school graduate 11.4 64.7 92.0 7.6 80.5 21.9 10.1 51.8 58.9 230.3
15.02 10.72 11.62 13.82 12.62
At least some college 6.1 68.4 94.6 4.4 85.6 22.3 8.8 55.1 69.1 240.1
14.72 10.52 11.82 14.02 12.62
Minority:
High school dropout 19.5 44.5 66.8 21.5 33.2 21.2 11.4 19.4 39.1 28.3
13.12 12.12 12.52 11.92 13.12
High school graduate 16.7 44.6 85.2 8.9 60.9 25.8 13.6 23.4 54.2 215.4
14.72 12.82 15.12 14.82 13.52
At least some college 10.3 52.1 89.1 5.8 73.3 25.4 11.1 38.4 66.2 222.3
14.92 12.02 14.32 13.82 17.22
Classified by ethnicity only:
White non-Hispanic 10.8 65.2 91.8 7.3 79.7 22.8 10.4 51.9 61.9 233.6
1all2 14.62 10.52 11.42 13.52 12.32
Black non-Hispanic 17.9 48.5 84.6 11.9 57.1 24.2 13.4 27.8 48.2 213.5
1all2 13.62 12.02 12.82 13.72 13.72
Hispanic 1all2 16.0 44.4 70.0 17.3 42.5 22.0 10.8 21.7 52.9 212.1
13.72 13.02 11.72 12.12 13.72

Note: Entries in odd-numbered columns are percentages of age 63- 64-year-olds in group with insurance characteristic
shown in column heading. Entries in even-numbered columns are estimated regression discontinuties at age 65, from
models that include quadratic control for age, fully interacted with dummy for age 65 or older. Other controls include
indicators for gender, race/ethnicity, education, region, and sample year. Estimates are based on linear probability
models fit to pooled samples of 1999–2003 NHIS.

Medicare coverage rises by 60 percentage points at age 65, from a base level of 12 percent
among 63- 64-year-olds. Consistent with DI enrollment patterns (David H. Autor and Mark G.
Duggan 2003), Medicare enrollment prior to 65 is higher for minorities and people with below-
average schooling, and these groups experience relatively smaller gains at age 65 (see rows 2–7).
 Other changes at age 65

Formal identification of an RD model relating an outcome y (insurance

coverage) to a treatment (Medicare age-eligibility) that depends on age (a)
relies on assumption that the conditional expectation functions for both
potential outcomes is continuous at a = 65. That is, E [Y 0 |a] and E [Y 1 |a] are
continuous at a = 65. If so, then the average treatment effect at age 65 is
identified as
lim65←a E [y 1 |a] − lima→65 E [y 0 |a]
Continuity requires that all other factors that might affect the outcome of
interest (insurance coverage) also trend smoothly at the cutoff a = 65.
What else changes, though, at 65?
Employment. 65 is the traditional age of retirements. Any abrupt change in
employment could lead to differences in health care utilization if non workers have
more time to visit doctors.
Test for potential discontinuities at age 65 for confounding variables using the
1996-2004 March CPS with age measured in years. No evidence for large
discontinuities in employment (next slide).
 Other changes at age 65

VOL. 98 NO. 5 CARD ET AL: THE EFFECT OF MEDICARE ON HEALTH CARE UTILIZATION 2249



 )JHIFE
XIJUFT‡BDUVBM

)JHIFE
XIJUFT‡QSFEJDUFE

0WFSBMM
TBNQMF‡BDUVBM

 0WFSBMM
TBNQMF‡QSFEJDUFE
'SBDUJPO
FNQMPZFE

-PXFE
NJOPSJUJFT‡BDUVBM
 -PXFE
NJOPSJUJFT‡QSFEJDUFE










                   

"HF

Figure 2. Employment Rates by Age and Demographic Group (1992–2003 NHIS)

gender, and show no large discontinuities for either men or women.6 As an additional check,
 Effect of cutoff on access to care and utilization
Since 1997, NHIS has asked two questions: (1) “During the past 12 months has medical care been delayed
for this person because of worry about the cost?” and (2) “‘During the past 12 months was there any time
when this person needed medical care but did not get it because (this person) could not afford it?”
2250 THE AMERICAN ECONOMIC REVIEW DECEMBER 2008

Table 3—Measures of Access to Care Just before 65 and Estimated Discontinuities at 65

1997–2003 NHIS 1992–2003 NHIS

Delayed care last year Did not get care last year Saw doctor last year Hospital stay last year
Age 63–64 RD at 65 Age 63–64 RD at 65 Age 63264 RD at 65 Age 63264 RD at 65
112 122 132 142 152 162 172 182

Overall sample 7.2 21.8 4.9 21.3 84.8 1.3 11.8 1.2
10.42 10.32 10.72 10.42

Classified by ethnicity and education:

White non-Hispanic:
High school dropout 11.6 21.5 7.9 20.2 81.7 3.1 14.4 1.6
11.12 11.02 11.32 11.32
High school graduate 7.1 0.3 5.5 21.3 85.1 20.4 12.0 0.3
12.82 12.82 11.52 10.72
At least some college 6.0 21.5 3.7 21.4 87.6 0.0 9.8 2.1
10.42 10.32 11.32 10.72
Minority:
High school dropout 13.6 25.3 11.7 24.2 80.2 5.0 14.5 0.0
11.02 10.92 12.22 11.42
High school graduate 4.3 23.8 1.2 1.5 84.8 1.9 11.4 1.8
13.22 13.72 12.72 11.42
At least some college 5.4 20.6 4.8 20.2 85.0 3.7 9.5 0.7
11.12 10.82 13.92 12.02
Classified by ethnicity only:
White non-Hispanic 6.9 21.6 4.4 21.2 85.3 0.6 11.6 1.3
10.42 10.32 10.82 10.52
Black non-Hispanic 1all2 7.3 21.9 6.4 20.3 84.2 3.6 14.4 0.5
11.12 11.12 11.92 11.12
Hispanic 1all2 11.1 24.9 9.3 23.8 79.4 8.2 11.8 1.0
10.82 10.72 10.82 11.62

Note: Entries in odd numbered columns are mean of variable in column heading among people ages 63–64. Entries in
even numbered columns are estimated regression discontinuties at age 65, from models that include linear control for
age interacted with dummy for age 65 or older (columns 2 and 4) or quadratic control for age, interacted with dummy
for age 65 and older (columns 6 and 8). Other controls in models include indicators for gender, race/ethnicity, educa-
tion, region, and sample year. Sample in columns 1–4 is pooled 1997–2003 NHIS. Sample in columns 5–8 is pooled
1992–2003 NHIS. Samples for regression models include people ages 55–75 only. Standard errors (in parentheses) are
clustered by quarter of age.
 Effect of cutoff on access to care and utilization
The right-hand columns of Table 3 present results for two key measures of healthcare utilization: (1) “Did
the individual have at least one doctor visit in the past year?” and (2) “Did the individual have one or more
overnight hospital stays in the past year?”
2250 THE AMERICAN ECONOMIC REVIEW DECEMBER 2008

Table 3—Measures of Access to Care Just before 65 and Estimated Discontinuities at 65

1997–2003 NHIS 1992–2003 NHIS

Delayed care last year Did not get care last year Saw doctor last year Hospital stay last year
Age 63–64 RD at 65 Age 63–64 RD at 65 Age 63264 RD at 65 Age 63264 RD at 65
112 122 132 142 152 162 172 182

Overall sample 7.2 21.8 4.9 21.3 84.8 1.3 11.8 1.2
10.42 10.32 10.72 10.42

Classified by ethnicity and education:

White non-Hispanic:
High school dropout 11.6 21.5 7.9 20.2 81.7 3.1 14.4 1.6
11.12 11.02 11.32 11.32
High school graduate 7.1 0.3 5.5 21.3 85.1 20.4 12.0 0.3
12.82 12.82 11.52 10.72
At least some college 6.0 21.5 3.7 21.4 87.6 0.0 9.8 2.1
10.42 10.32 11.32 10.72
Minority:
High school dropout 13.6 25.3 11.7 24.2 80.2 5.0 14.5 0.0
11.02 10.92 12.22 11.42
High school graduate 4.3 23.8 1.2 1.5 84.8 1.9 11.4 1.8
13.22 13.72 12.72 11.42
At least some college 5.4 20.6 4.8 20.2 85.0 3.7 9.5 0.7
11.12 10.82 13.92 12.02
Classified by ethnicity only:
White non-Hispanic 6.9 21.6 4.4 21.2 85.3 0.6 11.6 1.3
10.42 10.32 10.82 10.52
Black non-Hispanic 1all2 7.3 21.9 6.4 20.3 84.2 3.6 14.4 0.5
11.12 11.12 11.92 11.12
Hispanic 1all2 11.1 24.9 9.3 23.8 79.4 8.2 11.8 1.0
10.82 10.72 10.82 11.62

Note: Entries in odd numbered columns are mean of variable in column heading among people ages 63–64. Entries in
even numbered columns are estimated regression discontinuties at age 65, from models that include linear control for
age interacted with dummy for age 65 or older (columns 2 and 4) or quadratic control for age, interacted with dummy
for age 65 and older (columns 6 and 8). Other controls in models include indicators for gender, race/ethnicity, educa-
tion, region, and sample year. Sample in columns 1–4 is pooled 1997–2003 NHIS. Sample in columns 5–8 is pooled
1992–2003 NHIS. Samples for regression models include people ages 55–75 only. Standard errors (in parentheses) are
clustered by quarter of age.
 Changes in hospitalizations
2252 THE AMERICAN ECONOMIC REVIEW DECEMBER 2008

 

)PTQJUBM
BENJTTJPOT
GPS
IJQ
BOE
LOFF
SFQMBDFNFOU
QFS
 

"MM
BENJTTJPOT
MFGU
TDBMF
"MM
BENJTTJPOT
QSFEJDUFE
5PUBM
IPTQJUBM
BENJTTJPOT
QFS
 
QFSTPO
ZFBST

8IJUF
IJQ
BOE
LOFF
SFQMBDFNFOU
8IJUF
IJQ
BOE
LOFF
SFQMBDFNFOU
GJUUFE
)JTQBOJD
IJQ
BOE
LOFF
SFQMBDFNFOU
)JTQBOJD
IJQ
BOE
LOFF
SFQMBDFNFOU
GJUUFE


 #MBDL
IJQ
BOE
LOFF
SFQMBDFNFOU
#MBDL
IJQ
BOE
LOFF
SFQMBDFNFOU
GJUUFE





QFSTPO
ZFBST









 
          
"HF

Figure 3. Hospital Admission Rates by Race/Ethnicity

 Effect of cutoff on access to care and utilization
VOL. 98 NO. 5 CARD ET AL: THE EFFECT OF MEDICARE ON HEALTH CARE UTILIZATION 2253

Table 4—Hospital Admissions and Insurance Coverage at Age 65: California, Florida, and New York
All Whites Hispanics Blacks
Rate age RD at 65 Rate age RD at 65 Rate age RD at 65 Rate age RD at 65
60–64 60–64 60–64 60–64
112 122 132 142 152 162 172 182
Hospital admissions
All admissions 1,443 7.57 1,407 7.74 1,262 9.47 2,008 4.39
10.292 10.332 10.552 10.712
By route into hospital
ER admission 761 3.30 688 3.70 774 2.63 1,313 1.93
10.392 10.402 10.922 10.952
Non–ER admission 682 12.16 718 11.51 488 19.89 695 8.92
10.462 10.492 11.052 11.042
By admission diagnosis
Chronic ischemic heart disease 83 11.58 90 11.05 59 18.45 66 8.29
10.962 11.162 12.452 12.782
AMI 48 4.41 50 5.31 38 3.90 45 23.43
11.432 11.652 13.332 14.782
Heart failure 56 0.44 45 2.33 62 24.85 130 21.47
11.112 11.242 12.632 12.432
Chronic bronchitis 34 7.50 36 6.50 19 9.76 38 13.05
11.512 11.522 15.582 14.432
Osteoarthrosis 34 26.97 38 27.16 18 29.27 27 22.08
11.392 11.642 15.052 14.012
Pneumonia 34 2.44 32 2.05 30 3.39 51 3.81
11.422 11.742 14.342 13.212
By primary procedure
None 419 5.70 400 5.73 388 7.23 614 3.86
10.332 10.402 11.232 11.252
Diagnostic procedures on heart 51 9.18 53 8.17 40 16.78 58 8.76
11.032 11.202 13.212 13.322
Removal of coronary artery obstruction 38 10.67 43 10.49 23 18.77 22 0.49
11.462 11.602 13.942 15.332
Bypass anastomosis of heart 26 15.91 29 16.17 17 18.97 13 5.15
11.392 11.442 15.622 16.092
Joint replacement lower extremity 41 22.69 46 23.16 22 26.40 33 12.14
11.472 11.602 14.692 14.202
Diagnostic procedure on small intestine 35 7.35 31 6.60 37 13.07 58 4.09
11.272 11.472 13.272 13.132
Cholecystectomy 1gall bladder removal2 26 17.93 26 16.00 29 29.25 18 12.27
12.102 11.842 15.112 17.502
Insurance coverage
Probability of coverage 82.7 15.0 86.7 12.7 69.1 20.3 79.0 17.6
1March CPS data2 10.82 10.82 12.72 12.72

Notes: Insurance estimates are based on pooled March CPS 1996–2004 data for California, Florida, and New York.
Entries in top row columns 1, 3, 5, 7 are fractions of 60- and 64-year-olds with insurance coverage. Entries in top row
columns 2, 4, 6, 8 represent regression discontinuity estimates 131002 of the increase in coverage at age 65 from a
model with a quadratic in age, fully interacted with a post-65 dummy. Entries in lower rows columns 1, 3, 5, 7 are hospi-
tal admission rates 1 per 10,000 person years for 60- to 64-year-olds2 for California, Florida, and New York 1992–2002.
Entries in lower rows columns 2, 4, 6, 8 are regression discontinuity estimates 131002 of the increase in the log of the
number of admissions at age 65, from models with a quadratic in age, fully interacted with a post-65 dummy. Standard
errors are in parentheses.
 Changes in ownership

VOL. 98 NO. 5 CARD ET AL: THE EFFECT OF MEDICARE ON HEALTH CARE UTILIZATION 2255

 
 
 

 

'PSQSPGJU
DPVOUZ
,BJTFS
DIVSDI

 
1SJWBUF
OPOQSPGJU
BENJTTJPOT

1SJWBUF
OPOQSPGJU  
  $IVSDI3VO
1SJWBUF
GPSQSPGJU

BENJTTJPOT
 
 
,BJTFS

)PTQJUBM
EJTUSJDU
 
 

   
$PVOUZ

   

 
          

"HF

Figure 4. Hospital Admission in California by Ownership Type (1992–2002)

 Socioeconomic disparities
VOL. 98 NO. 5 CARD ET AL: THE EFFECT OF MEDICARE ON HEALTH CARE UTILIZATION 2257

Table 5—Summary of Effects of Insurance Coverage on Socioeconomic Disparities

Percent change in disparity due to

Disparities at ages 63–64 change in coverage at 65
Coefficient on Low-ed minority- Black- Hispanic- Low-ed minority- Black- Hispanic-
coverage RD R2 Hi-ed Whites White White Hi-ed Whites White White
Outcome (1) (2) (3) (4) (5) (6) (7) (8)

Panel A: Based on change in insurance coverage at 65

Delay in care last year 20.19 0.72 7.6 — — 242.8 — —
(0.06)
No care last year 20.12 0.39 8.0 — — 225.6 — —
(0.06)
Regular doctor visit last year 0.32 0.77 27.4 — — 273.9 — —
(0.09)
Hospital stay last year 20.09 0.26 4.7 — — 232.7 — —
(0.08)
Total hospital admissions 0.06 0.74 — 724 2193 — 1.8 20.5
(0.18)
Diagnostic procedures 0.59 0.62 — 9 225 — 35.6 4.5
of the heart (0.31)
Bypass anastomosis of heart 20.54 0.54 — 218 214 — 24.9 24.8
(0.93)
Joint replacement of lower 20.19 0.89 — 27 228 — 2.7 22.3
extremity (0.64)
Panel B: Based on change in incidence of multiple coverage at 65
Total hospital admissions 0.03 0.74 — 724 2193 — 20.1 5.8
(0.08)
Diagnostic procedures 20.21 0.55 — 9 225 — 22.3 218.7
of the heart (0.14)
Bypass anastomosis 0.46 0.64 — 218 214 — 29.4 37.1
of heart (0.34)
Joint replacement lower 0.42 0.94 — 27 228 — 59.4 25.7
of extremity (0.21)

Notes: Each entry in panel A, column 1, is estimated coefficient from regression of RDs in listed health outcome on
RDs in insurance coverage over six ethnicity/education groups (rows 1–4) or nine state-ethnicity groups (rows 5–8).
All regressions weighted by the inverse sampling variance of the estimated discontinuity in each outcome, and regres-
sions in rows 5–8 include state dummies. Entries in column 2 are corresponding R-squared coefficients from each
regression. Entries in columns 3, 4, and 5 are the observed disparities in each health outcome at ages 63–64, and entries
in columns 6, 7, and 8 are the percent change in the disparity attributable to the change in insurance coverage based on
the coefficient in column 1. Health disparities measured in the NHIS are characterized in terms of low-ed minorities
versus hi-ed whites, whereas health disparities measured in the hospital discharge data are characterized in terms of
black-white or hispanic-white differences. Panel B is similar to panel A except that the RDs in each health outcome are
regressed on the RDs in the incidence of multiple coverage at 65. Panel B regressions are based on data for New York
and Florida only (i.e., six state-ethnicity groups).
 Summary of Card, et al. (2009)

David Card, Carlos Dobkin and Nicole Maestas (2009), “Does

Medicare Save Lives?”, American Economic Review
 Impact of Medicare on admissions
DOES MEDICARE SAVE LIVES? 605

6.40

6.35
Unplanned ED admission Unplanned ED admissions fitted

Other admissions Other admissions fitted

6.30

6.25

Downloaded from http://qje.oxfordjournals.org/ at University of Kentucky Lib

6.20
Log(admissions)

6.15

6.10

6.05

6.00

5.95

5.90
60 61 62 63 64 65 66 67 68 69 70
Age in months

FIGURE II
Number of Admissions by Route into Hospital, California, 1992–2002
The lines are fitted values from regressions that include a second-order polyno-
mial in age fully interacted with a dummy for age !65 and a dummy variable for
the month before people turn 65. The dependent variable is the log of the number
of admissions by patient’s age (in days) at admission, for patients between 60 and
70 years of age. The count of admissions is based on hospital discharge records for
California and includes admissions from January 1, 1992, to November 30, 2002.
The points represent means of the dependent variable for 30-day cells. The age
profile for unplanned ED admissions includes admissions that occurred through
the emergency department and were unplanned. The category “Other Admissions”
includes all other admissions.
 Proportion with coverage
DOES MEDICARE SAVE LIVES? 615

Private Private fitted

0.9
Medicare Medicare fitted
Medicaid Medicaid fitted
0.8 Other insurance Other insurance fitted
Uninsured Uninsured fitted
0.7
Proportion with coverage

Downloaded from http://qje.oxfordjournals.org/ at University of Kentucky

0.6

0.5

0.4

0.3

0.2

0.1

0
60 61 62 63 64 65 66 67 68 69 70
Age at admission

FIGURE IV
Primary Insurance Coverage of Admitted Patients
See notes for Figure II. In this figure the y-axis represents the fraction of
patients with different classes of primary insurance coverage. Sample includes
425,315 patients with nondeferrable primary diagnoses, defined as unplanned
admissions through the emergency department for diagnoses with a t-statistic for
the test of equal weekday and weekend admission rates of 0.965 or less. Medicare
eligibility status of patients within one month of their 65th birthdays is uncertain
and we have excluded these observations.
 Impact of Medicare on type of coverage

616
TABLE III

QUARTERLY JOURNAL OF ECONOMICS

REGRESSION DISCONTINUITY MODELS FOR PROBABILITY OF DIFFERENT FORMS OF PRIMARY INSURANCE COVERAGE

Medicare Private Medicaid Uninsured

(1) (2) (3) (4) (5) (6) (7) (8)

Age over 65 (×100) 43.9 47.5 −24.8 −26.8 −10.1 −10.8 −7.4 −8.0
(0.4) (0.4) (0.4) (0.4) (0.3) (0.3) (0.2) (0.2)
Additional controls No Yes No Yes No Yes No Yes
Mean of dependent variable 24.0 43.3 43.3 9.7
for patients aged 64–65 (×100)

Notes. Standard errors in parentheses. Dependent variable is indicator for type of insurance listed as “primary insurer” on discharge record. Sample includes 425,315 observations
on patients between the ages of 60 and 70 admitted to California hospitals between January 1, 1992, and November 30, 2002 for an unplanned admission through the emergency
department, with a diagnosis (ICD-9) for which the t-test for equality of weekend and weekday admission rates is less than 0.96 in absolute value. All models include second-order
polynomial in age (in days) fully interacted with dummy for age over 65 and are fit by OLS. Models in even-numbered columns include the following additional controls: a dummy for
people who are within one month of their 65th birthday; dummies for month, year, sex, race/ethnicity, and admission on Saturday or Sunday; and a complete set of unrestricted fixed
effects for each ICD-9 admission diagnosis. In columns (1)–(8) the coefficient on “age over 65” and its standard error have been multiplied by 100.

Downloaded from http://qje.oxfordjournals.org/ at University of Kentucky Libraries on November 29, 2011

 Treatment intensity
618 QUARTERLY JOURNAL OF ECONOMICS

10.20 10

10.15
9

10.10
8
10.05

Days and number of procedures

7

Downloaded from http://qje.oxfordjournals.org/ at University of Kentucky

10.00
Log (list charges)

9.95 6

9.90
5

9.85
4
Log list charges Log charges fitted
9.80 Length of stay fitted Length of stay (days)
Number of procedures Procedures fitted
3
9.75

9.70 2
60 61 62 63 64 65 66 67 68 69
Age at admission

FIGURE V
Three Measures of Inpatient Treatment Intensity
See notes to Figure IV. Sample includes unplanned admissions through the
emergency department for diagnoses with a t-statistic for the test of equal weekday
and weekend admission rates of 0.965 or less. In this figure the sample is further
restricted to patients with valid SSNs (407,386 observations). Sample for log list
charges excludes patients admitted to Kaiser hospitals. Length of stay, number of
procedures, and list charges are cumulated over all consecutive hospitalizations.
List charges are measured in 2002 dollars.
 Treatment intensity

TABLE IV
REGRESSION DISCONTINUITY MODELS FOR CHANGES IN TREATMENT INTENSITY

DOES MEDICARE SAVE LIVES?

Length of stay (days) Number of procedures Log list charges (×100)

(1) (2) (3) (4) (5) (6)

Age over 65 0.37 0.35 0.09 0.11 2.5 2.6

(0.24) (0.26) (0.03) (0.03) (1.1) (1.0)
Additional controls No Yes No Yes No Yes
Mean of dependent variable 8.12 2.50 9.87
for patients aged 63 or 64

Notes. Standard errors in parentheses. Dependent variable is length of stay in days (columns (1) and (2)), number of procedures performed (columns (3) and (4)), and log of total
list charges (columns (5) and (6)). Sample includes 407,386 (352,652 in columns (5) and (6)) observations on patients with valid SSNs between the ages of 60 and 70 admitted to
California hospitals between January 1, 1992, and November 30, 2002 for an unplanned admission through the ED. Data on list charges are missing for Kaiser hospitals. See note to
Table III for additional details on sample, and list of additional covariates included in even-numbered columns. In columns (5) and (6) the coefficient on “age over 65” and its standard
error have been multiplied by 100.

619
Downloaded from http://qje.oxfordjournals.org/ at University of Kentucky Libraries on November 29, 2011
 Mortality

DOES MEDICARE SAVE LIVES? 621

0.30
Death 365 days Death 365 days fitted Death 28 days fitted
Death 180 days Death 180 days fitted Death 14 days fitted
Death 90 days Death 90 days fitted Death 7 days fitted
Death 28 days
0.25 Death 14 days
Death 7 days

0.20

Downloaded from http://qje.oxfordjournals.org/ at University of Kentuc

Death rate (fraction)

0.15

0.10

0.05

0.00
60 61 62 63 64 65 66 67 68 69 70
Age at admission

FIGURE VI
Patient Mortality Rates over Different Follow-Up Intervals
See notes to Figure IV. Sample includes unplanned admissions through the
emergency department for diagnoses with a t-statistic for the test of equal week-
day and weekend admission rates of 0.965 or less. In this figure the sample is
further restricted to patients with valid SSNs (407,386 observations). Deaths in-
clude include in-hospital and out-of-hospital deaths.
 Mortality

622
TABLE V
REGRESSION DISCONTINUITY ESTIMATES OF CHANGES IN MORTALITY RATES

Death rate in

QUARTERLY JOURNAL OF ECONOMICS

7 days 14 days 28 days 90 days 180 days 365 days

Estimated discontinuity at age 65 (×100)

Fully interacted quadratic with no −1.1 −1.0 −1.1 −1.1 −1.2 −1.0
additional controls (0.2) (0.2) (0.3) (0.3) (0.4) (0.4)
Fully interacted quadratic plus −1.0 −0.8 −0.9 −0.9 −0.8 −0.7
additional controls (0.2) (0.2) (0.3) (0.3) (0.3) (0.4)
Fully interacted cubic plus additional −0.7 −0.7 −0.6 −0.9 −0.9 −0.4
controls (0.3) (0.2) (0.4) (0.4) (0.5) (0.5)
Local linear regression procedure fit −0.8 −0.8 −0.8 −0.9 −1.1 −0.8
separately to left and right with (0.2) (0.2) (0.2) (0.2) (0.3) (0.3)
rule-of-thumb bandwidths
Mean of dependent variable (%) 5.1 7.1 9.8 14.7 18.4 23.0

Notes. Standard errors in parentheses. Dependent variable is indicator for death within interval indicated by column heading. Entries in rows (1)–(3) are estimated coefficients of
dummy for age over 65 from models that include a quadratic polynomial in age (rows (1) and (2)) or a cubic polynomial in age (row (3)) fully interacted with a dummy for age over 65.
Models in rows (2) and (3) include the following additional controls: a dummy for people who are within 1 month of their 65 birthdays, dummies for year, month, sex, race/ethnicity,
and Saturday or Sunday admissions, and unrestricted fixed effects for each ICD-9 admission diagnosis. Entries in row (4) are estimated discontinuities from a local linear regression
procedure, fit separately to the left and right, with independently selected bandwidths from a rule-of-thumb procedure suggested by Fan and Gijbels (1996). Sample includes 407,386
observations on patients between the ages of 60 and 70 admitted to California hospitals between January 1, 1992, and November 30, 2002, for unplanned admission through the ED
who have nonmissing Social Security numbers. All coefficients and their SEs have been multiplied by 100.

Downloaded from http://qje.oxfordjournals.org/ at University of Kentucky Libraries on November 29, 2011

 Overview
Part 1: The Core
Stratification
Part 2: Selection on observables
Matching
Part 3: Selection on unobservables
Propensity score matching
Part 4: Advanced material
Regression discontinuity designs
Conclusion
References

Probabilistic treatment assignment (i.e. “fuzzy RDD”)

The probability of receiving treatment changes discontinuously at the cutoff, X0 ,
but need not go from 0 to 1

limXi →X0 Pr (Di = 1|Xi = X0 ) 6= limX0 ←Xi Pr (Di = 1|Xi = X0 )

Examples: Incentives to participate in some program may change discontinuously at

the cutoff but are not powerful enough to move everyone from non participation to
participation.

In the sharp RDD, Di was determined by Xi ≥ X0 ; in the fuzzy RDD, the

conditional probability of treatment jumps at X0 .
The relationship between the probability of treatment and Xi can be written as:

P[Di = 1|Xi ] = g0 (Xi ) + [g1 (Xi ) − g0 (Xi )]Ti

where Ti = 1 if (Xi ≥ X0 ) and 0 otherwise.

Cunningham Causal Inference

 Visualization of Fuzzy RDD identification strategy

Figure: Imbens and Lemieux (2007), figure 3. Horizontal axis is the running variable. Vertical axis is the
conditional probability of treatment at each value of the running variable.
 Visualization of identification strategy (i.e. smoothness)

Conditional expectation of the 2 potential outcomes given X for D = 0, 1

Partly dashed, partly solid → continuous functions of the covariate
Conditional expectation of the observed outcome, Y
Solid → jumps at X = X0

Figure: Potential and observed outcome regressions (Imbens and Lemieux 2007)
 Use the discontinuity as IV

Wald estimator of treatment effect under Fuzzy RDD

Average causal effect of the treatment is the Wald IV parameter

limX →X0 E [Y |X = X0 ] − limX0 ←X E [Y |X = X0 ]

δFuzzy RDD =
limX →X0 E [D|X = X0 ] − limX0 ←X E [D|X = X0 ]

Fuzzy RDD is numerically equivalent and conceptually similar to instrumental

variables
Numerator: “jump” in the regression of the outcome on the running variable, X .
“Reduced form”.
Denominator: “jump” in the regression of the treatment indicator on the running
variable X . “First stage”.
Use software package to estimate (e.g., ivregress 2sls in STATA).
Same IV assumptions, caveats about compliers vs. defiers, and statistical tests
that we discussed with instrumental variables apply here – e.g., check for weak
instruments using F test on instrument in first stage, etc.
 First stage relationship between X and D

One can use both Ti as well as the interaction terms as instruments for Di . If
one uses only Ti as IV, then the it is a “just identified” model which usually
has good finite sample properties.
In the just identified case, the first stage would be:

Di = γ0 + γ1 Xi + γ2 Xi2 + · · · + γp Xip + πTi + ζ1i

where π is the causal effect of T on the conditional probability of treatment.

The fuzzy RD reduced form is:

Yi = µ + κ1 Xi + κ2 Xi2 + · · · + κp Xip + ρπTi + ζ2i

Fuzzy RDD with varying Treatment Effects - Second Stage

As in the sharp RDD case one can allow the smooth function
to be different on both sides of the discontinuity.
The second stage model with interaction terms would be the
same as before:

Yi = α + β01 x̃i + β02 x̃i2 + · · · + β0p x̃ip

+ρDi + β1∗ Di x̃i + β2∗ Di x̃i2 + · · · + βp∗ Di x̃ip + ηi

Where x̃ are now not only normalized with respect to x0 but

are also fitted values obtained from the first stage regression.
Fuzzy RDD with Varying Treatment Effects - First Stages

Again one can use both Ti as well as the interaction terms as

instruments for Di
Only using T the estimated first stages would be:

Di = γ00 + γ01 X̃i + γ02 X̃i2 + · · · + γ0p X̃ip

+πTi + γ1∗ X̃i Ti + γ2∗ X̃i2 Ti + · · · + γp∗ Ti + ζ1i

We would also construct analogous first stages for X̃i Di ,

X̃i2 Di , . . . , X̃ip Di .
 What does Fuzzy RDD Estimate?

As Hahn, Todd and van der Klaauw (2001) point out, one
needs the same assumptions as in the standard IV framework
As with other binary IVs, the fuzzy RDD is estimating LATE:
the average treatment effect for the compliers
In RDD, the compliers are those whose treatment status
changed as we moved the value of xi from just to the left of
x0 to just to the right of x0
 Overview
Part 1: The Core
Stratification
Part 2: Selection on observables
Matching
Part 3: Selection on unobservables
Propensity score matching
Part 4: Advanced material
Regression discontinuity designs
Conclusion
References

Challenges to RDD

Treatment is not as good as randomly assigned around the cutoff, X0 , when

agents are able to manipulate their running variable scores. This happens
when:
1 the assignment rule is known in advance
2 agents are interested in adjusting
3 agents have time to adjust
4 Examples: re-take an exam, self-reported income, etc.
Some other unobservable characteristic changes at the threshold, and this has a
direct effect on the outcome.
In other words, the cutoff is endogenous
Example: Age thresholds used for policy (i.e., person turns 18, and faces more
severe penalties for crime) is correlated with other variables that affect the
outcome (i.e., graduation, voting rights, etc.)
Econometricians and applied social scientists have developed several formalized tests
to evaluate the severity of these problems which we now discuss.

Cunningham Causal Inference

 Test 1: Manipulation of the running variable

Sorting on the running variable (i.e., Manipulation)

Assume a desirable treatment, D, and an assignment rule X ≥ X0 . If individuals
sort into D by choosing X such that X ≥ X0 , then we say individuals are sorting on
the running variable.

Motivating example: Suppose a doctor plans to randomly assign heart patients

to a statin and a placebo to study the effect of the statin on heart attacks
within 10 years. The doctor randomly assigns patients to two different waiting
rooms, A and B, and plans to give those in A the statin and those in B the
placebo. If some of the patents learn of the planned treatment assignment
mechanism, what would we expect to happen? And how would you check for
it?
 McCrary Density Test

We would expect waiting room A to become crowded. In the RDD context, sorting
on the running variable implies heaping on the “good side” of X0
McCrary (2008) suggests a formal test. Under the null the density should be
continuous at the cutoff point. Under the alternative hypothesis, the density
should increase at the kink (where D is viewed as good)
1 Partition the assignment variable into bins and calculate frequencies (i.e., number
of observations) in each bin
2 Treat those frequency counts as dependent variable in a local linear regression
The McCrary Density Test has become mandatory for every analysis using
RDD.
If you can estimate the conditional expectations, you evidently have data on the
running variable. So in principle you can always do a density test
You can download the (no longer supported) STATA ado package, DCdensity, to
implement McCrary’s density test
(http://eml.berkeley.edu/~jmccrary/DCdensity/)
You can install rdd for R too
(http://cran.r-project.org/web/packages/rdd/rdd.pdf)
 Caveats about McCrary Density Test

For RDD to be useful, you already need to know something about the
mechanism generating the assignment variable and how susceptible it could be
to manipulation. Note the rationality of economic
Fig. 1. The agent’s problem. actors that this test is built
on.
A discontinuity
0.50 in the density is “suspicious” 0.50 – it suggests manipulation of X
Conditional Expectation

Conditional Expectation
around the0.30
cutoff is probably going on. 0.30In principle one doesn’t need continuity.
Estimate

Estimate
0.10 0.10
This is a-0.10high-powered test. You need a-0.10lot of observations ` at X0 to distinguish
a discontinuity
-0.30
in the density from noise. -0.30

-0.50 -0.50
5 10 15 20 5 10 15 20
Income Income

0.16 0.16
0.14 0.14
Density Estimate

Density Estimate
0.12 0.12
0.10 0.10
0.08 0.08
0.06 0.06
0.04 0.04
0.02 0.02
0.00 0.00
5 10 15 20 5 10 15 20
Income Income

Fig. 2. Hypothetical example: gaming the system with an income-tested job training program: (A) conditional expectation of returns to
Figure:
treatmentPanel
with noCpre-announcement
is density of income when there
and no manipulation; is no pre-announcement
(B) conditional expectation of returns and no manipulation.
to treatment Panel D is
with pre-announcement
theand
density of income
manipulation; when
(C) density therewith
of income is no
pre-announcement
pre-announcement and and manipulation.
no manipulation; From
(D) density McCrary
of income (2008).
with pre-announcement
and manipulation.

also necessary, and we may characterize those who reduce their labor supply as those with coai pc=f i and
bi 4ai ð1 " f i Þ=d.
Fig. 2 shows the implications of these behavioral effects using a simulated data set on 50,000 agents with
 Visualizing manipulation

Figure 3: Running McCrary z-statistic

Figure 2: Distribution of marathon finishing times (n = 9, 378, 546)

150
100
McCrary z
50
0
-50
2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00 6:30 7:00
threshold

Non-categorical 10 minute threshold

30 minute threshold

NOTE: The McCrary test is run at each minute threshold from 2:40 to 7:00 to test whether there is a significant discontinuity
NOTE: The dark bars highlight the density in the minute bin just prior to each 30 minute threshold. in the density function at that threshold.

Figure: Figures 2 and 3 from Eric Allen, Patricia Dechow, Devin Pope and George Wu’s (2013)
“Reference-Dependent Preferences: Evidence from Marathon Runners”.
http://faculty.chicagobooth.edu/devin.pope/research/pdf/Website_Marathons.pdf

12 14
 More examples of testing for manipulation:
Lee (2008) Incumbency Effect

David Lee’s (2008) “Randomized Experiments from non-random selection in

US House Elections”, Journal of Econometrics analyzes the incumbency effect
using Democratic incumbents for US congressional elections.
A large political science literature on the “incumbency advantage” – having won
an election once helps you win subsequent elections.
Empirical challenge: how to separate incumbency advantage from selection –
(i.e., candidates win multiple elections because they are better)
Identification: Incumbency is assigned to a candidate discontinuously at 50%
voteshare under two-party democracy with majority rule.
Lee (2008) analyzes the probability of winning the election in year t + 1 by
comparing candidates who just won to candidates who just lost the election in
year t.
 Popular Elections
Density test
to the U.S. House of Representatives, 1900-1990:
!"#$%&'()'*&+,-%./"-'0,/&'12.%&'3&4./"5&'/,'6$/,778
A Smooth Density
9,:$4.%';4&-/",<='/,'/2&'>,$=&',7'3&:%&=&</./"5&=?'@ABBC@AAB

%'!
%)#!

%&! %)+!

%)&!
!%&E$&<-F'6,$</

*&<="/F';=/"+./&
$!
%)!!

!)*!
#!
!)#!

!)+!
"!
!)&!

! !)!!
(% (!)* (!)# (!)+ (!)& ! !)& !)+ !)# !)* %
*&+,-%./"-'D.%#"<
,
Figure: Democratic vote share relative to cutoff: popular elections to the
!"#$%&'G)'9&%-&</'0,/"<#'H&.F8
3,44'6.44'0,/&=?'I)1)'>,$=&',7'3&:%&=&</./"5&=?'@JGKCLBB(
House of Representatives, 1900-1990 (McCrary 2008).
 %)&!

!%&E$&<-F'6,$</

*&<="/F';=/"+./&
$! More evidence of manipulation %)!!

!)*!
Contrast this with
#! roll call voting in the US House of Representatives
!)#!

Coordination is expected because these are repeated games, votes !)+! are public
"!
records, and side payments are possible in the form of future votes !)&!
Bills around! the cutoff are more likely to be passed than not. Seems !)!!
like a good
candidate for(% RDD (!)* (!)# (!)+ (!)& ! !)& !)+ !)# !)* %
*&+,-%./"-'D.%#"<
Fails McCrary Density Test; cannot use RDD because policy decisions , are not
quasi-randomly
Roll Call assigned
Votes in thearound cutoff
House, 1857-2008:
!"#$%&'G)'9&%-&</'0,/"<#'H&.F8
A Not Very Smooth Density
3,44'6.44'0,/&=?'I)1)'>,$=&',7'3&:%&=&</./"5&=?'@JGKCLBB(

"!!

&)'!
&'!

&)!!
&!!
!%&E$&<-F'6,$</

*&<="/F';=/"+./&
%)'!
%'!

%!! %)!!

'! !)'!

! !)!!
! !)% !)& !)" !)+ !)' !)# !)- !)* !)$ %
9&%-&</'0,/"<#'"<'!.5,%',7'9%,:,=&M'N"44
,
Figure: Percent voting Yeay: Roll Call Votes, US House of Representatives, 1857-2004
(McCrary 2008)
 Test 2: Balance test on covariates

This is a type of placebo test. For RDD to be valid in your

study, you do not want to observe a discontinuity around the
cutoff, X0 , for average values of covariates that should not be
affected by the treatment – e.g., pretreatment characteristics
Question: What does a jump in the average values of
pre-treatment characteristics have to do with the continuous
(smoothness) assumption?
Choose other pre-treatment covariates, Z and do a similar
graphical plot as you did for Y
You do not want to see a jump around the cutoff, X0
A formal balance test involves the same procedure used to
estimate the treatment effect, only use Z instead of Y as a
LHS variable
Can combine test on multiple covariates into a single test
statistic using seemingly unrelated regression (SUR) or a single
“stacked” regression
 Visualizing Placebo Test

DO VOTERS AFFECT OR ELECT POLICIES? 835

Downloaded from http://qje.oxfordjournals.org/ at Baylor University on

FIGURE III
Similarity
Figure: Figure 3 from of Constituents’
Lee, Moretti Characteristics
and Butler (2004), “Do in Bare Democrat
Voters Affectand Republican
or Elect Policies?” Quarterly
Districts–Part 1
Journal of Economcis.PanelsPanels
refer refer to (top
to (from lefttoto
top left bottom
bottom right)
right) the following
the following district
district characteristics: real
character-
income, percentage with
istics: high-school
real degree,with
income, percentage percentage
high-schoolblack, percentage
degree, percentageeligible to vote. Circles represent
black, percent-
age eligible to vote. Circles represent the average characteristic within intervals
the average characteristic within intervals of 0.01 in Democratic vote share. The continuous line represents
of 0.01 in Democrat vote share. The continuous line represents the predicted
the predicted values from
values a fourth-order
from a fourth-orderpolynomial
polynomial in in vote share fitted
vote share fittedseparately
separatelyforfor points above and
points
abovethreshold.
below the 50 percent and belowThe the dotted
50 percentlinethreshold.
representsThe thedotted line represents
95 percent confidence theinterval.
95
percent confidence interval.
 Test 3: Jumps at non-discontinuous points

Imbens and Lemieux (2008) suggest to look at one side of the

discontinuity (e.g., X < X0 ), take the median value of the
running variable in that section, and pretend it was a
discontinuity, X00
Then test whether in reality there is a discontinuity at X00 .
You do not want to find anything.
Another kind of placebo test. Similar to synthetic control
falsification exercises where you look for non-effects in
non-treatment periods. (See Abadie, Diamond and
Hainmueller 2014)
 Overview
Part 1: The Core
Stratification
Part 2: Selection on observables
Matching
Part 3: Selection on unobservables
Propensity score matching
Part 4: Advanced material
Regression discontinuity designs
Conclusion
References

RDD graphs

A graphical analysis should be an integral part of any RDD.

1 Outcome by running variable, (X ):
i
The standard graph showing the discontinuity in the outcome variable
Construct bins and average the outcome within bins on both sides of the cutoff
You should also look at different bin sizes when constructing these graphs (see
Lee and Lemieux, 2010, for details)
Plot the running variables, Xi , on the horizontal axis and the average of Yi for
each bin on the vertical axis
You may also want to plot a relatively flexible regression line on top of the bin
means
Inspect whether there is a discontinuity at x0
Inspect whether there are other unexpected discontinuities

Cunningham Causal Inference

Example: Outcomes by Forcing Variable
Example: Outcomes by Running Variables
From Lee and Lemieux (2010) based on Lee (2008)

Waldinger (Warwick) 26 / 48
Example: Outcomes by Forcing Variable - Smaller Bins
Example:
From Lee Outcomes
and Lemieux by Running
(2010) based Variables with smaller bins
on Lee (2008)

Waldinger (Warwick) 27 / 48
 More RDD Graphs

2 Probability of treatment by running variable if fuzzy

RDD
In a fuzzy RDD, you also want to see that the treatment
variable jumps at x0
This tells you whether you have a first stage
3 Covariates by a running variable
Construct a similar graph to the one before but using a
covariate as the “outcome”
There should be no jump in other covariates at the
discontinuity, x0 .
If the covariates jump at the discontinuity, one would doubt
the identifying assumption
 Example:
Example Covariates
Covariates by ForcingbyVariable
Running Variable
From Lee and Lemieux (2010) based on Lee (2008)

Waldinger (Warwick) 29 / 48
 More RDD graphs!

4 The density of the running variable

One should plot the number of observations in each bin.
This plot allows to investigate whether there is a discontinuity
in the distribution of the running variable at the threshold
This would suggest that people can manipulate the running
variable around the threshold.
This is an indirect test of the identifying assumption that each
individual has imprecise control over the assignment variable
 Density
Density of the forcingofvariable
the running variable
From Lee & Lemieux (2010) based on Lee (2008)

Waldinger (Warwick) 31 / 48
 “Do Voters Affect or Elect Policies?”
by Lee, Moretti and Butler (2004)

How do voters affect policy? There are two fundamentally different views of the role
of elections in a representative democracy.
1 Convergence: Heterogenous voter ideology forces each candidates to
moderate their positions (e.g., median voter theorem).
“Competition for votes can force even the most partisan Republicans
and Democrats to moderate their policy choices. In the extreme case,
competition may be so strong that it leads to ‘full policy
convergence’: opposing parties are forced to adopt identical policies”
(Lee, Moretti, and Butler 2004; Downs 1957).
2 Divergence: When partisan politicians cannot credibly commit to certain
policies, then convergence is undermined. The result can be full policy
divergence. Divergence is when the winning candidate, after taking office,
simply pursues his most-preferred policy. In this case, voters fail to compel
candidates to reach any kind of policy compromise.
 Simplified model

Candidates: R and D are candidates in a Congressional race. Policy space is

unidimensional where D’s and R’s per-period policy preferences are quadratic
loss functions, u(l) and v (l) and l is the policy variable

D’s bliss point: l ∗ = c(> 0) R’s bliss point: l ∗ = 0

1 1
maxl u(l) = − (l − c)2 maxl v (l) = − l2
2 2
∂u(l) 1 ∂v (l) 1
= 2 × − (l − c) = 0 = − × 2l = 0
∂l 2 ∂l 2
l∗ = c(> 0) l∗ = 0

Voters: Ex ante, voters expect D (R) to choose policy x e (y e ), and expect D

to win with probability P(x e , y e )
∂P ∂P
When x e > y e then ∂x e > 0, ∂y e < 0. Democrats could gain more votes by

moderating the policy position.

P ∗ represents the underlying popularity of D party, or put differently, the
probability D would win if x = c and y = 0.
if D (R) wins, policy x (y ) will be implemented; rational expectations
equilibrium assumed (x = x e ; y = y e ), game repeats in periods t and t + 1
where a period includes the election and the Congressional session.
 Multiple Nash equilibria

1 Partial/Complete Convergence: Voters affect policies

∗ ∗
∂x ∂y
Key result: ∂P ∗ > 0 and ∂P ∗ > 0

If we did a “helicopter drop” of more Democrats in the district

to exogenously increase P ∗ and this resulted in candidates
∂x ∗ ∂y ∗
changing their policy positions, then ∂P ∗ > 0 and ∂P ∗ > 0

2 Complete divergence: Voters elect politicians with fixed

policies
∗ ∗
∂x ∂x
Key result: ∂P ∗ = ∂P ∗ = 0

An exogenous shock to P ∗ (e.g., the helicopter drop of

Democrats) does nothing to equilibrium policies. Voters elect
politicians’ fixed policies.
 Estimation

Potential roll-call voting record outcomes of the representative in the district

following election t is
RCt = Dt xt + (1 − Dt )yt
where Dt is an indicator for whether D won the election. That is, only the
winning candidate’s policy is observed.
This expression can be transformed into regression equations:

RCt = α0 + π0 Pt∗ + π1 Dt + εt (95)

∗
RCt+1 = β0 + π0 Pt+1 + π1 Dt+1 + εt+1 (96)

where α0 and β0 are constants

They parameterized the derivatives from the prior slide as π0 , the coefficient on
the probability, P. See their Appendix 1, p. 849 for details.
This also allows an independent effect of the party, π1
 Estimation (cont.)

Equation 198 cannot be directly estimated because we don’t observe P ∗ . But

suppose we could randomize Dt . Then Dt would be independent of Pt∗ and εt .
Then taking conditional expectations with respect to Dt we get:
∗D ∗R
E [RCt+1 |Dt = 1] − E [RCt+1 |Dt = 0] = π0 [Pt+1 − Pt+1 ]
| {z }
Observable
D
+ π1 [Pt+1 − PR ]
| {z t+1}
Observable
= γ (97)
|{z}
Total effect of initial win on future roll call votes
E [RCt |Dt = 1] − E [RCt |Dt = 0] = π1 (98)
| {z }
Observable
D R
E [Dt+1 |Dt = 1] − E [Dt+1 |Dt = 0] = Pt+1 − Pt+1 (99)
| {z }
Observable
 Why this works

D − P R ] and it’s estimated as the difference

The “elect” component is π1 [Pt+1 t+1
in mean voting records between the parties at time t
The fraction of districts won by Democrats in t + 1 is an estimate of
D − PR ]
[Pt+1 t+1
Because we can estimate the total effect, γ, of a Democrat victory in t on
RCt+1 , we can net out the elect component to implicitly get the “affect”
component
Random assignment of Dt is crucial. Without it, equation 206 would reflect π1
and selection (i.e., that Dem districts have more liberal bliss points).
 Data and RDD Jargon

Two main datasets: liberal voting score from the Americans for Democratic
Action (ADA) linked with House of Representatives election results for
1946-1995
Authors use the ADA score for all US House Representatives from 1946 to 1995
as their voting record index
For each Congress, ADA chooses about twenty high-profile roll-call votes and
creates an index varying 0 and 100 for each Representative of the House. Higher
scores correspond to a more “liberal” voting record.
RDD Jargon
The running variable is voteshare which is the share of all votes that went to a
Democrat. ADA scores are linked to election returns data during that period.
They use exogenous variation in Democratic wins to check whether convergence
or divergence is correct.
Discontinuity in the running variable occurs at voteshare= 0.5. When
voteshare> 0.5, the Democratic candidate wins.
Download STATA do file from my website: http://business.baylor.edu/
scott_cunningham/teaching/causalinf/lmb2004.txt, save it on your
computer, and rename the extension to say lmb2004.do. Open it in your
STATA do-editor.
 Statistical results
832 QUARTERLY JOURNAL OF ECONOMICS

TABLE I
RESULTS BASED ON ADA SCORES—CLOSE ELECTIONS SAMPLE

Total effect Elect component Affect component

! #1 (PD R D R *D *R
t"1 $ Pt"1) #1[(Pt"1 $ Pt"1)] #0[P t "1 $ P t "1]

Variable ADA t"1 ADAt DEMt"1 (col. (2)*(col. (3)) (col. (1)) $ (col. (4))
(1) (2) (3) (4) (5)

Estimated gap 21.2 47.6 0.48

(1.9) (1.3) (0.02)
22.84 $1.64

Downloaded from http://qje.oxfordjournals.org/ at Ba

(2.2) (2.0)

Standard errors are in parentheses. The unit of observation is a district-congressional session. The
sample includes only observations where the Democrat vote share at time t is strictly between 48 percent and
52 percent. The estimated gap is the difference in the average of the relevant variable for observations for
which the Democrat vote share at time t is strictly between 50 percent and 52 percent and observations for
which the Democrat vote share at time t is strictly between 48 percent and 50 percent. Time t and t " 1 refer
to congressional sessions. ADA t is the adjusted ADA voting score. Higher ADA scores correspond to more
liberal roll-call voting records. Sample size is 915.

Figure: Lee, Moretti, and Butler 2004, Table 1.

primarily elect policies (full divergence) rather than affect poli-
cies (partial convergence).
Here we quantify our estimates more precisely. In the analy-
sis that follows, we restrict our attention to “close elections”—
where the Democrat vote share in time t is strictly between 48
and 52 percent. As Figures I and II show, the difference between
. ** Tables - using OLS
. * Table 1: Results Based on ADA Scores
. reg score lagdemocrat if lagdemvoteshare>.48 & lagdemvoteshare<.52, cluster(id)

Linear regression Number of obs = 915

<snip>
------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
lagdemocrat | 21.28387 1.951234 10.91 0.000 17.45445 25.11329
cons | 31.19576 1.333983 23.39 0.000 28.57773 33.81378
------------------------------------------------------------------------------

. reg score democrat if lagdemvoteshare>.48 & lagdemvoteshare<.52, cluster(id)

Linear regression Number of obs = 915

<snip>
------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
democrat | 47.7056 1.356011 35.18 0.000 45.04434 50.36686
cons | 18.7469 .8432428 22.23 0.000 17.09198 20.40182
------------------------------------------------------------------------------

. reg democrat lagdemocrat if lagdemvoteshare>.48 & lagdemvoteshare<.52, cluster(id)

Linear regression Number of obs = 915

<snip>
------------------------------------------------------------------------------
| Robust
democrat | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
lagdemocrat | .4843287 .0289322 16.74 0.000 .4275475 .5411099
cons | .2417582 .0200939 12.03 0.000 .2023227 .2811938
------------------------------------------------------------------------------
 Bandwidth, bias and efficiency

Relaxing bandwidth requirement is about trading off bias and efficiency

. * Table 1 (continued): Examine variations in model specification
. reg score democrat , cluster(id2)

Linear regression Number of obs = 13588

<snip>
(Std. Err. adjusted for 505 clusters in id2)
------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
democrat | 40.76266 1.495659 27.25 0.000 37.82416 43.70115
cons | 17.5756 .8225887 21.37 0.000 15.95947 19.19172
------------------------------------------------------------------------------

. reg score democrat if lagdemvoteshare>.48 & lagdemvoteshare<.52, cluster(id2)

Linear regression Number of obs = 915

<snip>
(Std. Err. adjusted for 250 clusters in id2)
------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
democrat | 47.7056 2.043658 23.34 0.000 43.68054 51.73066
cons | 18.7469 1.38312 13.55 0.000 16.02279 21.47101
------------------------------------------------------------------------------
 Control for the running variable?

. * Center vote share at 0.5 to improve interpretation in interaction models

. gen demvoteshare_c = demvoteshare - 0.5
(11 missing values generated)

. * Next, control for the (centered) running variable as a linear control variable.
. * This is the simplest RDD.
. reg score democrat demvoteshare_c, cluster(id2)

Linear regression Number of obs = 13577

<snip>
(Std. Err. adjusted for 505 clusters in id2)
--------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
democrat | 58.50236 1.555847 37.60 0.000 55.44561 61.5591
demvoteshare_c | -48.93761 4.441693 -11.02 0.000 -57.66412 -40.21109
cons | 11.03413 .9602657 11.49 0.000 9.147518 12.92075
--------------------------------------------------------------------------------

. * Modeling the linearity such that slopes can differ above vs. below discontinuity
. xi: reg score i.democrat*demvoteshare_c, cluster(id2)
i.democrat _Idemocrat_0-1 (naturally coded; _Idemocrat_0 omitted)
i.demo~t*demv~c _IdemXdemvo_# (coded as above)

Linear regression Number of obs = 13577

<snip>
(Std. Err. adjusted for 505 clusters in id2)
--------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
_Idemocrat_1 | 55.43136 1.448568 38.27 0.000 52.58538 58.27734
demvoteshare_c | -5.682785 5.939863 -0.96 0.339 -17.35273 5.987156
_IdemXdemvo_1 | -55.15188 8.236231 -6.70 0.000 -71.33346 -38.97031
cons | 16.81598 .9050024 18.58 0.000 15.03794 18.59403
--------------------------------------------------------------------------------
 Other bandwidths

. * Use +/-0.1 from cutoff (i.e., .4 to .6)

. xi: reg score i.democrat*demvoteshare_c if demvoteshare>.40 & demvoteshare<.60, cluster(id2)
i.democrat _Idemocrat_0-1 (naturally coded; _Idemocrat_0 omitted)
i.demo~t*demv~c _IdemXdemvo_# (coded as above)

Linear regression Number of obs = 4632

<snip>
(Std. Err. adjusted for 428 clusters in id2)
--------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
_Idemocrat_1 | 47.15915 1.810413 26.05 0.000 43.60072 50.71758
demvoteshare_c | 9.421594 18.33431 0.51 0.608 -26.61514 45.45832
_IdemXdemvo_1 | -9.127629 31.8722 -0.29 0.775 -71.77357 53.51831
cons | 17.22656 1.239458 13.90 0.000 14.79036 19.66276
--------------------------------------------------------------------------------

* Use +/- 0.05

. xi: reg score i.democrat*demvoteshare_c if demvoteshare>.45 & demvoteshare<.55, cluster(id2)
i.democrat _Idemocrat_0-1 (naturally coded; _Idemocrat_0 omitted)
i.demo~t*demv~c _IdemXdemvo_# (coded as above)

Linear regression Number of obs = 2387

<snip>
(Std. Err. adjusted for 367 clusters in id2)
--------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
_Idemocrat_1 | 46.77845 2.491464 18.78 0.000 41.87906 51.67783
demvoteshare_c | 54.82604 50.12314 1.09 0.275 -43.73946 153.3915
_IdemXdemvo_1 | -91.1152 81.05893 -1.12 0.262 -250.5149 68.28449
cons | 18.09713 1.693516 10.69 0.000 14.76689 21.42737
--------------------------------------------------------------------------------

* Use +/- 0.01

. xi: reg score i.democrat*demvoteshare_c if demvoteshare>.49 & demvoteshare<.51, cluster(id2)
i.democrat _Idemocrat_0-1 (naturally coded; _Idemocrat_0 omitted)
i.demo~t*demv~c _IdemXdemvo_# (coded as above)

Linear regression Number of obs = 441

<snip>
(Std. Err. adjusted for 160 clusters in id2)
--------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
_Idemocrat_1 | 45.27375 5.813497 7.79 0.000 33.79212 56.75539
demvoteshare_c | 55.75797 716.0422 0.08 0.938 -1358.423 1469.939
_IdemXdemvo_1 | -88.56438 955.1282 -0.09 0.926 -1974.939 1797.81
cons | 20.60166 4.514799 4.56 0.000 11.68495 29.51837
--------------------------------------------------------------------------------
 No voteshare control vs. polynomials
. * Narrow bandwidth, no vote share control (this is comparing means close to cutoff)
. reg score democrat if demvoteshare>.49 & demvoteshare<.51, cluster(id2)

Linear regression Number of obs = 441

F( 1, 159) = 244.90
Prob > F = 0.0000
R-squared = 0.5525
Root MSE = 20.456

(Std. Err. adjusted for 160 clusters in id2)

------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
democrat | 45.3931 2.900628 15.65 0.000 39.66438 51.12183
cons | 20.31812 2.026848 10.02 0.000 16.3151 24.32114
------------------------------------------------------------------------------

. * What about a polynomial of degree 5?

. gen demvoteshare2=demvoteshare^2
(11 missing values generated)

. gen demvoteshare3=demvoteshare^3
(11 missing values generated)

. gen demvoteshare4=demvoteshare^4
(11 missing values generated)

. gen demvoteshare5=demvoteshare^5
(11 missing values generated)

. reg score demvoteshare demvoteshare2 demvoteshare3 demvoteshare4 demvoteshare5 demvoteshare5 democrat, cluster(id2)
note: demvoteshare5 omitted because of collinearity

Linear regression Number of obs = 13577

<snip>
(Std. Err. adjusted for 505 clusters in id2)
-------------------------------------------------------------------------------
| Robust
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
--------------+----------------------------------------------------------------
demvoteshare | -19.30281 71.97002 -0.27 0.789 -160.701 122.0954
demvoteshare2 | 424.0805 543.3939 0.78 0.436 -643.5157 1491.677
demvoteshare3 | -1746.451 1475.699 -1.18 0.237 -4645.731 1152.828
demvoteshare4 | 2559.367 1666.599 1.54 0.125 -714.9694 5833.704
demvoteshare5 | -1248.399 664.662 -1.88 0.061 -2554.249 57.45029
demvoteshare5 | 0 (omitted)
democrat | 49.13039 1.764906 27.84 0.000 45.66291 52.59787
cons | 16.36452 3.07522 5.32 0.000 10.32269 22.40635
-------------------------------------------------------------------------------
 Polynomial with interactions

. * Center the running variable and use polynomials and interactions to model
* the nonlinearities below and above discontinuity
. gen x c=demvoteshare-0.5
(11 missing values generated)

. gen x2 c=x c^2

(11 missing values generated)

. gen x3 c=x c^3

(11 missing values generated)

. gen x4 c=x c^4

(11 missing values generated)

. gen x5 c=x c^5

(11 missing values generated)

.
. reg score i.democrat##(c.x c c.x2 c c.x3 c c.x4 c c.x5 c)

Source | SS df MS Number of obs = 13577

-------------+------------------------------ F( 11, 13565) = 1058.32
Model | 6677033.81 11 607003.073 Prob > F = 0.0000
Residual | 7780253.69 13565 573.553534 R-squared = 0.4618
-------------+------------------------------ Adj R-squared = 0.4614
Total | 14457287.5 13576 1064.91511 Root MSE = 23.949

---------------------------------------------------------------------------------
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
----------------+----------------------------------------------------------------
1.democrat | 47.73325 2.042906 23.37 0.000 43.72887 51.73763
x c | -28.79765 74.09167 -0.39 0.698 -174.0276 116.4323
x2 c | -1138.232 1144.497 -0.99 0.320 -3381.605 1105.141
x3 c | -10681.29 7137.315 -1.50 0.135 -24671.42 3308.839
x4 c | -33490.23 18844.92 -1.78 0.076 -70428.88 3448.424
x5 c | -32873.77 17212.08 -1.91 0.056 -66611.83 864.302
|
democrat#c.x c |
1 | -5.793828 97.79088 -0.06 0.953 -197.4775 185.8899
|
democrat#c.x2 c |
1 | 1768.225 1433.252 1.23 0.217 -1041.149 4577.599
|
democrat#c.x3 c |
1 | 6279.346 8553.95 0.73 0.463 -10487.58 23046.28
|
democrat#c.x4 c |
1 | 47111.44 21834.51 2.16 0.031 4312.767 89910.11
|
democrat#c.x5 c |
1 | 17786.85 19488.24 0.91 0.361 -20412.81 55986.51
|
cons | 17.05847 1.469785 11.61 0.000 14.17749 19.93946
---------------------------------------------------------------------------------
 More polynomial and bandwidth regressions

. * We could limit the regressions to a window using a 2nd degree polynomial

. * interacted with the treatment variable
. reg score i.democrat##(c.x c c.x2 c) if (demvoteshare>0.4 & demvoteshare<0.6)

Source | SS df MS Number of obs = 4632

-------------+------------------------------ F( 5, 4626) = 1153.29
Model | 2622762.02 5 524552.404 Prob > F = 0.0000
Residual | 2104043.2 4626 454.829918 R-squared = 0.5549
-------------+------------------------------ Adj R-squared = 0.5544
Total | 4726805.22 4631 1020.6878 Root MSE = 21.327

---------------------------------------------------------------------------------
score | Coef. Std. Err. t P>|t| [95% Conf. Interval]
----------------+----------------------------------------------------------------
1.democrat | 45.9283 1.892566 24.27 0.000 42.21797 49.63863
xc | 38.63988 60.77525 0.64 0.525 -80.5086 157.7884
x2 c | 295.1723 594.3159 0.50 0.619 -869.9704 1460.315
|
democrat#c.x c |
1 | 6.507415 88.51418 0.07 0.941 -167.0226 180.0374
|
democrat#c.x2 c |
1 | -744.0247 862.0435 -0.86 0.388 -2434.041 945.9916
|
cons | 17.71198 1.310861 13.51 0.000 15.14207 20.28189
---------------------------------------------------------------------------------
 Nonparametric estimation

Hahn, Todd and Van der Klaauw (2001) clarified assumptions about RDD (i.e.,
continuity in conditional expectation regression functions)
Also framed estimation as a non-parametric problem and emphasized using
local polynomial regressions
Nonparametric methods mean a lot of different things to different people in
statistics.
In RDD context, the idea is to estimate a model that doesn’t assume a
functional form for the relationship between Y (outcome variable) and X
(running variable)
That model would be something general like

Y = f (X ) + ε

A very basic method is to calculate E [Y ] for each bin on X – like a histogram.

STATA has an option to do this called cmogram and it has a lot of useful
options. We can recreate Figures I, IIA and IIB using it
 Graphical estimate of equation 205
828 QUARTERLY JOURNAL OF ECONOMICS

Downloaded from http://qje.oxfordjournals.org/ at Baylor University on April 7, 2014

FIGURE I
Total Effect of Initial Win on Future ADA Scores: "
This figure plots ADA scores after the election at time t ! 1 against the
Democrat vote share, time t. Each circle is the average ADA score within 0.01
intervals of the Democrat vote share. Solid lines are fitted values from fourth-
order polynomial regressions on either side of the discontinuity. Dotted lines are
pointwise 95 percent confidence intervals. The discontinuity gap estimates
" ! # 0$P *t !1
D
" P *t !1
R
% # # 1$P *t !1
D
" P *t !1
R
%.
“Affect” “Elect”

Figure:
be a Lee, Moretti,
continuous andfunction
and smooth Butler 2004,
of vote sharesFigure I. γ ≈ 20
everywhere,
except at the threshold that determines party membership. There
is a large discontinuous jump in ADA scores at the 50 percent
 Nonparametric estimation with cmogram

. cmogram score lagdemvoteshare , cut(0.5) scatter line(0.5) qfitci

Plotting mean of score, conditional on lagdemvoteshare.

n = 13577

Bin #1: [0,.0135101361854656] (n = 181) (mean = 18.90983436977007)

Bin #2: (.0135101361854656,.0270202723709312] (n = 0) (mean = .)
Bin #3: (.0270202723709312,.0405304085563968] (n = 0) (mean = .)
Bin #4: (.0405304085563968,.0540405447418624] (n = 0) (mean = .)
Bin #5: (.0540405447418624,.067550680927328] (n = 0) (mean = .)
Bin #6: (.067550680927328,.0810608171127936] (n = 2) (mean = 11.2350001335144)
<snip>
Bin #36: (.9605263157894751,.9736842105263173] (n = 2) (mean = 96.58000183105469)
Bin #37: (.9736842105263173,.9868421052631594] (n = 0) (mean = .)
Bin #38: (.9868421052631594,1] (n = 1646) (mean = 33.014483642933)
 Linear and lowess fits

. cmogram score lagdemvoteshare , cut(0.5) scatter line(0.5) lfit

Plotting mean of score, conditional on lagdemvoteshare.

. cmogram score lagdemvoteshare , cut(0.5) scatter line(0.5) lowess

Plotting mean of score, conditional on lagdemvoteshare.

 Graphical estimate of equation 206
830 QUARTERLY JOURNAL OF ECONOMICS

Downloaded from http://qje.oxfordjournals.org/ at Baylor U

FIGURE IIa
Effect of Party Affiliation: #1

Figure: Lee, Moretti, and Butler 2004, Figure IIa. π1 ≈ 45

* Lowess
cmogram score demvoteshare, cut(.5) scatter line(.5) lowess

Figure: Cubic feet with confidence intervals

 ://qje.oxfordjournals.org/ at Baylor University on April 7, 2014
Graphical estimate of equation 207
FIGURE IIa
Effect of Party Affiliation: #1

FIGURE IIb
Effect of Initial Win on Winning Next Election: (P D R
t!1 " P t!1 )
Top panel plots ADA scores after the election at time t against the Democrat
vote share, time t. Bottom panel plots probability of Democrat victory at t ! 1
against Democrat vote share, time t. See caption of Figure III for more
D details.R
Figure: Lee, Moretti, and Butler 2004, Figure IIb. (Pt+1 − Pt+1 ) ≈ 0.50
 Kernel weighted local polynomial regression

Hahn, Todd and Van der Klaauw (2001) showed that the one-sided kernel
estimation (such as lowess) may have poor properties because the point of
interest is at a boundary (i.e., the discontinuity), called the “boundary problem”
They proposed to use “local linear nonparametric regressions” instead
STATA’s poly estimates kernel-weighted local polynomial regressions. Think of
it as a weighted regression restricted to a window like we’ve been doing (hence
“local”) where the kernel provides the weights
A rectangular kernel would give the same result as E [Y ] at a given bin on X .
The triangular kernel gives more importance to observations close to the center.
This method will be sensitive to how large the bandwidth (window) you choose
. * Note kernel-weighted local polynomial regression is a smoothing method.
. lpoly score demvoteshare if democrat == 0, nograph kernel(triangle) gen(x0 sdem0) ///
> bwidth(0.1)

. lpoly score demvoteshare if democrat == 1, nograph kernel(triangle) gen(x1 sdem1) ///

> bwidth(0.1)

. scatter sdem1 x1, color(red) msize(small) || scatter sdem0 x0, msize(small) color(red) ///
> xline(0.5,lstyle(dot)) legend(off) xtitle("Democratic vote share") ytitle("ADA score")

. * Next, let’s get the treatment effect at the cutoff where demvoteshare=0.5
. capture drop sdem0 sdem1
. gen forat=0.5 in 1
. lpoly score demvoteshare if democrat==0, nograph kernel(triangle) gen(sdem0) at(forat) bwidth(0.1)
. lpoly score demvoteshare if democrat==1, nograph kernel(triangle) gen(sdem1) at(forat) bwidth(0.1)
. gen late=sdem1 - sdem0
. list sdem1 sdem0 late in 1/1
+----------------------------------+
| sdem1 sdem0 late |
|----------------------------------|
1. | 64.395204 16.908821 47.48639 |
+----------------------------------+
. * What happens when we change the bandwidth? Use 0.01, 0.05, 0.2, 0.3, 0.4
. capture drop smoothdem0* smoothdem1* x0* x1*
. local co 0
. foreach i in 0.01 0.05 0.1 0.20 0.30 0.40
2. local co = ‘co’ +1
3. lpoly score demvoteshare if democrat == 0, nograph kernel(triangle) gen(x0‘co’ smoothdem0‘co’) ///
> bwidth(‘i’)
4. lpoly score demvoteshare if democrat == 1, nograph kernel(triangle) gen(x1‘co’ smoothdem1‘co’) ///
> bwidth(‘i’)
5.

. line smoothdem01 x01, msize(small) color(gray) sort || line smoothdem11 x11, sort color(gray) || ///
> line smoothdem02 x02, color(black) sort || line smoothdem12 x12, sort color(black) || ///
> line smoothdem03 x03, color(red) sort || line smoothdem13 x13, sort color(red) || ///
> line smoothdem04 x04, color(blue) sort || line smoothdem14 x14, sort color(blue) || ///
> line smoothdem05 x05, color(green)sort || line smoothdem15 x15, sort color(green)|| ///
> line smoothdem06 x06, color(orange) sort || line smoothdem16 x16, sort color(orange) ///
> xline(0.5,lstyle(dot)) legend(off) xtitle("Democratic vote share") ytitle("ADA score") ///
> title("Bandwidths: 0.01, 0.05, 0.1, 0.2, 0.3, 0.4")
Several methods for choosing the optimal bandwidth (window), but it’s always
a trade off between bias and variance
In practical applications, you want to check for balance around that window
Standard error of the treatment effects can be bootstrapped but there are also
other alternatives
You could add other variables to nonparametric methods.
Calonico, Cattaneao and Titiunik (2013b) propose local-polynomial regression
discontinuity estimators with robust confidence intervals
STATA ado package and R package are both called rdrobust
. rdrobust score demvoteshare, c(0.5) all bwselect(IK)
Preparing data.
Computing Bandwidth Selectors.
Computing Variance-Covariance Matrix.
Computing RD Estimates.
Estimation Completed.

Sharp RD Estimates using Local Polynomial Regression.

Cutoff c = .5 | Left of c Right of c Number of obs = 13577

----------------------+---------------------- NN Matches = 3
Number of obs | 3535 3318 BW Type = IK
Order Loc. Poly. (p) | 1 1 Kernel Type = Triangular
Order Bias (q) | 2 2
BW Loc. Poly. (h) | 0.152 0.152
BW Bias (b) | 0.191 0.191
rho (h/b) | 0.795 0.795

Outcome: score. Running Variable: demvoteshare.

--------------------------------------------------------------------------------------
Method | Coef. Std. Err. z P>|z| [95% Conf. Interval]
----------------------+---------------------------------------------------------------
Conventional | 47.171 .98058 48.1046 0.000 45.2488 49.0926
Robust | - - 36.4839 0.000 44.0965 49.1034
--------------------------------------------------------------------------------------

All Estimates.
--------------------------------------------------------------------------------------
Method | Coef. Std. Err. z P>|z| [95% Conf. Interval]
----------------------+---------------------------------------------------------------
Conventional | 47.171 .98058 48.1046 0.000 45.2488 49.0926
Bias-Corrected | 46.6 .98058 47.5226 0.000 44.678 48.5219
Robust | 46.6 1.2773 36.4839 0.000 44.0965 49.1034
--------------------------------------------------------------------------------------
 McCrary Density Test

. * McCrary (2008) density test to check for manipulation of the running variable (DCdensity)
. DCdensity demvoteshare_c if (demvoteshare_c>-0.5 & demvoteshare_c<0.5), breakpoint(0) generate(Xj Yj r0 fhat se_fhat)
Using default bin size calculation, bin size = .003047982
Using default bandwidth calculation, bandwidth = .104944836

Discontinuity estimate (log difference in height): .011195629

(.061618519)
Performing LLR smoothing.
296 iterations will be performed
 “Pioneers take the arrows in the back”
– Jim West during a faculty meeting

Lee (2008) is important for the econometric theory, but this

particular application turned out to be wrong:
“Contrary to the assumptions of RD, we show that bare winners and bare
losers in US House elections (1942-2008) differ markedly on pretreatment
ovariates. Bare winners possess large ex ante financial, experience, and
incumbency advantages over their opponents and are usually the
candidates predicted to win by Congressional Quarterly—’s pre-election
ratings. Covariate imbalance actually worsens in the closest House
election. National partisan tides help explain these patterns. We present
evidence suggesting that sorting in close House elections is due mainly to
activities on or before Election Day rather than post election recounts or
other manipulation. The sorting is so strong that it is impossible to
achieve covariate balance between matched treated and control
observations, making covariate adjustment a dubious enterprise. Although
RD is problematic for postwar House elections, this example does
highlight the design’s advantages over alternatives: RD’s assumptions are
clear and weaker than model-based alternatives, and their implications are
empirically testable.” (Caughey and Sekhon, 2011).
 Regression Kink Design

Card, Lee, Pei and Weber (2012) introduces a variant of the

RDD which they call regression kink design (RKD)
They essentially use a kink in some policy rule to identify the
causal effect of the policy
Instead of a jump in the outcome you now expect a jump in
the first derivative
 Unemployment benefits in Austria

They apply their design to answer the question whether the

level of unemployment benefits affects the length of
unemployment in Austria
Unemployment benefits are based on income in a base period
The benefit formula for unemployment exhibits 2 kinks
There is a minimum benefit level that isn’t binding for people
with low earnings
Then benefits are 55% of the earnings in the base period
There is a maximum benefit level that is adjusted every year
People with dependents get small supplements (which is the
reason why one can distinguish five “solid” lines in the
following graph)
Not everyone receives benefits that correspond one to one to
the formula because of mistakes in the administrative data
 Base Year Earnings and Unemployment Benefits
Base Year Earnings and Unemployment Benefits

The graph shows unemployment benefits (vertical axis) as a function of

pre-unemployment earnings (horizontal axis).
The Waldinger
graph(Warwick)
shows unemployment benefits (vertical axis)46 as
/ 48 a

function of pre-unemployment earnings (horizontal axis)

Base Year Earnings and Benefits for Single Individuals
Base Year Earnings and Benefits for Single Individuals

Bin-Size: 100 Euros

For single individuals UI benefits are flat below the cuto§. The
Bin-size: 100is euros
relationship still upward sloping because of family benefits.
ForWaldinger
single individuals UI benefits are flat below the cutoff.
(Warwick) 47 / 48
The
relationship is still upward sloping because of family benefits.
 Time to Next Job for Single Individuals
Time to Next Job For Single Individuals

People with higher base earnings have less trouble finding a job
(negative slope).
People with higher base earnings have less trouble finding
There is a kink: the relationship becomes shallower once benefits
a
job (negative
increase more.slope)
Waldinger (Warwick) 48 / 48
There is a kink: the relationship becomes shallower once
benefits increase more.
 Selection on unobservables

Recall the identifying assumption when estimating causal

effects using a conditioning strategy

Independence: (Y 0 , Y 1 ) ⊥
⊥D

Use methods like propensity score matching and condition on

p(x), or linear regression
What if treatment assignment to units is based on some
unobserved variable, u, which is correlated with the outcome
Conditioning strategies are invalid
Selection on unobservable methods, though, may work
Natural experiments may be useful
 Natural experiments

Natural experiments are neither an estimator or an experiment

(natural or otherwise). Rather, it is simply an event that
occurs naturally which causes exogenous variation in some
treatment variable of interest
An attempt to find in the world some rare circumstance such
that a consequential treatment was handed to some people
and denied to others haphazardly
“The word natural has various connotations, but a natural
experiment is a wild experiment not a wholesome experiment;
nature in the way that a tiger is natural, not in the way that
oatmeal is natural.” (Rosenbaum 2005)
Haphazard variation in treatment is not like the variation
induced by a randomized experiment
 Natural experiments

Naive matching – “people who look comparable are

comparable”. False.
Matching can create matched samples of units that look
similar on some x
Matching cannot create a matched sample who are similar on
u (unobservables), though
Definition of naive: someone who believes something because
it’s convenient to believe it.
 Broad Street Pump

John Snow is believed to be the bastard son of Edward Stark

but is actually a Targaryan
But before that John Snow, there was another John Snow – a
practicing anesthesiologist in 19th century London
This Snow is considered the father of epidemiology
He is famous for providing convincing evidence that cholera
was a waterborne disease
He lived from 1813-1858
 19th century science

Microscopes were around but had horrible resolution

Most human pathogens couldn’t be seen
Isolating these microorganisms wouldn’t occur for half a
century
Two views of what caused disease:
Majority view - “Miasmas”. Minute, inanimate poison particles
in the air
Minority view - “infection theory”.
 Cholera background

Cholera arrives in the early 1800s and exhibits “epidemic

waves”.
Cholera attacked victims suddenly, was usually fatal, and
symptoms included vomiting and acute diarrhea
Snow observed the clinical course of the disease and made the
following conjecture
The active agent was a living organism that entered the body,
got into the alimentary canal with food or drink, multiplied in
the body, and generated some poison that caused the body to
expel water
The organism passed out of the body with these evacuations,
entered the water supply and infected new victims
The process repeated itself, growing rapidly through the
common water supply, causing an epidemic
 History

There were three main epidemics in London

1831-1832
1848-1849 (>15,000 deaths)
1853-1854 (>30,000 deaths)
 Background

Snow is an early advocate for the infection theory – cholera is

being spread person-to-person through some unknown
mechanism
His earlier evidence was based on years of observations
including
Cholera transmission tended to follow human commerce
A sailor on a ship from a cholera-free country who arrived at a
cholera-stricken port would only get sick after landing or
taking on supplies
Cholera hit the poor communities the worst, who also lived in
the most crowded housing with the worst hygeine
Facts were difficult to reconcile with the miasma theory, but
are consistent with infection theory
 More evidence

Snow identifies Patient Zero: the first case of an early

epidemic
“a seaman named John Harnold, who had arrived by the Elbe
steamer from Hamburg, where the disease was prevailing”
The second case? John Harnold’s roommate
 More evidence

Snow studied two apartment buildings. The first was heavily

hit with cholera but the second wasn’t.
He found the water supply in the first building was
contaminated by runoff from privies but the water supply in
the second was cleaner
Earlier water supply studies
In the London of the 1800s, there were many different water
companies serving different areas of the city
Some were served by more than one company
Several took their water from the Thames, which was heavily
polluted by sewage
The service areas of such companies had much higher rates of
cholera
The Chelsea water company was an exception, but it had an
exceptionally good filtration system
 Broad Street Water Pump

1849: Lambeth water company moves the intake point

upstream along the Thames, above the main sewage discharge
point (i.e., purer water)
1849: Southwark and Vauxhall water company left their
intake point downstream from where the sewage discharged
(gross) (i.e., infected water)
Comparisons of data on cholera deaths from the 1853-1854
year showed the epidemic hit the Southwark and Vauxhall
services areas harder, and largely spared the Lambeth areas
 Snow’s Table IX

No. houses Cholera deaths Deaths / 10,000 houses

Southwark and Vauxhall 40,046 1,263 315
Lambeth 26,107 98 37
Rest of London 256,423 1,422 59
 Shoe leather research

“As a piece of statistical technology, [Snow’s Table IX] is

by no means remarkable. But the story it tells is very
persuasive. The force of the argument results from the
clarity of the prior reasoning, the bringing together of
many different lines of evidence, and the amount of shoe
leather Snow was willing to use to get the data.”
“Snow did some brilliant detective work on
nonexperimental data. What is impressive is not the
statistical technique but the handling of the scientific
issues. He made steady progress from shrewd observation
through case studies to analyze ecological data. In the
end, he found and analyzed a natural experiment.”
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Introduction to instrumental variables

Pearl argues that there are three ways to estimate causal effects: the backdoor
criterion (“selection on observables”), instrumental variables, and the front
door criterion.
Now we move into the instrumental variables research design
Illustrious history – discovered by Philip Wright and published as an appendix
in his 1928 book
One of the most powerful methods for identifying causal effects in the social
sciences

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Sometimes, the researcher can find the flag. That is, the researcher knows of a
variable (Z ) that actually is randomly assigned and that affects fertility
decisions. Such a variable is called an “instrument”.
Example: Angrist and Evans (1998), “Children and their parents’ labor supply”
American Economic Review,
Z is a dummy variable indicating whether the first two children born were of the
same gender
Many parents have a preference for having at least one child of each gender
Consider a couple whose first two kids were both boys; they will often have a
third, hoping to have a girl
Consider a couple whose first two kids were girls; they will often have a third,
hoping for a boy
Consider a couple with one boy and one girls; they will often not have a third kid
The gender if your kids is arguably randomly assigned (maybe not exactly, but
close enough)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

“No causation without manipulation” (Holland, 1985). If you want to use IV,
then ask:

What moves around the covariate of interest that might be plausibly viewed as random?

“Focusing on the selection process”

How was the covariate of interest selected?
Do researchers select randomly, as in drawing balls from an urn?
Or do subjects choose the covariate partially based on random factors and
partially based on non-random factors?
If the random factors underlying the decision can be observed by the researcher,
then you can use IV to estimate the effect of the covariate of interest on the
outcome
Angrist and Evans (1988) example:
Families with at least two kids are the subject population
The covariate of interest is the number of kids (numkids).
There are several outcomes of interest measuring labor market decisions of the
couple, such as whether the mother worked for pay in the last year (workforpay).
Once you have identified such a variable, begin to think about what data sets
might have information on an outcome of interest, the covariate of interest,
and the instrument you have put your finger on.
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

In a pinch, you can even get by with two different data sets,
one of which has information on the outcome and the
instrument, and the other of which has information on the
covariate of interest and the instrument.
This is known as “Two sample IV” because there are two
samples involved, rather than the traditional one sample.
Once we define what IV is measuring carefully, you will see
why this works.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Instrumental variables strategies formalize untainted inference, which is the

inference drawn by an intelligent layperson with no particular training or
background in statistics.
Example of what I mean by “untainted inference”:
The researchers tell a layperson that the gender of a woman’s first two children is
predictive of her labor force attachment. In particular, women whose first two
children are of the same gender work substantially less than women whose first
two children are of different genders
On its face, this is a puzzling fact – without further information, it is hard to see
why the gender of your children would be so predictive of labor market
participation
The researchers additionally point out that women whose first two children are of
the same gender are more likely to have additional children than women whose
first two children are of different genders
The layperson then wonders whether the labor market differences are due solely
to the differences in the number of kids the woman has

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Traditional and Contemporary IV Pedagogy

We want to learn the IV framework in two iterations:

1 Constant treatment effects (i.e., β is constant across all individual units)

Constant treatment effects is the traditional econometric pedagogy when first

learning instrumental variables, and is not based explicitly on the potential
outcomes model or notation
Constant treatment effects is identical to assuming that ATE=ATT=ATU
because constant treatment effects assumes βi = β−i = β for all units
2 Heterogeneous treatment effects (i.e., βi varies across individual units)
This is the “modern IV pedagogy”, and you may not have learned it in
econometrics if only because potential outcomes is not ordinarily the basis of
most first year econometrics sequences
Heterogeneous treatment effects means that the ATE 6= ATT 6= ATU because βi
differs across the population
This is equivalent to assuming the coefficient, βi , is a random variable that varies
across the population
Heterogenous treatment effects is based on work by Angrist, Imbens and Rubin
(1996) and Imbens and Angrist (1994) which introduced the “local average
treatment effect” (LATE) concept
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

When should you think of using instrumental variables?

Instrumental variables methods are typically used to address

the following kinds of problems encountered in OLS
regressions:
1 Omitted variable bias
2 Measurement error
3 Simultaneity bias, or “reverse causality”

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Omitted Variable Bias (Angrist and Pischke, 2009)

Labor economists have been studying the returns to schooling a long time –
typically some version of a “Mincer regression”:

Yi = α + ρSi + γAi + νi

Yi = log of earnings
Si = schooling measured in years
Ai = individual ability

Typically the econometrician cannot observe Ai ; for instance, the CPS tells us
nothing about adult respondents’ family background, intelligence, or
motivation.
What are the consequences of leaving ability out of the regression? Suppose
you estimated this short regression instead:

Yi = α + ρSi + ηi

where ηi = γAi + νi ; α, ρ, and γ are population regression coefficients; Si is

correlated with ηi through Ai only; and vi is a regression residual uncorrelated
with all regressors by definition.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Derivation of Ability Bias

Suppressing the i subscripts, the OLS estimator for ρ is:

Cov (Y , S) E [YS] − E [Y ]E [S]

ρb = =
Var (S) Var (S)

Plugging in the true model for Y , we get:

Cov [(α + ρS + γA + ν), S]

ρb =
Var (S)
E [(αS + S 2 ρ + γSA + νS)] − E (S)E [α + ρS + Aγ + ν]
=
Var (S)
ρE (S 2 ) − ρE (S)2 + γE (AS) − γE (S)E (A) + E (νS) − E (S)E (ν)
=
Var (S)
Cov (AS)
= ρ+γ
Var (S)

If γ > 0 and Cov(A, S)> 0 the coefficient on schooling in the shortened

regression (without controlling for A) would be upward biased

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

How can IV be used to obtain unbiased estimates?

Suppose there exists a variable, Zi , that is correlated with Si .

We can estimate ρ with this variable, Z :

Cov (Y , Z ) = Cov (α + ρS + γA + ν, Z )
= E [(α + ρS + γA + ν), Z ] − E [α + ρS + γA + ν]E [Z ]
= {αE (Z ) − αE (Z )} + ρ{E (SZ ) − E (S)E (Z )}
+γ{E (AZ ) − E (A)E (Z )} + E (νZ ) − E (ν)E (Z )
Cov (Y , Z ) = ρCov (S, Z ) + γCov (A, Z ) + Cov (ν, Z )

What conditions must hold for a valid instrumental variable?

Cov (S, Z ) 6= 0 – “first stage” exists. S and Z are correlated
Cov (A, Z ) = Cov (ν, Z ) = 0 – “exclusion restriction”. Z is orthogonal to the
factors in η, such as unobserved ability or the structural disturbance term, η
Assuming the first stage exists and that the exclusion restriction holds, then we
can estimate ρ with ρIV :

Cov (Y , Z )
ρIV = =ρ
Cov (S, Z )

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

IV is Consistent if IV Assumptions are Satisfied

The IV estimator is consistent if the IV assumptions are satisfied. Substitute

true model for Y :
Cov ([α + ρS + γA + ν], Z )
ρIV =
Cov (S, Z )
Cov ([S], Z ) Cov ([A], Z ) Cov ([ν], Z )
= ρ +γ +
Cov (S, Z ) Cov (S, Z ) Cov (S, Z )
Cov (η, Z )
= ρ+γ
Cov (S, Z )

Taking the plim:

plim ρbIV = ρ
because Cov ([A], Z ) = 0 and Cov ([ν], Z ) = 0 due to the exclusion restriction,
and Cov (S, Z ) 6= 0 (due to the first stage)
But, if Z is not independent of η (either correlated with A or ν), and if the
correlation between S and Z is “weak”, then the second term blows up. We
will return to this later when we discuss the problem of “weak instruments”.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

In which DAG is Z a valid instrument for D?

Z e

D Y
(a)

Z e

D Y
(b)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Reviewing some of the IV Jargon

Causal model. Sometimes called the structural model:

Yi = α + ρSi + ηi

First-stage regression. Gets the name because of two-stage least squares:

Si = α + ρZi + ζi

Second-stage regression. Notice the fitted values, S:

b

Yi = α + ρSbi + νi

Reduced form. Notice this is just a regression of Y onto the instrument:

Yi = α + πZi + εi

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Two-stages least squares

Suppose you have a sample of data on Y , X , and Z . For each observation i we

assume the data are generated according to

Yi = α + βXi + εi
Xi = γ + δZi + νi

where Cov (Z , ε) = 0 and δ 6= 0.

Plug in covariance, and using the result that ni=1 (xi − x) = 0, write out the
P
IV estimator:
1 Pn 1 Pn
Cov (Z , Y ) (Zi − Z )(Yi − Y ) i=1 (Zi − Z )Yi
β
dIV = = n1 Pi=1
n = n
1 Pn
Cov (Z , X ) n i=1 (Zi − Z )(Xi − X ) n i=1 (Zi − Z )Xi

Substitute the causal model definition of Y to get:

1 Pn
n i=1 (Zi − Z ){α + βX + ε}
β IV =
d
1 Pn
n i=1 (Zi − Z )Xi
1
(Zi − Z )εi
= β+ Pnn
1
i=1 (Zi − Z )Xi
n
= β + ”small if n is large”

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Two-stages least squares

Note βIV is ratio of “reduced form” (π) to “first stage” coefficient (δ):
Cov (Z ,Y )
Cov (Z , Y ) Var (Z ) π
βbIV = = =
b
Cov (Z , X ) Cov (Z ,X )
Var (Z )
δb

Rewrite δb as
Cov (Z , X )
δb = ⇔ Cov (Z , X ) = δVar
b (Z ) (100)
Var (Z )
Then rewrite βIV

Cov (Z , Y ) δCov
b (Z , Y ) δCov
b (Z , Y )
βIV = = =
d
Cov (Z , X ) δCov (Z , X )
b 2
δ Var (Z )
b
Cov (δZ
b ,Y)
= (101)
Var (δZ
b )

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Two-stage least squares

Cov (δZ
b ,Y )
Recall X = γ + δZ + ν; β
dIV = and let Xb = γ
b + δZ
b .
Var (δZ
b )
Then the two-stage least squares (2SLS) estimator is

Cov (δZ
b ,Y) Cov (Xb , Y )
βIV = =
d
Var (δZ
b ) Var (Xb )

Proof.
We will show that δCov
b (Y , Z ) = Cov (Xb , Y ). I will leave it to you to show that
Var (δZ
b ) = Var (Xb )

Cov (Xb , Y ) = E [Xb Y ] − E [Xb ]E [Y ]

= E (Y [b b ]) − E (Y )E (b
γ + δZ γ + δZ
b )
= γ b (YZ ) − γ
bE (Y ) + δE bE (Y ) − δE
b (Y )E (Z )
b (YZ ) − E (Y )E (Z )]
= δ[E
Cov (Xb , Y ) = δCov
b (Y , Z )

The 2SLS estimator replaces X with the fitted values of X (i.e., Xb ) from the
first stage regression of X onto Z and all other covariates.

Cunningham Causal Inference

 Intuition of 2SLS

I’ve said that learning about instrumental variables through

the “intuition” of two-stage least squares is valuable, but
what do I mean exactly?
2SLS emphasizes the use of the fitted values of the
endogenous regressor – what has that transformation done?
By using the fitted values of the endogenous regressor from
the first stage regression, our regression now uses only the
exogenous variation in the regressor due to the instrumental
variable itself
We recover exogenous variation in other words
. . . but think about it – that variation was there before, but
was just a subset of all the variation in the regressor
Instrumental variables therefore reduces the variation in the
data, but that variation which is left is exogenous
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Implementing 2SLS in STATA

In a sample of data, you could get the reduced form and first stage coefficients
manually by the following two regression commands in STATA:

. reg Y Z

. reg X Z
While it is always a good idea to run these two regressions, don’t compute your
IV estimate this way
Example: It is often the case that a pattern of missing data will differ between Y
and X ; in such a case, the usual procedure of “casewise deletion” is to keep the
subsample with non-missing data on Y , X , and Z .
But the reduced form and first stage regressions would be estimated off of
different sub-samples if you used the two step method above
The standard errors from the second stage regression are also wrong
Best practice is to use your built-in procedure (which also gives standard
errors):
. ivregress Y (X=Z)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Implementing 2SLS in STATA

You can also estimate 2SLS using the auxiliary regression approach we just
covered:

. reg X Z
. predict Xhat
. reg Y Xhat

For the same reasons that you shouldn’t actually implement 2SLS manually
using the ratio of the reduced form and first stage coefficients, you shouldn’t
manually use the auxiliary regression approach because, again, the standard
errors are incorrect, and any complex missing patterns may leave you with
different samples
This “two stage least squares” interpretation of IV – called an interpretation,
because it is not the actual suggested procedure – is useful for understanding
what IV does, but stick with ivregress

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Instrument for Education using Compulsory Schooling Laws

Instrument foris Education

In practice, it often difficult using
to find Compulsory Schooling
convincing instruments Laws
– usually because
potential instruments don’t satisfy the exclusion restriction
In In
thepractice
returns ittoiseducation literature,
often di¢cult to findAngrist and Krueger
convincing (1991)
instruments (in had a very
influential
particularstudy where
because theypotential
many used quarter of not
IVs do birthsatisfy
as an the
instrumental
exclusion variable
forrestriction).
schooling
In In
thethe
US,returns
you could drop out literature
to education of school Angrist
once you turned
and 16 (1991) had
Krueger
“School
a very districts
influentialtypically requirethey
study where a student to haveofturned
used quarter ageansix by January 1
birth as
of instrumental
the year in which he or
variable forshe enters school” (Angrist and Krueger 1991, p.
schooling.
980)
In the US you could drop out of school once you turned 16.
Children
Childrenhave
havedifferent ages
di§erent ageswhen they
when start
they school,
start schoolthough,
and thusanddi§erent
this creates
different
lengthslengths of schooling
of schooling at the at thethey
time timeturn
they16turn
when16 they
(potential
can drop out age):
potentially drop out.

Waldinger (Warwick) 9 / 45

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

First Stage
First Stages
Men born earlier in the year have lower schooling. This indicates that there is a
first
Menstage.
born earlier in the year have lower schooling. This indicates that
there is a first stage.

Waldinger (Warwick) 10 / 45
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Reduced Form
Reduced Form
Do differences in schooling due to different quarter of birth translate into
different earnings?in schooling due to di§erent quarter of birth translate
Do di§erences
into di§erent earnings?

Waldinger (Warwick)
Cunningham Causal Inference 11 / 45
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Two Stage Least Squares model

The first stage regression is:

Si = X π10 + π11 Zi + η1i

The reduced form regression is:

Yi = X π20 + π21 Zi + η2i

π21
The covariate adjusted IV estimator is the sample analog of the ratio, π11
Again, how was this estimator calculated?
1 Obtain the first stage fitted values:

Sbi = X π
b10 + π
b11 Zi

where π
b1j for j = 1, 2 are OLS estimates of the first stage regression
2 Plug the first stage fitted values into the “second-stage equation” to then
estimate
Yi = αX + Sρb + error

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Two Stage Least Squares

But as we note, they don’t actually manually do this – I remind you of this
because the 2SLS intuition is very useful. 2SLS only retains the variation in S
generated by the quasi-experimental variation, which we hope is exogenous
Angrist and Krueger use more than one instrumental variable to instrument for
schooling: they include a dummy for each quarter of birth. Their estimated
first-stage regression is therefore:

Si = X π10 + Z1i π11 + Z2i π12 + Z3i π13 + η1

The second stage is the same as before, but the fitted values are from the new
first stage

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

First Stage Regressions in Angrist

First stage &results
regression Krueger (1991)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

First stage regression results

First Stage
Quarter Regressions
of birth in Angrist
is a strong predictor of total&years
Krueger (1991)
of education

Quarter of birth is a strong predictor of total years of education.

Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

First stage regression results: Placebos

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

IV Results
IV Estimates
IV Estimates Birth Birth
Cohorts 20-29, 1980 Cohorts
Census 20-29, 1980 Census

Waldinger (Warwick) 17 / 45
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

IV Results - IV
Including Some
Estimates Covariates
- including some covariates
IV Estimates Birth Cohorts 20-29, 1980 Census

Waldinger (Warwick) 18 / 45
Cunningham Causal Inference
 Wald estimator

They also present an alternative to 2SLS called the Wald

estimator – both are versions of instrumental variables
Recall that 2SLS uses the predicted values from a first stage
regression – but we showed that the 2SLS method was
(Y ,Z )
equivalent to Cov
Cov (X ,Z )
The Wald estimator simply calculates the return to education
as the ratio of the difference in earnings by quarter of birth to
the difference in years of education by quarter of birth – it’s a
version of the above
E (Y |Z =1)−E (Y |Z =0)
Formally, IVWald = E (D|Z =1)−E (D|Z =0)
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References
IV Results - Including More Covariates and Interacting
Quarter of Birth
IV Estimates - more covariates and interacting quarter of birth

They
They alsoalso include
include specifications
specifications where where they
they use use 30 (quarter
30 (quarter of birth ×ofyear)
birth x
dummy
year)variables
dummies and 150150
and (quarter of birth
(quarter × state)
of birth dummies
x state) as instrumental
dummies as IVs
variables
(intuition: the e§ect of quarter of birth may vary by birth year or
What’s the intuition here? The effect of quarter of birth may vary by birth year or
state).
by state
This reduces standard errors.
It reduced the standard errors, but that comes at a cost of potentially having a
weakButinstruments
also comes at the cost of potentially having a weak instruments
problem
problem (see below).

Waldinger (Warwick) 19 / 45

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

plain

Mechanism

In addition to log weekly wage, they examined the impact of

compulsory schooling on log annual salary and weeks worked
The main impact of compulsory schooling is on the log weekly
wage – not on weeks worked

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Weak Instruments

As we mentioned earlier, IV is consistent but biased

For a long time, researchers were not attentive to this subtle difference and
didn’t care much about the small sample bias of IV
But in the early 1990s, a number of papers highlighted that IV can be severely
biased – in particular, when instruments have only a weak correlation with the
endogenous variable of interest and when many instruments are used to
instrument for one endogenous variable (i.e., there are many overidentifying
restrictions).
In the worst case, if the instruments are so weak that there is no first stage,
then the 2SLS sampling distribution is centered on the probability limit of OLS

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Weak instruments and bias towards OLS

Let’s consider a model with a single endogenous regressor and a simple

constant treatment effect
The causal model of interest is:

y = βx + ν

The matrix of instrumental variables is Z with the first stage equation:

x = Z’π + η

If νi and ηi are correlated, estimating the first equation by OLS would lead to
biased results, wherein the OLS bias is:
Cov (ν, x)
E [βOLS − β] =
Var (x)
σνη
If νi and ηi are correlated the OLS bias is therefore: σx2

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Weak instruments and bias towards OLS

It can be shown that the bias of 2SLS is approximately:

\ σνη 1
E [β2SLS − β] ≈
ση2 F + 1

where F is the population analogue of the F -statistic for the joint significance
of the instruments in the first stage regression. See Angrist and Pischke pp.
206-208 for a derivation.
σ
If the first stage is weak (i.e, F → 0), then the bias of 2SLS approaches σνη 2
η
This is the same as the OLS bias as for π = 0 in the second equation on the
earlier slide (i.e., there is no first stage relationship between Z and D) σx2 = ση2
σ σ
and therefore the OLS bias σνη2 becomes σνη2 .
x x
But if the first stage is very strong (F → ∞) then the IV bias goes to 0.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Weak Instruments - Adding More Instruments

Adding more weak instruments will increase the bias of 2SLS

By adding further instruments without predictive power, the first stage F -statistic
goes toward zero and the bias increases
If the model is “just identified” – mean the same number of instrumental
variables as there are endogenous covariates – weak instrument bias is less of a
problem
See Angrist and Pischke, p. 209 where they write that IV is “approximately
biased” – this is only true if the first stage is not zero
See http:
//econ.lse.ac.uk/staff/spischke/mhe/josh/solon_justid_April14.pdf
Bound, Jaeger and Baker (1995) highlighted this problem for the Angrist and
Krueger study. AK present findings from using different sets of instruments
1 Quarter of birth dummies → 3 instruments
2 Quarter of birth dummies + (quarter of birth) × (year of birth) + (quarter of
birth) × (state of birth) → 180 instruments

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Adding Instruments in Angrist & Krueger

Adding
Table from Bound, Jaeger, instruments in -Angrist
and Baker (1995) 3 and 30 and
IVs Krueger

Adding more weak instruments reduced the first stage F-statistic and
Adding more weak instruments reduced the first stage F -statistic and moves
moves the coe¢cient towards the OLS coe¢cient.
the coefficient towards the OLS coefficient
Waldinger (Warwick) 41 / 45
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Adding Instruments in Angrist

Adding instruments & Krueger
in Angrist and Krueger
Table from Bound, Jaeger, and Baker (1995) - 180 IVs

Adding more weak instruments reduced the first stage F-statistic and
Adding more weak instruments reduced the first stage F -statistic and moves
moves the coe¢cient towards the OLS coe¢cient.
the coefficient towards the OLS coefficient
Waldinger (Warwick) 42 / 45

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

What can you do if you have weak instruments?

WIth weak instruments, you have the following options:

1 Use a just identified model with your strongest IV
2 Use a limited information maximum likelihood estimator (LIML). This is
approximately median unbiased for over identified constant effects models. It
provides the same asymptotic distribution as 2SLS (under constant effects) but
provides a finite-sample bias reduction. (LIML is programmed for STATA in the
ivregress command.)
3 Find stronger instruments.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Practical Tips for IV Papers

1 Report the first stage

Does it make sense?
Do the coefficients have the right magnitude and sign?
2 Report the F -statistic on the excluded instrument(s).
Stock, Wright and Yogo (2002) suggest that F -statistics > 10 indicate that you
do not have a weak instrument problem – this is not a proof, but more like a rule
of thumb
If you have more than one endogenous regressor for which you want to
instrument, reporting the first stage F -statistic is not enough (because 1
instrument could affect both endogenous variables and the other could have no
effect – the model would be under identified). In that case, you want to report
the Cragg-Donald EV statistic.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Practical Tips for IV Papers

3 If you have many IVs, pick your best instrument and report the just identified
model (weak instrument problem is much less problematic)
4 Check over identified 2SLS models with LIML
5 Look at the reduced form
The reduced form is estimated with OLS and is therefore unbiased
If you can’t see the causal relationship of interest in the reduced form, it is
probably not there

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Angrist (1990) Veteran Draft Lottery

Angrist (1990) uses the Vietnam draft lottery as an instrumental variable for
military service
In the 1960s and 1970s, young American men were drafted for military service
to serve in Vietnam
Concerns about the fairness of the conscription policy led to the introduction of
a draft lottery in 1970
From 1970 to 1972, random sequence numbers were randomly assigned to each
birth date in cohorts of 19-year-olds
Men with lottery numbers below a cutoff were drafted; in other words
Higher numbers were less likely to be drafted;
Lower numbers were more likely to be drafted.
The draft did not perfectly determine military service:
Many draft-eligible men were exempt for health and other reasons
Exempt men would sometimes volunteer

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Summary of Findings on Vietnam Draft Lottery

1 First stage results: Having a low lottery number (i.e., being

eligible for the draft) increased veteran status by about 16
percentage points (the mean of veteran status was 27 percent)
2 Second stage results: Serving in the army lowers earnings by
between $2,050 and $2,741 per year

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

IV with Heterogenous Treatment Effects

Up to this point, we only considered models where the causal effect was the
same for all individuals (i.e., homogenous treatment effects where Yi1 − Yi0 = δ
for all i units)
Let’s now try to understand what instrumental variables estimation is
measuring if treatment effects are heterogenous (i.e, Yi1 − Yi0 = δi which varies
across the population)
Why do we care?
1 We care about internal validity: Does the design successfully uncover causal
effects for the population that we are studying?
2 We care about external validity: Does the study’s results inform us about
different populations?

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

IV with Heterogenous Treatment Effects

Similar potential outcomes notation and terminology:

Causal chain is Zi → Di → Yi
Yi (Di = 0, Zi = 1) is represented as Yi (0, 1)
“Potential treatment status” versus “observed treatment status
Di1 = i’s treatment status when Zi = 1
Di0 = i’s treatment status when Zi = 0
Observed treatment status switching equation:

Di = Di0 + (Di1 − Di0 )Zi

= π0 + π1 Zi + ζi

π0i = E [Di0 ]
π1i = (Di1 − Di0 ) is the heterogenous causal effect of the IV on Di .
E [π1i ] = The average causal effect of Zi on Di

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Identifying assumptions under heterogenous treatment effects

1 Stable Unit Treatment Value Assumption (SUTVA)

2 Random Assignment
3 Exclusion Restriction
4 Nonzero First Stage
5 Monotonicity

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Stable Unit Treatment Value Assumption (SUTVA)

Stable Unit Treatment Value Assumption (SUTVA)

If Zi = Zi0 , then Di (Z) = Di (Z0 )
If Zi = Zi0 and Di = Di0 , then Yi (D,Z) = Yi (D’,Z’)

Interpretation Potential outcomes for each person i are unrelated to the treatment
status of other individuals.
Example Veteran status of person at risk of being drafted is not affected by the
draft status of others at risk of being drafted.
Implication Rewrite Yi (D,Z) as Yi (Di , Zi ) and Di (Z) as Di (Zi ).

Definition 1: Intention-to-treat effects

Causal effect of Z on D is Di1 − Di0
Causal effect of Z on Y is Yi (Di1 , 1) − Yi (Di0 , 0)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Independence assumption

Independence assumption (e.g., “as good as random assignment”)

{Yi (Di1 , 1), Yi (Di0 , 0), Di1 , D 0i } ⊥
⊥ Zi

Interpretation The IV is independent of the vector of potential outcomes and

potential treatment assignments (i.e. “as good as randomly assigned”)
The independence assumption is sufficient for a causal interpretation of the reduced
form:

E [Yi |Zi = 1] − E [Yi |Zi = 0] = E [Yi (Di1 , 1)|Zi = 1] − E [Yi (Di0 , 0)|Zi = 0]
= E [Yi (Di1 , 1)] − E [Yi (Di0 , 0)]

Independence means that the first stage measures the causal effect of Zi on Di :

E [Di |Zi = 1] − E [Di |Zi = 0] = E [Di1 |Zi = 1] − E [Di0 |Zi = 0]

= E [Di1 − Di0 ]

Example Vietnam conscription for military service was based on randomly generated
draft lottery numbers. The assignment of draft lottery number was independent of
potential earnings or potential military service – as good as random.
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Exclusion Restriction
Exclusion Restriction
Y(D,Z) = Y(D,Z’) for all Z, Z’, and for all D

Interpretation Any effect of Z on Y must be via the effect of Z on D. In other

words, Yi (Di , Zi ) is a function of D only. Or formally:

Yi (Di , 0) = Yi (Di , 1) for D = 0, 1

Example In the Vietnam draft lottery example, an individual’s earnings potential as

a veteran or a non-veteran are assumed to be the same regardless of draft eligibility
status. The exclusion restriction would be violated if low lottery numbers may have
affected schooling (e.g., to avoid the draft). If this was the case, the lottery number
would be correlated with earnings for at least two cases:
1 through its effect on military service
2 through its effect on educational attainment
Implication Random lottery numbers (independence) does not imply that the
exclusion restriction is satisfied
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Exclusion restriction

Use the exclusion restriction to define potential outcomes indexed solely against
treatment status:

Yi1 = Yi (1, 1) = Yi (1, 0)

Yi0 = Yi (0, 1) = Yi (0, 0)

Rewrite the switching equation:

Yi = Yi (0, Zi ) + [Yi (1, Zi ) − Yi (0, Zi )]Di

Yi = Yi0 + [Yi1 − Yi0 ]Di

Random coefficients notation for this is:

Yi = α0 + ρi Di
with α0 = E [Yi0 ] and ρi = Yi1 − Yi0

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

First stage

Nonzero Average Causal Effect of Z on D

E [Di1 − Di0 ] 6= 0

Interpretation Z has to have some statistically significant effect on the average

probability of treatment
Example Having a low lottery number increases the average probability of service.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Monotonicity

Monotonicity
Either π1i ≥ 0 for all i or π1i ≤ 0 for all i = 1, . . . , N

Interpretation Recall that π + 1i is the reduced form causal effect of the

instrumental variable on an individual i’s treatment status. Monotonicity requires
that the instrumental variable (weakly) operate in the same direction on all
individual units. In other words, while the instrument may have no effect on some
people, all those who are affected are affected in the same direction (i.e., positively
or negatively, but not both).
Draft example While draft eligibility may have had no effect on the probability of
military service for some, monotonicity means the draft lottery either shifted people
into service, or it shifted people out of service – but it did not do both.
Schooling example In the quarter of birth example for schooling, this assumption
may not be satisfied (see Barua and Lang 2009). Being born in the 4th quarter
(which typically increases schooling) may have reduced schooling for some because
their school enrollment was held back by their parents
Implication Without monotonicity, IV estimators are not guaranteed to estimate a
weighted average of the underlying causal effects of the affected group, Yi1 − Yi0 .
Cunningham Causal Inference
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Local average treatment effect

If all 1-5 assumptions are satisfied, then IV estimates the local average treatment
effect (LATE) of D on Y :

Effect of Z on Y
δIV ,LATE =
Effect of Z on D

Instrumental variables (IV) estimand:

E [Yi (Di1 , 1) − Yi (Di0 , 0)]

δIV ,LATE =
E [Di1 − Di0 ]
= E [(Yi1 − Yi0 )|Di1 − Di0 = 1]

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Local Average Treatment Effect

The LATE parameters is the average causal effect of D on Y for those whose
treatment status was changed by the instrument, Z
Vietnam draft lottery example: IV estimates the average effect of military
service on earnings for the subpopulation who enrolled in military service
because of the draft but would not have served otherwise.
In other words, LATE would not tell us what the causal effect of military service
was for volunteers or those who were exempted from military service for medical
reasons
We have reviewed the properties of IV with heterogenous treatment effects
using a very simple dummy endogenous variable, dummy IV, and no additional
controls example.
The intuition of LATE generalizes to most cases where we have continuous
endogenous variables and instruments, and additional control variables.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

More IV Jargon!

The LATE framework partitions any population with an instrument into

potentially 4 groups:
1 Compliers: The subpopulation with Di1 = 1 and Di0 = 0. Their treatment status
is affected by the instrument in the “correct direction”.
2 Always takers: The subpopulation with Di1 = Di0 = 1. They always take the
treatment independently of Z .
3 Never takers: The subpopulation with Di1 = Di0 = 0. They never take the
treatment independently of Z .
4 Defiers: The subpopulation with Di1 = 0 and Di0 = 1. Their treatment status is
affected by the instrument in the “wrong direction”.
(You’ll notice that increasingly our terms are borrowing from the medical
literature where the treatment is taking a pill. Same here.)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Never-Takers Complier
Di1 − Di0 = 0 Di1 − Di0 = 1
Yi (0, 1) − Yi (0, 0) = 0 Yi (1, 1) − Yi (0, 0) = Yi (1) − Yi (0)
By Exclusion Restriction, causal effect of Z Average Treatment Effect among
on Y is zero. Compliers

Defier Always-taker
Di1 − Di0 = −1 Di1 − Di0 = 0
Yi (0, 1) − Yi (1, 0) = Yi (0) − Yi (1) Yi (1, 1) − Yi (1, 0) = 0
By Monotonicity, no one in this group By Exclusion Restriction, causal effect of Z
on Y is zero.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Monotonicity Ensures that there are no defiers

Monotonicity ensures that there are no defiers

Why is it important to not have defiers?
If there were defiers, effects on compliers could be (partly) canceled out by
opposite effects on defiers
One could then observe a reduced form which is close to zero even though
treatment effects are positive for everyone (but the compliers are pushed in one
direction by the instrument and the defiers in the other direction)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

What Does IV (Not) Estimate?

As outlined above, with all 5 assumptions satisfied, IV estimates the average

treatment effect for compliers
Contrast this with the traditional IV pedagogy with constant treatment effects
(i.e., δi = δ for all i units).
Question: What does IV estimate when treatment effects are assumed to be
constant?
Without further assumptions (e.g., constant causal effects), LATE is not
informative about effects on never-takers or always-takers because the
instrument does not affect their treatment status
So what? Well, it matters because in most applications, we would be mostly
interested in estimating the average treatment effect on the whole population:

ATE = E [Yi1 − Yi0 ]

But that’s not possible usually with IV

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Sensitivity to assumptions: exclusion restriction

Example Someone at risk of draft (low lottery number) changes education plans to
retain draft deferments and avoid conscription.
Implication Increased bias to IV estimand through two channels:
Average direct effect of Z on Y for compliers
Average direct effect of Z on Y for noncompliers multiplied by odds of being a
non-complier
Severity Depends on:
Odds of noncompliance (smaller → less bias)
“Strength” of instrument (stronger → less bias)
Effect of the alternative channel on Y

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Sensitivity to assumptions: Monotonicity violations

Example Someone who would have volunteered for Army when not at risk of draft
(high lottery number) chooses to avoid military service when at risk of being drafted
(low lottery number)
Implication Bias to IV estimand (multiplication of 2 terms):
Proportion defiers relative to compliers
Difference in average causal effects of D on Y for compliers and defiers
Severity Depends on:
Proportion of defiers (small → less bias)
“Strength” of instrument (stronger → less bias)
Variation in effect of D on Y (less → less bias)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Summarizing

The potential outcomes framework gives a more subtle interpretation of what

IV is measuring
In the constant coefficients world (i.e., traditional pedagogy), IV measures δ which
is “the” causal effect of Di on Yi , and assumed to be the same for all i units
In the random coefficients world, IV measures instead an average of heterogeneous
causal effects across a particular population – E [δi ] for some group of i units
IV, therefore, measures the local average treatment effect or LATE parameter,
which is the average of causal effects across the subpopulation of compliers, or
those units whose covariate of interest, Di , is influenced by the instrument.
Angrist and Evans (1996) example: not every woman whose first two kids
share gender will go on to have a third kid.
Under heterogenous treatment effects, Angrist and Evans (1996) identify the
causal effect of the gender composition of the first two kids on labor supply
Remark: That is not the same thing as identifying the causal effect of children on
labor supply; the former is a LATE whereas the latter might be better described
as an ATE
Ex post this is probably obvious, but ex ante this was a real breakthrough (see
Angrist, Imbens and Rubin 1996; Imbens and Angrist 1994)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Other potentially interesting treatment effects

Another effect which we may be potentially interested in estimating is the

familiar estimand, the average treatment effect on the treatment group (ATT)
Remark: LATE is not the same as ATT, though.

E [Yi1 − Yi0 |Di = 1] = E [Yi1 − Yi0 |Di0 = 1] P[Di0 = 1|Di = 1]

| {z } | {z }
ATT Effect on always takers

+ E [Yi1 − Yi0 |Di1 > Di0 ] P[Di1 > Di0 , Zi = 1|Di = 1]

| {z }
Effect on compliers

The average treatment effect on the treated, ATT, is a weighted average of the
effects on always-takers and compliers.
If there are no always takers we can, however, estimate ATT which is equal to
LATE in that case.

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Discussions and questions

When might we not be interested in the local average treatment effect?

Romneycare in Massachusetts examples: If the compliance rate or treatment
effects differ in the community than during some quasi-experimental expansion,
are we more interested in LATE, ATT or ATE?
What might be other examples of this in economics? In education?
This has a similar flavor to methodological questions regarding a study’s
“external” vs. “internal validity”
What do we make of the fact that LATE is defined for an unobservable
sub-population (i.e., can’t label all units in the population as compliers or
noncompliers)?
What do we make of the fact that IV identification is based on a set of
untestable assumptions?
For example: colonial settler mortality (Z ) influences economic development (Y )
only through Z ’s association with human capital accumulation rather than
institutions, D (Acemoglu, Johnson and Robinson, 2001).

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

IV in Randomized Trials

The use of IV methods may be helpful when evaluating a randomized trial

In many randomized trials, participation is nonetheless voluntary among those
randomly assigned to treatment
On the other hand, persons in the control group usually do not have access to
treatment
only those who are particularly likely to benefit from treatment therefore will
probably take up treatment which almost always leads to positive selection bias
if you just compare means between treated and untreated individuals using OLS,
you will obtain biased treatment effects even for the randomized trial due
non-compliance
Solution: instrument for treatment with whether you were offered treatment
and estimate LATE

Cunningham Causal Inference

Dean Eckles (Facebok), Rene Kizilcec and Eytan Bakshy
(2015). “Identifying Peer Effects in Social Networks with
Peer Encouragement Designs”
 Peer encouragement designs

Dean Eckles, et al. (2015) randomly assign vertices in

Facebook network to encouragement to behavior of interest
and examine how this spills over to others
Indirectly affect behaviors of existing peers
Eckles peer encouragement design allows them to estimate
the effect of peer behaviors itself through instrumental
variables – not just the encouragement itself
An alternative view: assign peers to behaviors, see effect it
has on the person.
 Effects of receiving feedback

Motivation – when an individual shares content in social

media, what are the effects of receiving additional feedback
like comments and “likes”?
Generating further conversation – ego replies to comments
In-kind peer effects in giving feedback (reciprocity)
Creating and sharing more content in the future
How do these effects vary by prior ego behaviors?
How can minor user interface changes affect user experiences
and engagement? How should we make these tradeoffs?
 Goals

Goal 1: Estimate peer effects

How does a marginal peer adopting affect your adoption?
Ideal experiment: directly assign behaviors of existing peers
Goal 2: Estimate effects of global treatment
What would happen if we gave everyone the treatment?
Average treatment effect (ATE) of global treatment Z1 vs. Z0

1 X
δ(z1 , z0 ) = E [Yi (Z = z1 ) − Yi (Z = z0 )]
N
i

Ideal experiment: assign connected components to treatment

This is an active area of research (Eckles, Karrer, Ugander
2014; Ugander, et al 2013)
 What are “Encouragement designs”?

Randomly assign units to encouragement Z to a focal

behavior D
Examples: randomly encourage students to study (Powers and
Swinton 1984)
Assign to take a drug or not (but they may not take it)
Formal analysis using potential outcomes model
Total effect of encouragement: intent-to-treat (ITT)

Yi (Zi = 1) − Yi (Zi = 0)

Effect of behavior (i.e., effect of D on Y )

Yi (Di = 1) − Yi (Di = 0)

Can we use Z to estimate the effect of D on Y ?

 Encouragement designs

The best instrumental variables are randomly assigned

Find a variable Z that affects D but is otherwise unrelated to
Y
Use this exogenous variation in D to estimate the effect of D
on Y

Z e

D Y
 IV analysis

Number of encouraged peers is an instrumental variable for

peer effects
Complete mediation (i.e., “exclusion restriction”) – Peer
encouragement only affects ego behavior via peer behavior
Even if effects are heterogenous, IV analysis of encouragement
designs identifies the LATE for the complier subpopulation
Eckles, et al (2015) argue that this is likely an advantage over
other instruments that aren’t encouragements
 So what exactly is the instrument??

Facebook claim as fact: Pre-expanding vs. not pre-expanding

Fact: Pre-expanding
comment vs.feedback
boxes (Z ) increases not pre-expanding
(D)
comment boxes modulates feedback

Expanded comment box Unexpanded comment box

Lower interaction rate

Higher interaction rate "

 Binary encouragement designs

Four types of people by potential outcomes

Compliers - treatment if encouraged, control if not
Always-takers - treatment whether encouraged or not
Never-takers - control whether encouraged or not
Defiers - control if encouraged, treatment if not encouraged
Not all of these may exist in a particular study
In a trial of a new drug or offering (removing) a new
(existing) feature, there are neither always-takers nor defiers
 Binary encouragement designs

D=0 D=1
Z=0 Compliers and Never-takers Defiers and Always-takers
Z=1 Defiers and Never-takers Compliers and Always-takers
 Heterogenous treatment effects

0
Monotonicity: With probability 1, Diz ≥ Diz for all z ≥ z 0 and
all i
Then local average treatment effect (LATE) is identified
In binary Z , D case, LATE is the average treatment effect for
the population of compliers
We always have to ask ourselves: are we interested in the
LATE?
Peer encouragement designs with a single behavior of
interest

Assign to encouragement to:

enroll in a retirement savings account (Duflo and Saez 2003)
post a thankful status update on Thanksgiving Day
Summarize peer assignments (e.g., number of peers assigned)
Summarize peer behaviors (e.g., number of adopter peers)
Compute average ego behaviors as a function of these
 Peer encouragement with dyads

Peer encouragement designs with dyads

1 Randomly encourage j or not
Binary encouragement, peer behavior, and ego outcome
2 Observe j’s behavior
1.  Randomly (endogenous
encourage j or not" treatment for i)
3 Observe i’s behavior
2.  Observe j’s behavior(outcome)
(endogenous treatment for i)"
3.  Observe i’s behavior (outcome)"
Z-i
Z-i
Zj Wj Yi Zj Wj Yi
Zj Wj Yi Zj Wj Yi
complier peer:

Zj Wj Yi Zj Wj Yi
Zj Wj Yi Zj Wj Yi

always-taker peer:

All variables represented by circles may have other common causes not shown. Variables represented by squares are root nodes."
Peer encouragements for dyads
Peer encouragements for dyads

Non-
Non-

Z W Z-i W-i
 Peer encouragements for dyads

Peer encouragement designs

Peer encouragement with multiple peers Non-compliance

With multiple peers

Z-i W-i Z-i W-i Z-i

Yi Yi

Encourage all peers Encourage no peers Encour

All variables represented by circles may have other common causes not shown. Variables represented by squares are root nodes."
 Peer encouragement designs
Noncompliance with multiple peers
Noncompliance with multiple peers
▪  An ego’s peer may be a mix of compliance types"

Z-i W-i Z-i W-i

Yi Always taker

Never taker

Yi Never taker

Complier

Complier

Non-compliance

Z-i W-i Z-i W-i

 Reduced form: Intent-to-treat (ITT)

One option is forego the IV analysis and instead estimate the

ITT
In other words, analyze ego outcome Y as a function of the
number of peers assigned Z
What else?
Compute probabilities of assignment, Pr (Z )
Use inverse probability weighting to estimate average outcomes
 Statistical inference for spillovers

Other issue is statistical inference.

Briefly: statistical inference in a network application needs to
account for network autocorrelation in regressors (i.e.,
regressors are correlated within a network)
Number of peers treated has high auto-correlation
“Randomization inference” – increasingly used. Possible to do
exact testing for spillovers using new versions of conditional
randomization inference (Athey, Barrios, Eckles and Imbens
2015)
 Instrumental variables analysis

Ego’s outcome caused by own assignment D and peer

behaviors Y
The number of adopter peers is what matters – how many
peers will be “encouraged”
Eckles, et al (2015) suggest two-stage least squares or other
IV methods (e.g., LIML)
See Bramoulle et al (2009) “Identification of Peer Effects
through Social Networks”, Journal of Econometrics for exact
formulation of using “peers of peers IV”
Exclusion restriction requires that peer of peer behavior or
covariates only affect ego behavior through peer behavior
 Results

Unfortunately, empirical results are currently not being

circulated from this study so this is from memory
First stage: the effect of encouragement on feedback was very
large. Receiving expanding boxes resulted in peers leaving
more comments.
Effects on behavior Y : the IV analysis showed that peers
influence our behavior – we are more likely to leave new
comments, reciprocal comments, likes, and even post new
updates
These effects were very large – in both stages
 Lottery designs

Another design is to use a randomized lottery as an

instrumental variable for some treatment
Examples might be randomized lottery for attending charter
schools to study effect of charter schools on educational
outcomes
We’ll discuss two papers from 2012 and 2014 evaluating a
lottery-based expansion of Medicaid health insurance on
Oregon on numerous health and financial outcomes
 Overarching question

What are the effects of expanding access to public health

insurance for low income adults?
Magnitudes, and even the signs, associated with that question
were uncertain
Limited existing evidence
Institute of Medicine review of evidence was suggestive, but a
lot of uncertainty
Observational studies are confounded by selection into health
insurance
Quasi-experimental work often focuses on elderly and children
Only one randomized experiment in a developed country: the
RAND health insurance experiment
1970s experiment on a general population
Randomized cost-sharing, not coverage itself
 The Oregon Health Insurance Experiment

Setting: Oregon Health Plan Standard

Oregon’s Medicaid expansion program for poor adults
Eligibility
Poor (<100% federal poverty line) adults 19-64
Not eligible for other programs
Uninsured > 6 months
Legal residents
Comprehensive coverage (no dental or vision)
Minimum cost-sharing
Similar to other states in payments, management
Closed to new enrollment in 2004
 The Oregon Medicaid Experiment

Lottery
Waiver to operate lottery
5-week sign-up period, heavy advertising (January to February
2008)
Low barriers to sign up, no eligibility pre-screening
Limited information on list
Randomly drew 30,000 out of 85,000 on list (March-October
2008)
Those selected given chance to apply
Treatment at household level
Had to return application within 45 days
60% applied; 50% of those deemed eligible → 10,000 enrollees
 Oregon Health Insurance Experiment

Evaluate effects of Medicaid using lottery as randomized

controlled trial (RCT)
Intent-to-treat: Reduced form comparison of outcomes
between treatment group (lottery selected individuals) and
controls (not selected)
LATE: IV using lottery as instrument for insurance coverage
First stage: about a 25 percentage point increase in insurance
coverage
Archived analysis plan
Massive data collect effort – primary and secondary
Similar to ACA expansion but limits to generalizability
Partial equilibrium vs. General equilibrium
Mandate and external validity
Oregon vs. other states
Short vs. Long-run
 Examine Broad Range of Outcomes

Costs: Health care utilization

Insurance increases resources (income) and lowers price,
increasing utilization
But improved efficiency (and improved health), decreasing
utilization (“offset”)
Additional uncertainty when comparing Medicaid to no
insurance
Benefits I: Financial risk exposure
Insurance supposed to smooth consumption
But for very low income, is most care de jure or de facto free?
Benefits II: Health
Expected to improve (via increased quantity / quality of care)
But could discourage health investments (“ex ante moral
hazard”)
 Data

Pre-randomization demographic information

From lottery sign-up
State administrative records on Medicaid enrollment
Primary measure of first stage (i.e., insurance coverage)
Outcomes
Administrative data (∼16 months post-notification): Hospital
discharge data, mortality, credit reports
Mail surveys (∼15 months): some questions ask 6-month
look-back; some ask current
In-person survey and measurements (∼25 months): Detailed
questionnaires, blood samples, blood pressure, body mass index
Study Population

10
 Empirical Framework

Reduced form – estimates of the causal effect of lottery

selection

Yihj = β0 + β1 LOTTERYh + Xih β2 + Vih β3 + εihj

Validity of experimental design: randomization; balance on

treatment and control
Instrumental variables – effect of insurance coverage

INSURANCEihj = δ0 + δ1 LOTTERYih + Xih δ2 + Vih δ3 + µihj

yihj = π0 + π1 INSURANCEih + Xih π2 + Vih π3 + vihj

Effect of lottery on coverage: about 25 percentage points

Additional assumption for causality: primary pathway
Could affect participation in other programs, but actually small
“Warm glow” of winning – especially early
Analysis plan, multiple inference adjustment
 Effect of lottery on coverage (first stage)
Effects of Lottery on Coverage (1st Stage)
Full sample Credit subsample Survey respondents
Control Estimated Control Estimated Control Estimated
mean FS mean FS mean FS
Ever on Medicaid 0.141 0.256 0.135 0.255 0.135 0.290
(0.004) (0.004) (0.007)
Ever on OHP Standard 0.027 0.264 0.028 0.264 0.026 0.302
(0.003) (0.004) (0.005)
# of Months on Medicaid 1.408 3.355 1.352 3.366 1.509 3.943
(0.045) (0.055) -0.09
On Medicaid, end of study period 0.106 0.148 0.101 0.151 0.105 0.189
(0.003) (0.004) (0.006)
Currently have any insurance (self report) 0.325 0.179
(0.008)
Currenty have private ins. (self report) 0.128 -0.008
(0.005)
Currently on Medicaid (self report) 0.117 0.197
(0.006)
Currently on Medicaid 0.093 0.177
(0.006)

12
 Amy Finkelstein, et al. (2012). “The Oregon Health
Insurance Experiment: Evidence from the First Year”,
Quarterly Journal of Economics, vol. 127, issue 3, August.
 Effects of Medicaid

Use primary and secondary data to gauge 1-year effects

Mail surveys: 70,000 surveys at baseline, 12 months
Administrative data
Medicaid enrollment records
Statewide Hospital discharge data, 2007-2010
Credit report data, 2007-2010
Mortality data, 2007-2010
 Mail survey data

Fielding protocol
∼70,000 people, surveyed at baseline and 12 months later
Basic protocol: three-stage male survey protocol,
English/Spanish
Intensive protocol on a 30% subsample included additional
tracking, mailings, phone attempts (done to adjust for
non-response bias)
Response rate
Effective response rate = 50%
Non-response bias aways possible, but response rate and
pre-randomization measures in administrative data were
balanced between treatment and control
 Administrative data

Medicaid records
Pre-randomization demographics from list
Enrollment records to assess “first stage” (how many of the
selected got insurance coverage)
Hospital discharge data
Probabilistically matched to list, de-identified at Oregon
Health Plan
Includes dates and source of admissions, diagnoses,
procedures, length of stay, hospital identifier
Includes years before and after randomization
Other data
Mortality data from Oregon death records
Credit report data, probabilistically matched, de-identified
 Sample

89,824 unique individuals on the waiting list

Sample exclusions (based on pre-randomization data only)
Ineligible for OHP Standard (out of state address, age, etc.)
Individuals with institutional addresses on list
Final sample: 79,922 individuals out of 66,385 households
29,834 treated individuals (surveyed 29,589)
40,088 control individuals (surveyed 28,816)
Sample Characteristics
Sample characteristics

19
 Outcomes

Access and use of care

Is access to care improved? Do the insured use more care? Is
there a shift in the types of care being used?
Mail surveys and hospital discharge data
Financial strain
How much does insurance protect against financial strain?
What are the out-of-pocket implications?
Mail surveys and credit reports
Health
What are the short-term impacts on self-reported physical and
mental health?
Mail surveys and vital statistics (mortality)
Results: Access
Effect of lottery&onUse of Care
coverage (first stage)

Gaining insurance resulted in better access to care and higher

Gaining insurance resulted in better access to care and higher
satisfaction
satisfaction withwith care (conditional
care (conditional on actually
on actually getting care)
getting care).

CONTROL RF Model IV Model P-Value

(ITT) (LATE)
Have a usual place of care 49.9% +9.9% +33.9% .0001
Have a personal doctor 49.0% +8.1% +28.0% .0001
Got all needed health care 68.4% +6.9% +23.9% .0001
Got all needed prescriptions 76.5% +5.6% +19.5% .0001
Satisfied with quality of care 70.8% +4.3% +14.2% .001

SOURCE: Survey data

21
22
 Effect of lottery on coverage (first stage)
Results: Access & Use of Care
Gaining insurance resulted in increased probability of hospital
Gaining insurance
admissions, resulted
primarily in increased
driven probability of department
by non-emergency hospital
admissions,
admissions primarily driven by non-ED admissions.

CONTROL RF Model IV Model P-Value

(ITT) (LATE)
Any hospital admission 6.7% +.50% +2.1% .004
--Admits through ED 4.8% +.2% +.7% .265
--Admits NOT through 2.9% +.4% +1.6% .002
ED

SOURCE: Hospital Discharge Data

Overall, this represents a 30% higher probability of admission, although admissions
are still rare events.
Overall, this represents a 30% higher probability of admission,
although admissions are still rare events 23
Total Use By Condition

24
 Summary: Access and use of care

Overall, utilization and costs went up relative to controls

30% increase in probability of an inpatient admission
35% increase in probability of an outpatient visit
15% increase in probability of taking prescription medications
Total $777 increase in average spending (a 25% increase)
With this increased spending, those who gained insurance
were
35% more likely to get all needed care
25% more likely to get all needed medications
Far more likely to follow preventive care guidelines, such as
mammograms (60%) and PAP tests (45%)
 Results: Financial Strain
Results: Financial Strain
Gaining insurance resulted in a reduced probability of having
medical collections in credit reports, and in lower amounts
Gaining
owed insurance resulted in a reduced probability of having medical
collections in credit reports, and in lower amounts owed.

CONTROL RF Model IV Model P-Value

(ITT) (LATE)
Had a bankruptcy 1.4% +0.2% +0.9% .358
Had a collection 50.0% -1.2% -4.8% .013
--Medical collections 28.1% -1.6% -6.4% .0001
--Non-medical collections 39.2% -0.5 -1.8% .455
$ owed medical collections $1,999 -$99 -$390 .025

SOURCE: Credit report data

Source: Credit report data

26
27
 Summary: Financial Strain

Overall, reductions in collections on credit reports were

evident
25% decrease in probability of a medical collection
Those with a collection owed significantly less
Household financial strain related to medical costs was
mitigated
Substantial reduction across all financial strain measures
Captures “informal channels” people use to make it work
Implications for both patients and providers
Only 2% of bills sent to collections are ever paid
 Results: Self-Reported Health
Results: Self-reported health

Self-reported measures showed significant improvements one

Self-reported measures showed significant improvements one year
year after randomization
after randomization
CONTROL RF Model IV Model P-Value
(ITT) (LATE)
Health good, v good, excellent 54.8% +3.9% +13.3% .0001
Health stable or improving 71.4% +3.3% +11.3% .0001
Depression screen NEGATIVE 67.1% +2.3% +7.8% .003

CDC Healthy Days (physical) 21.86 +.381 +1.31 .018

CDC Healthy Days (mental) 18.73 +.603 +2.08 .003

SOURCE: Survey data

Source: Survey data

29
 Summary: Self-reported health

Overall, big improvements in self-reported physical and mental

health
25% increase in probability of good, very good or excellent
health
10% decrease in probability of screening for depression
Physical health measures open to several interpretations
Improvements consistent with findings of increased utilization,
better access, and improved quality
BUT in their baseline surveys, results appeared shortly after
coverage (∼2/3rds magnitude of full result)
May suggest increase in perception of well-being rather than
physical health
Biomarker data can shed light on this issue
 Discussion

At 1 year, found increases in utilization, reductions in financial

strain, and improvements in self-reported health
Medicaid expansion had benefits and costs – didn’t “pay for
itself”
Confirmed biases inherent in observational studies – would
have estimated bigger increases in use and smaller
improvements in outcomes
Policy-makers may have different views on value of different
aspects of improved well-being
“I have an incredible amount of fear because I don’t know if
the cancer has spread or not.”
“A lot of times I wanted to rob a bank so I could pay for the
medicine I was just so scared . . . People with cancer either
have a good chance or no chance. In my case it’s hard to
recover from lung cancer but it’s possible. Insurance took so
long to kick in that I didn’t think I would get it. Now there is
a big bright light shining on me.” (Anecdotes)
Important to have broad evidence on multifaceted effects of
Medicaid expansions
Baicker, Katherine, et al. (2014). “The Oregon Experiment
– Effects of Medicaid on Clinical Outcomes”, The New
England Journal of Medicine.
 In-person data collection

Questionnaire and health examination including

Survey questions
Anthropometric and blood pressure measurement
Dried blood spot collection
Catalog of all medications
Fielded between September 2009 and December 2010
Average response ∼25 months after lottery began
Limited to Portland area: 20,745 person sample
12,229 interviews for effective response rate of 73%
 Analytic approach

Intent to treat effect of lottery selection

Comparing all selected with all not selected
Random treatment assignment
No differential selection for outcome measurement
Local average treatment effect on Medicaid coverage
Using lottery selection as an instrument for coverage
∼24 percentage point increase in Medicaid enrollment
No change in private insurance (no crowd-out)
No effect of lottery except via Medicaid coverage
Statistical inference is the same for both
 Results

1 Health care use

2 Financial strain
3 Clinical health outcomes
36
37
 Health care use results

Increases in use in various settings

Increases in probability and number of outpatient visits
Increases in probability and number of prescription drugs
No discernible change in hospital or ED use (imprecise)
Increases in preventive care across range of services
Increases in perceived access and quality
Implied 35% increase in spending for insured
 Results

1 Health care use

2 Financial strain
3 Clinical health outcomes
40
 Financial Hardship Results

Reduction in strain, out-of-pocket (OOP), money owed

Substantial reduction across measures
Elimination of catastrophic OOP health spending
Implications for distribution of burden/benefits
Some borne by patients, some by providers
Non-financial burden of medical expenses and debt
 Results

1 Health care use

2 Financial strain
3 Clinical health outcomes
 Focusing on specific conditions

Measured:
Blood pressure
Cholesterol levels
Glycated hemoglobin
Depression
Reasons for selecting these:
Reasonably prevalent conditions
Clinically effective medications exist
Markers of longer term risk of cardiovascular disease
Can be measured by trained interviewers and lab tests
A limited window into health status
44
45
46
47
 Results on specific conditions

Large reductions in depression

Increases in diagnoses and medication
In-person estimate of −9 percentage points in being depressed
Glycated hemoglobin
Increases in diagnosis and medication
No significant effect on HbA1c; wide confidence intervals
Blood pressure and cholesterol
No significant effects on diagnosis or medication
No significant effects on outcomes
Framingham risk score
No significant effect (in general or sub-poplulations)
49
 Summary

One to two years after expanded access to Medicaid:

Increases in health care use and associated costs
Increases in compliance with recommended preventive care
Improvements in quality and access
Reductions in financial strain
Improvements in self-reported health
Improvements in depression
No significant change in specific physical measures
Sense of the relative magnitude of the effects
Use and access, financial benefits, general health, depression
Physical measures of specific chronic conditions
 Extrapolation to Obamacare (ACA) Expansion

Context quite relevant for health care reform:

States can choose to cover a similar population in planned
2014 Medicaid expansions (up to 138% of federal poverty line)
But important caveats to bear in mind
Oregon and Portland vs. US generally
Voluntary enrollment vs. mandate
Partial vs. general equilibrium effects
Short-run (1-2 years) vs. medium or long run
 Updating Priors based on Study’s Findings

“Medicaid is worthless or worse than no insurance”’

Studies found increases in utilization and perceived access and
quality
Reductions in financial strain, improvement in self-reported
health
Improvement in depression
Can reject large declines in several physical measures
“Health insurance expansion saves money”
In short run, studies showed increases in utilization and cost
and no change in ED use
Increases in preventive care, improvements in self-reported
health, improvements in depression
 Conclusion

Effects of expanding Medicaid likely to be manifold

Hard to establish with observational data and often misleading
Expanding Medicaid generates both costs and benefits
Increased spending
Measurably improves some aspects of health but not others
Important caveats about generalizability
Weighing them depends on policy priorities
Further research on alternative policies needed
Many steps in pathway between insurance and outcome
Role for innovation in insurance coverage
Complements to health care (e.g., social determinants)
 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Topics covered

1 Review fixed effects regression models

2 Differences-in-differences basics: Card and Krueger (1994)
3 Regression differences-in-differences
4 Synthetic control: Abadie, Diamond and Hainmueller (2010)
5 Combining differences-in-differences with IV: Waldinger (2010)

Cunningham Causal Inference

 Overview
Part 1: The Core Natural Experiments
Part 2: Selection on observables Instrumental variables
Part 3: Selection on unobservables Panel data
Part 4: Advanced material Differences-in-differences
Conclusion Synthetic Control
References

Panel Methods

Panel data: we observe the same units (individuals, firms,

countries, schools, etc.) over several time periods
Often our outcome variable depends on unobserved factors
which are also correlated with our explanatory variable of
interest
If these omitted variables are constant over time, we can use
panel data estimators to consistently estimate the effect of
our explanatory variable
Main estimators for panel data:
Pooled OLS
Fixed effects estimator
Random effects estimator
Cunningham Causal Inference
 Panel setup

Let y and x ≡ (x1 , x2 , . . . , xk ) be observable random variables

and c be an unobservable random variable
We are interested in the partial effects of variable xj in the
population regression function

E [y |x1 , x2 , . . . , xk , c]

We observe a sample of i = 1, 2, . . . , N cross-sectional units

for t = 1, 2, . . . , T time periods (a balanced panel)
For each unit i, we denote the observable variables for all time
periods as {(yit , xit ) : t = 1, 2, . . . , T }
xit ≡ (xit1 , xit2 , . . . , xitk ) is a 1 × K vector
Typically assume that cross-sectional units are i.i.d. draws
from the population: {yi , xi , ci }N
i=1 ∼ i.i.d. (cross-sectional
independence)
yi ≡ (yi1 , yi2 , . . . , yiT )0 and xi ≡ (xi1 , xi2 , . . . , xiT )
Consider asymptotic properties with T fixed and N → ∞
 Panel setup

Single unit:

   
yi1 Xi,1,1 Xi,1,2 Xi,1,j ... Xi,1,K
 ..   .. .. .. .. 
 .   . . . . 
   
yi =  yit 
  Xi =  Xi,t,1 Xi,t,2 Xi,t,j
 ... Xi,t,K 
 ..  .. .. .. ..

 
 .   . . . . 
yiT T ×1 Xi,T ,1 Xi,T ,2 Xi,T ,j ... Xi,T ,K T ×K

Panel with all units:

   
y1 X1
 ..   .. 
 .   . 
   
 yi 
y =  Xi 
X =
 
 ..   .. 
 .   . 
yN NT ×1 XN NT ×K
 Unobserved effects model: Farm output

For a randomly drawn cross-sectional unit i, the model is

given by
yit = xit β + ci + εit , t = 1, 2, . . . , T

yit : output of farm i in year t

xit : 1 × K vector of variable inputs for farm i in year t, such as
labor, fertilizer, etc. plus an intercept
β : K × 1 vector of marginal effects of variable inputs
ci : sum of all time-invariant inputs known to farmer i (but
unobserved for the researcher), e.g., soil quality, managerial
ability, etc.
often called the unobserved effect, unobserved heterogeneity,
etc
εit : time-varying unobserved inputs, such as rainfall, unknown
to the farmer at the time the decision on the variable inputs xit
is made
often called the idiosyncratic error
What happens when we regress yit on xit ?
 Pooled OLS

When we ignore the panel structure and regress yit on xit we

get
yit = xit β + vit ; t = 1, 2, . . . , T
with composite error vit ≡ ci + εit
Main assumption to obtain consistent estimates for β is:
E [vit |xi1 , xi2 , . . . , xiT ] = E [vit |xit ] = 0 for t = 1, 2, . . . , T
xit are strictly exogenous: the composite error vit in each time
period is uncorrelated with the past, current and future
regressors
But: labour input xit likely depends on soil quality ci and so
we have omitted variable bias and βb is not consistent
No correlation between xit and vit implies no correlation
between unobserved effect ci and xit for all t
Violations are common: whenever we omit a time-constant
variable that is correlated with the regressors (heterogeneity
bias)
Additional problem: vit are serially correlated for same i since
ci is present in each t and thus pooled OLS standard errors are
invalid
 Unobserved effects model: program evaluation

Program evaluation model:

yit = progit β + ci + εit ; t = 1, 2, . . . , T

yit : log wage of individual i in year t

progit : indicator coded 1 if individual i participants in training
program at t and 0 otherwise
β: effect of program
ci : sum of all time-invariant unobserved characteristics that
affect wages, such as ability, etc.
What happens when we regress yit on progit ? βb not
consistent since progit is likely correlated with ci (e.g., ability)
Always ask: is there a time-constant unobserved variable (ci )
that is correlated with the regressors? If yes, then pooled OLS
is problematic
 Fixed effect regression

Our unobserved effects model is:

yit = xit β + ci + εit ; t = 1, 2, . . . , T

If we have data on multiple time periods, we can think of ci

as fixed effects or “nuisance parameters” to be estimated
OLS estimation with fixed effects yields
N X
X T
(β,
b cb1 , . . . , cbN ) = argmin (yit − xit b − mi )2
b,m1 ,...,mN i=1 t=1

this amounts to including N farm dummies in regression of yit

on xit
 Derivation: fixed effects regression

N X
X T
(β,
b cb1 , . . . , cbN ) = argmin (yit − xit b − mi )2
b,m1 ,...,mN i=1 t=1

The first-order conditions (FOC) for this minimization problem are:

N X
X T
xit0 (yit − xit βb − cbi ) = 0
i=1 t=1

and
T
X
(yit − xit βb − cbi ) = 0
t=1

for i = 1, . . . , N.
 Derivation: fixed effects regression

Therefore, for i = 1, . . . , N,
T
1 X
cbi = (yit − xit β)
b = ȳi − x̄i β,
b
T
t=1

where
T T
1 X 1 X
x̄i ≡ xit ; ȳi ≡ yit
T T
t=1 t=1

Plug this result into the first FOC to obtain:

N X
X T N X
−1  X T 
βb = (xit − x̄i )0 (xit − x̄i ) (xit − x̄i )0 (yit − ȳ )
i=1 t=1 i=1 t=1

N X
X T N X
−1  X T 
βb = ẍit0 ẍit ẍit0 ẍit
i=1 t=1 i=1 t=1

with time-demeaned variables ẍit ≡ xit − x̄, ÿit ≡ yit − ȳi

 Fixed effects regression

Running a regression with the time-demeaned variables

ÿit ≡ yit − ȳi and ẍit ≡ xit − x̄ is numerically equivalent to a
regression of yit on xit and unit specific dummy variables.

Even better, the regression with the time demeaned variables is

consistent for β even when Cov [xit , ci ] 6= 0 because
time-demeaning eliminates the unobserved effects

yit = xit β + ci + εit

ȳi = x̄i β + ci + ε̄i

(yit − ȳi ) = (xit − x̄)β + (ci − ci ) + (εit − ε̄i )

ÿit = ẍit β + ε̈it
 Fixed effects regression: main results

Identification assumptions:
1 E [εit |xi1 , x + i2, . . . , xiT , ci ] = 0; t = 1, 2, . . . , T
regressors are strictly exogenous conditional on the unobserved
effect
allows xit to be arbitrarily related to ci
 
PT 0
2 rank t=1 E [ẍ ẍ
it it ] =K
regressors vary over time for at least some i and not collinear
Fixed effects estimator
1 Demean and regress ÿit on ẍit (need to correct degrees of
freedom)
2 Regress yit on xit and unit dummies (dummy variable
regression)
3 Regress yit on xit with canned fixed effects routine
STATA: xtreg y x, fe i(PanelID)
Properties (under assumptions 1-2):
βbFE is consistent: plim βbFE ,N = β
N→∞
βbFE is unbiased conditional on X
 Fixed effects regression: main issues

Inference:
Standard errors have to be “clustered” by panel unit (e.g.,
farm) to allow correlation in the εit ’s for the same i.
STATA: xtreg , fe i(PanelID) cluster( PanelID )
Yields valid inference as long as number of clusters is
reasonably large
Typically we care about β, but unit fixed effects ci could be of
interest
cbi from dummy variable regression is unbiased but not
consistent for ci (based on fixed T and N → ∞)
xtreg , fe routine demeans the data before running the
regression and therefore does not estimate cbi
intercept shows average cbi across units
we can recover cbi using cbi = ȳi − x̄i βb
predict c_i, u
 Example: Direct Democracy and Naturalizations

Do minorities fare worse under direct democracy than under

representative democracy?
Hainmueller and Hangartner (2012) examine data on
naturalization requests of immigrants in Switzerland, where
municipalities vote on naturalization applications in:
referendums (direct democracy)
elected municipality councils (representative democracy)
Annual panel data from 1,400 municipalities for the
1991-2009 period
no.naturalizationsit
yit : naturalization rate = eligibleforeignpopulation i,t−1
xit : 1 if municipality used representative democracy, 0 if
municipality used direct democracy in year t
 Fixed E↵ects Regression

Naturalization Panel Data

Naturalization Panel Data

. des muniID muni_name year nat_rate repdem

storage display value

variable name type format label variable label

muniID float %8.0g municipality code

muni_name str43 %43s municipality name
year float %ty year
nat_rate float %9.0g naturalization rate (percent)
repdem float %9.0g 1 representative democracy, 0 direct

J. Hainmueller (MIT) 20 / 64
 Fixed E↵ects Regression

Panel Data LongPanel

Format
Data Long Format

. list muniID muni_name year nat_rate repdem in 31/40

muniID muni_name year nat_rate repdem

31. 2 Affoltern A.A. 2002 4.638365 0

32. 2 Affoltern A.A. 2003 4.844814 0
33. 2 Affoltern A.A. 2004 5.621302 0
34. 2 Affoltern A.A. 2005 4.387827 0
35. 2 Affoltern A.A. 2006 8.115358 1

36. 2 Affoltern A.A. 2007 7.067371 1

37. 2 Affoltern A.A. 2008 8.977719 1
38. 2 Affoltern A.A. 2009 6.119704 1
39. 3 Bonstetten 1991 .8333334 0
40. 3 Bonstetten 1992 .8403362 0

J. Hainmueller (MIT) 21 / 64
 Fixed E↵ects Regression

Pooled OLS Pooled OLS

. ** pooled ols
. reg nat_rate repdem , cl(muniID)

Linear regression Number of obs = 4655

F( 1, 244) = 130.04
Prob > F = 0.0000
R-squared = 0.0748
Root MSE = 3.98

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem 2.503318 .2195202 11.40 0.000 2.070921 2.935714

_cons 2.222683 .10088 22.03 0.000 2.023976 2.421389

J. Hainmueller (MIT) 22 / 64
 Fixed E↵ects Regression

Decompose Within and Between Variation

Decompose within and between variation

. tsset muniID year , yearly

panel variable: muniID (strongly balanced)
time variable: year, 1991 to 2009
delta: 1 year

. xtsum nat_rate

Variable Mean Std. Dev. Min Max Observations

nat_rate overall 2.938992 4.137305 0 24.13793 N = 4655

between 1.622939 0 7.567746 n = 245
within 3.807039 -3.711323 24.80134 T = 19

J. Hainmueller (MIT) 23 / 64
 Fixed E↵ects Regression

Time-Demeaning for FixedforE↵ects:

Time-demeaning yit ! ÿityit → ÿit
fixed effects:

. * get municipality means

. egen means_nat_rate = mean(nat_rate) , by(muniID)

. * compute deviations from means

. gen dm_nat_rate = nat_rate - means_nat_rate

. list muniID muni_name year nat_rate means_nat_rate dm_nat_rate in 20/40 ,ab(20)

muniID muni_name year nat_rate means_nat_rate dm_nat_rate

20. 2 Affoltern A.A. 1991 .2173913 3.595932 -3.37854

21. 2 Affoltern A.A. 1992 .9473684 3.595932 -2.648563
22. 2 Affoltern A.A. 1993 1.04712 3.595932 -2.548811
23. 2 Affoltern A.A. 1994 .8342023 3.595932 -2.761729
24. 2 Affoltern A.A. 1995 2.002002 3.595932 -1.59393

25. 2 Affoltern A.A. 1996 1.7769 3.595932 -1.819031

26. 2 Affoltern A.A. 1997 1.862745 3.595932 -1.733186
27. 2 Affoltern A.A. 1998 2.054155 3.595932 -1.541776
28. 2 Affoltern A.A. 1999 2.402135 3.595932 -1.193796
29. 2 Affoltern A.A. 2000 2.195946 3.595932 -1.399986

J. Hainmueller (MIT) 24 / 64
 Fixed effects regression with demeaned data
Fixed E↵ects Regression

Fixed E↵ects Regression with Demeaned Data

. * demean direct demoracy dummy
. egen means_repdem = mean(repdem) , by(muniID)

. gen dm_repdem = repdem - means_repdem

.
. * regression with demeaned data
. reg dm_nat_rate dm_repdem , cl(muniID)

Linear regression Number of obs = 4655

F( 1, 244) = 265.18
Prob > F = 0.0000
R-squared = 0.1052
Root MSE = 3.6017

(Std. Err. adjusted for 245 clusters in muniID)

Robust
dm_nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

dm_repdem 3.0228 .1856244 16.28 0.000 2.657169 3.388431

_cons 6.65e-10 5.81e-09 0.11 0.909 -1.08e-08 1.21e-08

J. Hainmueller (MIT) 25 / 64
 Fixed effects regression
Fixed E↵ects Regression with canned routine
Fixed E↵ects Regression with Canned Routine
. xtreg nat_rate repdem , fe cl(muniID) i(muniID)

Fixed-effects (within) regression Number of obs = 4655

Group variable: muniID Number of groups = 245

R-sq: within = 0.1052 Obs per group: min = 19

between = 0.0005 avg = 19.0
overall = 0.0748 max = 19

F(1,244) = 265.18
corr(u_i, Xb) = -0.1373 Prob > F = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem 3.0228 .1856244 16.28 0.000 2.657169 3.388431

_cons 2.074036 .0531153 39.05 0.000 1.969413 2.178659

sigma_u 1.7129711
sigma_e 3.69998
rho .17650677 (fraction of variance due to u_i)

J. Hainmueller (MIT) 26 / 64
 Fixed E↵ects Regression
Fixed effects regression with dummies
Fixed E↵ects Regression with Dummies

. reg nat_rate repdem i.muniID, cl(muniID)

Linear regression Number of obs = 4655

F( 0, 244) = .
Prob > F = .
R-squared = 0.2423
Root MSE = 3.7

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem 3.0228 .1906916 15.85 0.000 2.647188 3.398412

muniID
2 1.367365 5.17e-14 2.6e+13 0.000 1.367365 1.367365
3 1.292252 5.17e-14 2.5e+13 0.000 1.292252 1.292252
9 1.284652 5.17e-14 2.5e+13 0.000 1.284652 1.284652
10 1.271783 5.17e-14 2.5e+13 0.000 1.271783 1.271783
13 .3265469 5.17e-14 6.3e+12 0.000 .3265469 .3265469

J. Hainmueller (MIT) 27 / 64
 Applying fixed effects

We can use fixed effects for other data structures to restrict

comparisons to within unit variation
Matched pairs
Twin fixed effects to control for unobserved effects of family
background
Cluster fixed effects in hierarchical data
School fixed effects to control for unobserved effects of school
 Problems that even fixed effects do not solve

yit = xit β + ci + εit ; t = 1, 2, . . . , T

Where yit is murder rate and xit is police spending per capita
What happens when we regress y on x and city fixed effects?
βbFE inconsistent unless strict exogeneity conditional on ci holds
E [εit |xi1 , xi2 , . . . , xiT , ci ] = 0; t = 1, 2, . . . , T
implies εit uncorrelated with past, current and future regressors
Most common violations
1 Time-varying omitted variables
Economic boom leads to more police spending and less murders
Can include time-varying controls, but avoid post-treatment bias (i.e., collider)
2 Simultaneity
if city adjusts police based on past murder rate, then spendingt+1 is correlated
with εt (since higher εt leads to higher murder rate at t)
strictly exogenous x cannot react to what happens to y in the past or the future!
Fixed effects do not obviate need for good research design!
 Random Effects

Reconsider our unobserved effects model:

yit = xit β + ci + εit , t = 1, 2, . . . , T

Cannot use the fixed effects regression to estimate effects of

time-constant regressors in xit (eg., soil quality, farm location,
etc.)
Since fixed effect estimator allows ci to be correlated with xit ,
we cannot distinguish the effects of time-invariant regressors
from the time-invariant unobserved effect ci
Need orthogonality assumption: Cov [xit , ci ] = 0;
t = 1, . . . , T
Strong assumption: Unobserved effects ci are uncorrelated
with each explanatory variable in xit in each time period
For example if we include soil quality in xit we have to assume
it is uncorrelated with all other time-invariant inputs
 Random effects assumptions

yit = xit β + ci + εit ; t = 1, . . . , T

1 E [εit |xi , ci ] = 0; t = 1, 2, . . . , T : explanatory variables are

strictly exogenous conditional on the unobserved effect
2 E [ci |xi ] = E [ci ] = 0: unobserved effects ci are uncorrelated
with regressors
3 rank E [Xi0 ΩXi ] = K : no collinearity among regressors
Ω = E [vi vi0 ] : the variance matrix of the composite error
vit = ci + εit
4 We typically also assume that Ω takes a special form:
E [εi ε0i |xi ] = σε2 IT : idiosyncratic errors are homoskedastic for
all t and serially uncorrelated
E [ci2 |xi ] = σc2 : unobserved effect ci is homoscedastic
 Random effects assumptions

yit = xit β + ci + εit ; t = 1, . . . , T

1 E [εit |xi , ci ] = 0; t = 1, 2, . . . , T : explanatory variables are strictly exogenous

conditional on the unobserved effect
2 E [ci |xi ] = E [ci ] = 0: unobserved effects ci are uncorrelated with regressors
3 rank E [Xi0 ΩXi ] = K : no collinearity among regressors
Ω = E [vi vi0 ] : the variance matrix of the composite error vit = ci + εit
4 We typically also assume that Ω takes a special form:
E [εi ε0i |xi ] = σε2 IT : idiosyncratic errors are homoskedastic for all t and serially
uncorrelated
E [ci2 |xi ] = σc2 : unobserved effect ci is homoscedastic
σc2 + σε2 σc2 σc2
 
...
..
σc2 σc2 + σε2 . . .
 
.
Assumption 4 implies Ω = E [vi vi0 |xi ] = 
 
..

..
. σc2
 
 . 
σc2 σc2 + σε2 T ×T
 Random effects assumptions

yit = xit β + ci + εit ; t = 1, . . . , T

1 E [εit |xi , ci ] = 0; t = 1, 2, . . . , T : explanatory variables are strictly exogenous

conditional on the unobserved effect
2 E [ci |xi ] = E [ci ] = 0: unobserved effects ci are uncorrelated with regressors
3 rank E [Xi0 ΩXi ] = K : no collinearity among regressors
Ω = E [vi vi0 ] : the variance matrix of the composite error vit = ci + εit
4 We typically also assume that Ω takes a special form:
E [εi ε0i |xi ] = σε2 IT : idiosyncratic errors are homoskedastic for all t and serially
uncorrelated
E [ci2 |xi ] = σc2 : unobserved effect ci is homoscedastic

Given assumptions 1-3, pooled OLS is consistent, since composite error vit is
uncorrelated with xit for all t
However, pooled OLS ignores the serial correlation in vit
 Random effects assumptions

yit = xit β + ci + εit ; t = 1, . . . , T

1 E [εit |xi , ci ] = 0; t = 1, 2, . . . , T : explanatory variables are strictly exogenous

conditional on the unobserved effect
2 E [ci |xi ] = E [ci ] = 0: unobserved effects ci are uncorrelated with regressors
3 rank E [Xi0 ΩXi ] = K : no collinearity among regressors
Ω = E [vi vi0 ] : the variance matrix of the composite error vit = ci + εit
4 We typically also assume that Ω takes a special form:
E [εi ε0i |xi ] = σε2 IT : idiosyncratic errors are homoskedastic for all t and serially
uncorrelated
E [ci2 |xi ] = σc2 : unobserved effect ci is homoscedastic

Random effects estimator βbRE exploits this serial correlation in a generalized

least squares (GLS) framework
βbRE is consistent under assumption 1-3: plim βbRE ,N = β
N→∞
βbRE is asymptotically efficient given assumption 4 (in the class of estimators
consistent under E [vi |xi ] = 0)
 Random effects estimator

Consider the transformation parameter

1
σε2

2
λ=1− 2 2
with 0 ≤ λ ≤ 1
σε + T σc

σε2 = Var [εit ]: variance of idiosyncratic error

σc2 = Var (ci ): Variance of unobserved effect
βbRE is equivalent to pooled OLS on:
yit − ȳi = (xit − λx̄i )β + (vit − λv̄i ), ∀i, t
ỹit = x̃it β + ṽit

As λ → 1, βbRE → βbFE
As λ → 0, βbRE → βbPooled OLS
λ → 1 as T → ∞ or if variance of ci is large relative to
variance of εit
1
λ can be estimated from data λ σε2 /(b
b = 1 − (b σε2 + T εb2c )) 2
Usually wise to cluster the standard errors since assumption 4
is strong
 Random effects regression
Random E↵ects

Random E↵ects Regression

. xtreg nat_rate repdem , re cl(muniID) i(muniID)

Random-effects GLS regression Number of obs = 4655

Group variable: muniID Number of groups = 245

R-sq: within = 0.1052 Obs per group: min = 19

between = 0.0005 avg = 19.0
overall = 0.0748 max = 19

Wald chi2(1) = 227.99

corr(u_i, X) = 0 (assumed) Prob > chi2 = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. z P>|z| [95% Conf. Interval]

repdem 2.859397 .1893742 15.10 0.000 2.48823 3.230564

_cons 2.120793 .0972959 21.80 0.000 1.930096 2.311489

sigma_u 1.3866768
sigma_e 3.69998
rho .1231606 (fraction of variance due to u_i)

J. Hainmueller (MIT) 34 / 64
Summary: Fixed effects, random effects, Pooled OLS

Main assumptions
1 Regressors are strictly exogenous conditional on the
time-invariant unobserved effects
2 Regressors are uncorrelated with the time-invariant unobserved
effects
Results
Fixed effects estimator is consistent given assumption 1, but
rules out time-invariant regressors
Random effects estimators and pooled OLS are consistent
under assumptions 1-2, and allow for time-invariant regressors
Given homoskedasticity assumptions (random effects
assumption 4), the random effects estimator is asymptotically
efficient
Assumption 2 is strong so fixed effects are typically more
credible
Often the main reason for using panel data is to rule out all
time-invariant unobserved confounders!
 Hausman test

βbRE βbFE
H0 : Cov [xit , ci ] = 0 Consistent and efficient Consistent
H1 : Cov [xit , ci ] 6= 0 Inconsistent Consistent

Then,
Under H0 , βbRE − βbFE should be close to zero
Under H1 , βbRE − βbFE should be different from zero
It can be shown that in large samples, under H0 , the test
statistic
d
(βbFE − βbRE )0 (Var d [βRE ])−1 (βbFE − βbRE ) →
d [βFE ] − Var χ2k

where k is the number of time-varying regressors.

We may reject the null hypothesis of “random effects” and
stick with the less efficient, but consistent fixed effects
specification
 Random effects regression
Random E↵ects

Hausman Test
. ** hausman test
. quietly: xtreg nat_rate repdem , fe i(muniID)

. estimates store FE

.
. quietly: xtreg nat_rate repdem , re i(muniID)

. estimates store RE

. hausman FE RE

Coefficients
(b) (B) (b-B) sqrt(diag(V_b-V_B))
FE RE Difference S.E.

repdem 3.0228 2.859397 .1634027 .0304517

b = consistent under Ho and Ha; obtained from xtreg

B = inconsistent under Ha, efficient under Ho; obtained from xtreg

Test: Ho: difference in coefficients not systematic

chi2(1) = (b-B)'[(V_b-V_B)^(-1)](b-B)
= 28.79
Prob>chi2 = 0.0000

J. Hainmueller (MIT) 37 / 64
 Hausman test
Random E↵ects

Hausman
HausmanTest
test does not test if the fixed effect model is correct;
the test assumes that the fixed effects estimator is consistent!
Conventional Hausman test assumes homoskedastic model
Hausman test does not test if the fixed e↵ects model is correct, the test
and doesthat
assumes nottheallow
fixed for clustering
e↵ects estimator is consistent!
There are Haumsman
Conventional Hausman testlike testshomoscedastic
assumes that allow model
for clustered
and does not
allow for clustering
standard errors
There are Hausman like tests that allow for clustered standard errors

. * hausman test with clustering

. quietly: xtreg nat_rate repdem , re i(muniID) cl(muniID)

. xtoverid

Test of overidentifying restrictions: fixed vs random effects

Cross-section time-series model: xtreg re robust cluster(muniID)
Sargan-Hansen statistic 26.560 Chi-sq(1) P-value = 0.0000
 Adding Time Effects

Reconsider our unobserved effects model:

yit = xit β + ci + εit , t = 1, 2, . . . , T

Fixed effects assumption: E [εit |xi , ci ] = 0; t = 1, 2, . . . , T :

regressors are strictly exogenous conditional on the unobserved
effect
Typical violation: Common shocks that affect all units in the
same way and are correlated with xit
Trends in farming technology or climate affect productivity
Trends in immigration inflows affect naturalization rates
We can allow for such common shocks by including time
effects into the model
 Random
Modelingeffects
Time regression
Naturalization Rate Over Time

6
naturalization rate (percent)
2 0 4

1990 1995 2000 2005 2010

year

naturalization rate (percent) lb/ub

xtgraph nat rate

xtgraph nat_rate
J. Hainmueller (MIT) 41 / 64
 Random
Modelingeffects
Time regression
Representative Democracy Over Time

1
representative democracy (1/0)
.2 .4 0 .6 .8

1990 1995 2000 2005 2010

year

representative democracy lb/ub

xtgraph repdem
xtgraph repdem
J. Hainmueller (MIT) 42 / 64
 Fixed effects: adding time effects

Linear time trend:

yit = xit β + ci + t + εit ; t = 1, 2, . . . , T

Linear time trend common to all units

Time fixed effects:

yit = xit β + ci + tt + εit ; t = 1, 2, . . . , T

Common shock in each time period

Generalized difference-in-difference model
Unit specific linear time trends:

yit = xit β + ci + gi · t + tt + εit ; t = 1, 2, . . . , T

Linear time trends that vary by unit

 Modeling Time
Modeling time effects
Modeling Time E↵ects

J. Hainmueller (MIT) 44 / 64
 Fixed effects: adding
Modeling Time time effects
Fixed E↵ects Regression: Adding Time E↵ects
. egen time = group(year)

. list muniID muni_name year time in 20/40 ,ab(20)

muniID muni_name year time

20. 2 Affoltern A.A. 1991 1

21. 2 Affoltern A.A. 1992 2
22. 2 Affoltern A.A. 1993 3
23. 2 Affoltern A.A. 1994 4
24. 2 Affoltern A.A. 1995 5

25. 2 Affoltern A.A. 1996 6

26. 2 Affoltern A.A. 1997 7
27. 2 Affoltern A.A. 1998 8
28. 2 Affoltern A.A. 1999 9
29. 2 Affoltern A.A. 2000 10

30. 2 Affoltern A.A. 2001 11

J. Hainmueller (MIT) 45 / 64
 Fixed effects: linear time trend
Modeling Time

Fixed E↵ects Regression: Linear Time Trend

. xtreg nat_rate repdem time , fe cl(muniID) i(muniID)

Fixed-effects (within) regression Number of obs = 4655

Group variable: muniID Number of groups = 245

R-sq: within = 0.1604 Obs per group: min = 19

between = 0.0005 avg = 19.0
overall = 0.1350 max = 19

F(2,244) = 247.57
corr(u_i, Xb) = -0.0079 Prob > F = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem .8247928 .2590615 3.18 0.002 .3145106 1.335075

time .2313692 .0171752 13.47 0.000 .1975386 .2651997
_cons .3892908 .1309232 2.97 0.003 .1314069 .6471747

sigma_u 1.6271657
sigma_e 3.584409
rho .17086519 (fraction of variance due to u_i)

J. Hainmueller (MIT) 46 / 64
 Fixed effects: year fixed effects
Modeling Time

Fixed E↵ects Regression: Year Fixed E↵ects

. xtreg nat_rate repdem i.time , fe cl(muniID) i(muniID)

Fixed-effects (within) regression Number of obs = 4655

Group variable: muniID Number of groups = 245

R-sq: within = 0.1885 Obs per group: min = 19

between = 0.0005 avg = 19.0
overall = 0.1575 max = 19

F(19,244) = 31.48
corr(u_i, Xb) = -0.0168 Prob > F = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem 1.203658 .3031499 3.97 0.000 .6065335 1.800783

time
2 .3829173 .1723225 2.22 0.027 .0434879 .7223468
3 .2789777 .1514124 1.84 0.067 -.0192644 .5772198
4 .7034078 .167466 4.20 0.000 .3735443 1.033271

J. Hainmueller (MIT) 47 / 64
 Fixed effects: unit
Modeling Time specific time trends

Fixed E↵ects Regression: Unit Specific Time Trends

. xtreg nat_rate repdem muniID#c.time i.time , fe cl(muniID) i(muniID)
note: 19.time omitted because of collinearity

Fixed-effects (within) regression Number of obs = 4655

Group variable: muniID Number of groups = 245

R-sq: within = 0.2650 Obs per group: min = 19

between = 0.5185 avg = 19.0
overall = 0.2864 max = 19

F(18,244) = .
corr(u_i, Xb) = -0.3963 Prob > F = .

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem .9865241 .322868 3.06 0.002 .3505601 1.622488

muniID#c.time
1 .333343 .024298 13.72 0.000 .2854823 .3812036
2 .2914274 .024298 11.99 0.000 .2435667 .339288
3 .248985 .024298 10.25 0.000 .2011244 .2968457

J. Hainmueller (MIT) 48 / 64
 Unit specific time trends often eliminate “results”
Modeling Time

" AOt"
Unit Specific Time Trends Often Eliminate “Results” 240 Chapter 5
'I( ,
Fixed Effects, DD, and Panel Da

5.2.3
TABLE
the estimates in columns 2 and 3 ad
Estimated effects of labor regulation on the performance of firms
specific covariates, such as government
in Indian states
and state population. This is in the s
(1) (2) (3) (4) addition of state-level adult employme
Labor regulation (lagged) -.186 -.185 -.104 .0002 the minimum wage study. The addition
(.064) (.051) (.039) (.020) Besley and Burgess estimates little. Bu
specific trends kills the labor regulatio
Log development .240 .184 .241 in column 4. Apparently, labor regula
expenditure per capita (.128) (.119) (.106)
in states where output was declining an
Log installed electricity .089 .082 .023 trend therefore drives the estimated reg
capacity per capita (.061) (.054) (.033)
I
Log state population .720 0.310 -1.419 Picking Controls
(.96) (1.192) (2.326)
We've la beled the two dimensions in t
Congress majority -.0009 .020 time because this is the archetypical D
(.01) (.010)
econometrics. But the DD idea is much
Hard left majority -.050 -.007 of states, the subscript s might denot
(.017) (.009) some of which are affected by a poli
Janata majority .008 -.020 For example, Kugler, Jimeno, and Her
(.026) (.033) effects of age-specific employment prot
Likewise, instead of time, we might g
Regional majority .006 .026
other types of characteristics. An exam
(.009) (.023)
(1999), who studied the effect of ch
State-specific trends No No No Yes laws on teen pregnancy using variatio
Adjusted R2 .93 .93 .94 .95 birth. Regardless of the group labels
Notes: Adapted from Besley and Burgess (2004), table IV. The table reports always set up an implicit treatment-co
regression DD estimates of the effects of labor regulation on productivity. The question of whether this comparison
dependent variable is log manufacturing output per capita. All models include careful consideration.
state and year effects. Robust standard errors clustered at the state level are One potential pitfall in this contex
reported in parentheses. State amendments to the Industrial Disputes Act are
“labor regulation increased in states where output was declining
coded 1 = pro-worker, 0 = neutral, -1 = pro-employer and then cumulated anyway”
position of the treatment and contro
over the period to generate the labor regulation measure. Log of installed result of treatment. Going back to a
electrical capacity is measured in kilowatts, and log development expenditure and time comparisons, suppose we're
is real per capita state spending on social and economic services. Congress, of the generosity of public assistance O
J. Hainmueller (MIT) hard left, Janata, and regional majority are counts of the number of years 49 / 64
cally, U.S. states have offered widely V
 Distributed Lag model

yit = xit β0 + xi,t−1 β1 + xi,t−2 β2 + ci + εit , t = 1, 2, . . . , T

Model recognizes the effect of change in x may occur with a

late
effect of new tax credit for children on fertility rate
Interpretation of coefficients:
Consider temporary increase in xit from level m to m + 1 at t
which lasts only one period
yt−1 = mβ0 + mβ1 + mβ2 + ci
yt = (m + 1)β0 + mβ1 + mβ2 + ci
yt+1 = mβ0 + (m + 1)β1 + mβ2 + ci
yt+2 = mβ0 + mβ1 + (m + 1)β2 + ci
yt+3 = mβ0 + mβ1 + mβ2 + ci
 Distributed Lag model

yit = xit β0 + xi,t−1 β1 + xi,t−2 β2 + ci + εit , t = 1, 2, . . . , T

Model recognizes the effect of change in x may occur with a

late
effect of new tax credit for children on fertility rate
Interpretation of coefficients:
Consider temporary increase in xit from level m to m + 1 at t
which lasts only one period
yt−1 = mβ0 + mβ1 + mβ2 + ci
yt = (m + 1)β0 + mβ1 + mβ2 + ci
yt+1 = mβ0 + (m + 1)β1 + mβ2 + ci
yt+2 = mβ0 + mβ1 + (m + 1)β2 + ci
yt+3 = mβ0 + mβ1 + mβ2 + ci
β0 = yt − yt−1 immediate change in y due to temporary
one-unit increase in x (impact propensity)
 Distributed Lag model

yit = xit β0 + xi,t−1 β1 + xi,t−2 β2 + ci + εit , t = 1, 2, . . . , T

Model recognizes the effect of change in x may occur with a

late
effect of new tax credit for children on fertility rate
Interpretation of coefficients:
Consider temporary increase in xit from level m to m + 1 at t
which lasts only one period
yt−1 = mβ0 + mβ1 + mβ2 + ci
yt = (m + 1)β0 + mβ1 + mβ2 + ci
yt+1 = mβ0 + (m + 1)β1 + mβ2 + ci
yt+2 = mβ0 + mβ1 + (m + 1)β2 + ci
yt+3 = mβ0 + mβ1 + mβ2 + ci
β1 = yt+1 − yt change in y one period after temporary
one-unit increase in x
 Distributed Lag model

yit = xit β0 + xi,t−1 β1 + xi,t−2 β2 + ci + εit , t = 1, 2, . . . , T

Model recognizes the effect of change in x may occur with a

late
effect of new tax credit for children on fertility rate
Interpretation of coefficients:
Consider temporary increase in xit from level m to m + 1 at t
which lasts only one period
yt−1 = mβ0 + mβ1 + mβ2 + ci
yt = (m + 1)β0 + mβ1 + mβ2 + ci
yt+1 = mβ0 + (m + 1)β1 + mβ2 + ci
yt+2 = mβ0 + mβ1 + (m + 1)β2 + ci
yt+3 = mβ0 + mβ1 + mβ2 + ci
β2 = yt+2 − yt−1 change in y two periods after temporary
one-unit increase in x
 Distributed Lag model

yit = xit β0 + xi,t−1 β1 + xi,t−2 β2 + ci + εit , t = 1, 2, . . . , T

Model recognizes the effect of change in x may occur with a

late
effect of new tax credit for children on fertility rate
Interpretation of coefficients:
Consider temporary increase in xit from level m to m + 1 at t
which lasts only one period
yt−1 = mβ0 + mβ1 + mβ2 + ci
yt = (m + 1)β0 + mβ1 + mβ2 + ci
yt+1 = mβ0 + (m + 1)β1 + mβ2 + ci
yt+2 = mβ0 + mβ1 + (m + 1)β2 + ci
yt+3 = mβ0 + mβ1 + mβ2 + ci
β3 = yt−1 change in y is zero three periods after temporary
one-unit increase in x
 Distributed Lag model

yit = xit β0 + xi,t−1 β1 + xi,t−2 β2 + ci + εit , t = 1, 2, . . . , T

Interpretation of coefficients:
Consider permanent increase in xit from level m to m + 1 at
t, i.e., (xs = m, s < t and xs = m + 1, s ≥ t)
yt−1 = mβ0 + mβ1 + mβ2 + ci
yt = (m + 1)β0 + mβ1 + mβ2 + ci
yt+1 = (m + 1)β0 + (m + 1)β1 + mβ2 + ci
yt+2 = (m + 1)β0 + (m + 1)β1 + (m + 1)β2 + ci
yt+3 = (m + 1)β0 + (m + 1)β1 + (m + 1)β2 + ci
After one period y has increased by β0 + β1 , after two periods
y has increased by β0 + β1 + β2 and there are no further
increases after two periods
Long-run increase in y : β0 + β1 + β2 (long-run propensity)
 Modeling Dynamic E↵ects
Lagged effects of direct democracy
Lagged E↵ects of Direct Democracy

. xtreg nat_rate repdem L1.repdem L2.repdem L3.repdem i.year, fe cl(muniID) i(muniID)

Fixed-effects (within) regression Number of obs = 3920

Group variable: muniID Number of groups = 245

R-sq: within = 0.1536 Obs per group: min = 16

between = 0.0012 avg = 16.0
overall = 0.1235 max = 16

F(19,244) = 21.63
corr(u_i, Xb) = -0.0206 Prob > F = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem
--. .6364802 .3593924 1.77 0.078 -.0714272 1.344388
L1. 1.201065 .4233731 2.84 0.005 .367133 2.034998
L2. -.1648692 .4697434 -0.35 0.726 -1.090139 .7604003
L3. -.5245206 .4109918 -1.28 0.203 -1.334065 .2850239

J. Hainmueller (MIT) 53 / 64
 Modeling Dynamic E↵ects

Long-run E↵ect of Direct Democracy

Long-run effect of direct democracy

. lincom repdem + L1.repdem + L2.repdem + L3.repdem

( 1) repdem + L.repdem + L2.repdem + L3.repdem = 0

nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

(1) 1.294485 .4426322 2.92 0.004 .4226175 2.166353

J. Hainmueller (MIT) 54 / 64
 Lags and Leads model

yit = xi,t+1 β−1 +xit β0 +xi,t−1 β1 +xi,t−2 β2 +ci +εit ; t = 1, 2, . . . , T

Can use estimate of β−1 to test for anticipation effects

Consider temporary increase in xit from level m to m + 1 at t
yt−2 = β−1 m + mβ0 + mβ1 + mβ2 + ci
yt−1 = β−1 (m + 1) + mβ0 + mβ1 + mβ2 + ci
Anticipation effect: β−1 = yt−1 − yt−2 change in y in period
t − 1, the period before the temporary one-unit increase in x
Placebo test: if x causes y , but y does not cause x, then β−1
should be close to zero
 Modeling Dynamic E↵ects

Leads and Lags Leads and Lags

. xtreg nat_rate F1.repdem repdem L1.repdem L2.repdem L3.repdem i.year, fe cl(muniID) i(muniID)

Fixed-effects (within) regression Number of obs = 3675

Group variable: muniID Number of groups = 245

R-sq: within = 0.1621 Obs per group: min = 15

between = 0.0010 avg = 15.0
overall = 0.1269 max = 15

F(19,244) = 20.34
corr(u_i, Xb) = -0.0353 Prob > F = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

repdem
F1. .1707685 .3212906 0.53 0.596 -.4620886 .8036255
--. .6975731 .4397095 1.59 0.114 -.1685376 1.563684
L1. .8723962 .4619322 1.89 0.060 -.0374873 1.78228
L2. .014941 .4583628 0.03 0.974 -.8879119 .9177939
L3. -.2904252 .4108244 -0.71 0.480 -1.09964 .5187895

J. Hainmueller (MIT) 56 / 64
 The Autor Test

Let Dit be a binary indicator equaling 1 if unit i switched

from control to treatment between t and t − 1; 0 otherwise
Lags: Di,t−1 : unit switched between t − 1 and t − 2
Leads: Di,t+1 : unit switches between t + 1 and t
Include lags and leads into the fixed effects model:

yit = Di,t+2 β−2 +Di,t+1 β−1 +Dit β0 +Di,t−1 β1 +Di,t−2 β2 +ci +εit

Interpretation of coefficients:
Leads β−1 , β−2 , etc. test for anticipation effects
Switch β0 tests for immediate effect
Lags β1 , β2 , etc. test for long-run effects
highest lag dummy can be coded 1 for all post-switch years
 Modeling Dynamic E↵ects

Lags and Leads of Switch to Representative Democracy

Lags and Leads of Switch to Representative Democracy

. list muni_name year repdem switch_t sw_lag1 sw_lag2 sw_lag3 ///

> sw_lead1 sw_lead2 sw_lead3 in 806/817

muni_n~e year repdem switch_t sw_lag1 sw_lag2 sw_lag3 sw_lead1 sw_lead2 sw_lead3

806. Stäfa 1998 0 0 0 0 0 0 0 0

807. Stäfa 1999 0 0 0 0 0 0 0 0
808. Stäfa 2000 0 0 0 0 0 0 0 0
809. Stäfa 2001 0 0 0 0 0 0 0 0
810. Stäfa 2002 0 0 0 0 0 0 0 1

811. Stäfa 2003 0 0 0 0 0 0 1 0

812. Stäfa 2004 0 0 0 0 0 1 0 0
813. Stäfa 2005 1 1 0 0 0 0 0 0
814. Stäfa 2006 1 0 1 0 0 0 0 0
815. Stäfa 2007 1 0 0 1 0 0 0 0

816. Stäfa 2008 1 0 0 0 1 0 0 0

817. Stäfa 2009 1 0 0 0 1 0 0 0

J. Hainmueller (MIT) 58 / 64
Dynamic Effect of Switching to Representative Democracy
Modeling Dynamic E↵ects

Dynamic E↵ect of Switching to Representative Democracy

. xtreg nat_rate sw_lag3 sw_lag2 sw_lag1 switch_t ///
> sw_lead1 sw_lead2 sw_lead3 sw_lead4 sw_lead5 i.year, fe cluster(muniID) i(muniID)

Fixed-effects (within) regression Number of obs = 4655

Group variable: muniID Number of groups = 245

R-sq: within = 0.1913 Obs per group: min = 19

between = 0.0011 avg = 19.0
overall = 0.1601 max = 19

F(27,244) = 23.76
corr(u_i, Xb) = -0.0162 Prob > F = 0.0000

(Std. Err. adjusted for 245 clusters in muniID)

Robust
nat_rate Coef. Std. Err. t P>|t| [95% Conf. Interval]

sw_lag3 1.160345 .5080271 2.28 0.023 .1596665 2.161023

sw_lag2 1.743682 .5395212 3.23 0.001 .680969 2.806396
sw_lag1 1.881663 .4880099 3.86 0.000 .9204133 2.842913
switch_t .7564792 .428627 1.76 0.079 -.0878019 1.60076
sw_lead1 .2138757 .3899881 0.55 0.584 -.5542971 .9820485
sw_lead2 .0843676 .3575292 0.24 0.814 -.61987 .7886051
sw_lead3 .1440446 .3194086 0.45 0.652 -.4851054 .7731945
sw_lead4 .0750194 .2990359 0.25 0.802 -.5140018 .6640405
sw_lead5 -.0942415 .2599789 -0.36 0.717 -.6063307 .4178477

J. Hainmueller (MIT) 59 / 64
 Modeling Dynamic E↵ects

DynamicEffect
Dynamic E↵ectofofSwitching
SwitchingtotoRepresentative
RepresentativeDemocracy
Democracy

J. Hainmueller (MIT) 60 / 64
 Lagged Dependent Variable

yit = αyi,t−1 + ci + εit , t = 1, 2, . . . , T

yit could be capital stock of firm i at time t and α the capital

depreciation rate
For simplicity, we assume that εit are uncorrelated in time (as
well as across individuals)
Note that
yi,t−1 = αyi,t−2 + ci + εi,t−1
So we have Cov [yi,t−1 , ci ] 6= 0 and therefore we need to
include fixed effects ci into the regression
Does this work though?
 Lagged dependent variable

With T = 3 we have

yi3 = αyi2 + ci + εi3

yi2 = αyi1 + ci + εi2

and we can take time differences to eliminate ci (similar to fixed

effects)

yi3 − yi2 = α(yi2 − yi1 ) + (ci − ci ) + (εi3 − εi2 )

∆yi3 = α∆yi2 + ∆εi3

Since εi2 affects both ∆yi2 = yi2 − yi1 and ∆εi3 = εi3 − εi2 we get

Cov [∆yi2 , ∆εi3 ] 6= 0 and thus still have endogeneity

Models with fixed effects and lagged y do not produce consistent

estimators Might use past levels yi1 as an instrument for ∆yi2 , but

this requires strong assumptions (e.g., no serial correlation in εit )

Cheng, Cheng and Mark Hoestra 2013. “Does Strengthening
Self-Defense Law Deter Crime or Escalate Violence?
Evidence from Expansions to Castle Doctrine”. Journal of
Human Resources, 48(3), pp. 821-854.
 Summary

Cheng and Hoekstra (2013) are interested in whether

expansions to “castle doctrine statutes” at the state level
increase or decrease gun violence.
Prior to these expansions, English common law principle
required “duty to retreat” before using lethal force against an
assailant except when the assailant is an intruder in the home
The home is one’s “castle” – hence, “castle doctrine”
When intruders threatened the victim in the home, the duty to
retreat was waived and lethal force in self-defense was allowed
 Castle doctrine law explained

In 2005, Florida passed a law that expanded self-defense

protections beyond the house
2000 to 2010, 21 states explicitly put ?castle doctrine? into
statute, and (more importantly) extended it to places outside
the home
In other words, 21 states removed the duty to retreat in
specified circumstances
Other changes:
Presumption of reasonable fear is added
Civil liability for those acting under the law is removed
 Texas example

Duty to retreat
“For purposes of subsection (a), in determining whether an actor described by
subsection (e) reasonably believed that the use of force was necessary, a finder of
fact may not consider whether the actor failed to retreat.”
Also: Language stating a person is justified using deadly force against another “if
a reasonable person in the actor’s situation would not have retreated” is removed
from the statute
Presumption of reasonableness
“Except as provided in subsection (b), a person is justified in using force against
another when and to the degree the actor [he] reasonably believes the force is
immediately necessary to protect the actor [himself] against the other’s use or
attempted use of unlawful force. The actor’s belief that the force was
immediately necessary as described by this subsection is presumed to be
reasonable if the actor . . . ”
Civil Liability
“A defendant who uses force or deadly force that is justified under Chapter 9
Penal code is immune from civil liability for personal injury or death that results
from the defendant’s use of force or deadly force, as applicable.”
 Economic theory

Workers supply legal or illegal labor

Costs: foregone wages, risk of arrest, risk of conviction,
penalties conditional on conviction
Benefits: valuable goods
Castle doctrine expansions lowered the (expected) cost of
killing someone in self-defense
If people are rational, then lowering the price of lethal
self-defense should increase lethal homicides
 Economic theory

Two types of lethal force in self-defense:

True positive: use of lethal force against criminals committing
serious crimes
False positive: arguments escalate to lethal force that could
have been averted with “duty to retreat”; cases of mistaken
identity caused by lower expected penalties for being wrong,
etc.
Caveat:
Although deterrence is a theoretical possibility, note that the
goal of the laws was to protect/enhance victim rights, not
deter crime
 Castle Doctrine Laws in US, 2000-2010

Removes duty to Removes duty to retreat Removes

Effective Presumption of
State retreat somewhere in any place one has a civil
Year reasonable fear
outside home legal right to be liability

Alabama 2006 Yes Yes No Yes

Alaska 2006 Yes No Yes Yes
Arizona 2006 Yes Yes Yes Yes
Florida 2005 Yes Yes Yes Yes
Georgia 2006 Yes Yes No Yes
Indiana 2006 Yes Yes No Yes
Kansas 2006 Yes Yes No Yes
Kentucky 2006 Yes Yes Yes Yes
Louisiana 2006 Yes Yes Yes Yes
Michigan 2006 Yes Yes No Yes
Mississippi 2006 Yes Yes Yes Yes
Missouri 2007 Yes No No Yes
Montana 2009 Yes Yes Yes No
North Dakota 2007 Yes No Yes Yes
Ohio 2008 Yes No Yes Yes
Oklahoma 2006 Yes Yes Yes Yes
South Carolina 2006 Yes Yes Yes Yes
South Dakota 2006 Yes Yes No No
Tennessee 2007 Yes Yes Yes Yes
Texas 2007 Yes Yes Yes Yes
West Virginia 2008 Yes Yes No No
 Economic theory concluded

Summary:
21 states passed laws removing “duty to retreat” in places
outside the home
17 states removed “duty to retreat” in any place one had a
legal right to be
13 states include a presumption of reasonable fear
18 states remove civil liability when force was justified under
law
 Cheng and Hoekstra’s identification strategy

Research design:
Estimate the state-level causal effect: what would’ve happened
to these same states had they not passed the law?
Compare the changes in outcomes after castle doctrine law
adoption to changes in the outcomes in other states in the
same region of the country
Estimation: Panel fixed effects estimation

Yit = β1 Di + β2 Tt + β3 (CDLit ) + α1 Xit + ci + ut + εit

CDL is a dummy variable equalling 1 if the state has a castle

doctrine law and zero otherwise
Note: most of their specifications will include region dummies
interacted with year dummies or “region-by-year fixed effects”
which means the estimated coefficient must originate from
variation within a given region (but across states within that
region) in a year
 Data

FBI Uniform Crime Reports (2000-2010)

State-level crime rates, or “offenses per 100,000 population”
Falsification outcomes: motor vehicle theft and larceny
Deterrence outcomes:
Burglary: the unlawful entry of a structure to commit a felony
or a theft
Robbery: the talking or attempting to take anything of value
from the care, custody or control of a person or persons by
force or threat of force or violence and/or putting the victim in
fear
Aggravated assault: unlawful attack by one person upon
another for the purpose of inflicting severe or aggravated
bodily injury
Homicide categories
1 Total homicides – murder plus non-negligent manslaughter
(∼14,000 per year)
2 Justifiable homicides by private citizens (∼250/year)
 Standard Errors

Statistical inference:
Cluster the standard errors at the state level
Are disturbances random draws from individually identical
distribution?
It’s likely that within a state, unobserved determinants of
crime are serially correlated
Bertand, Duflo and Mullainathan (2004) recommend adjusting
for serial correlation in unobserved disturbances within states
at the level of the treatment
“Randomization inference”
“how likely is it that we estimate effects of this magnitude
when using randomly chosen pre-treatment time periods and
randomly assigning placebo treatments?”
Becoming increasingly commonly done
 Region-by-year fixed effects

Absent passing castle doctrine laws, outcomes in these 21

states would have changed similar to other states in their
same region
Recall the “region-by-year fixed effects” in the X term
By including “region-by-year fixed effects”, they are arguing
that unobserved changes in crime are running “parallel” to the
treatment states within region over time
Need not hold across regions since the across region variation
is not being used in this analysis due to the saturation of the
model with “region-by-year fixed effects”
 Testing the identification strategy

Assess how including time-varying controls affected estimates

Examine the effect of the laws on “placebo” outcomes as a
falsification (e.g., automobile thefts and larceny)
Include state-specific linear time trends
Alabama, et al. dummy interacted with TREND which equals
1 in 2000, 2 in 2001, . . . , 11 in 2010
Forces the identification to come from variation in outcomes
around the state-specific linear trend
Stronger requirements, in other words
Outcomes must be large enough and different enough from a
state-specific linear trend otherwise it is collinear with the
state-trend
Include “leads” to test for divergence in the year prior to
adoption
Test for historical tendency for outcomes in adopting states to
diverge from control states
 Control variables

Controls (X matrix in earlier equation)

Full-time police employment per 100,000 state residents from
the LEKOA data (FBI data)
Persons incarcerated in state prison per 100,000 residents
Shares of white/black men in 15-24 and 25-44 age groups
State per capita spending on public assistance
State per capita spending on public welfare
 Summary Statistics

Table 2: Descriptive Statistics

Mean (Unweighted) Mean (Weighted by
Population)
Dependent Variables
Homicides per 100,000 Population 4.8 5.5
(2.5) (1.9)
Justifiable Homicide by Private Citizens (count) 5.1 11.8
(8.2) (12.9)
Justifiable Homicide by Police (count) 8.0 23.4
(16.9) (34.3)
Robberies per 100,000 Population 107.2 143.1
(59.6) (47.5)
Aggravated Assault per 100,000 Population 267 296
(131) (114)
Burglary per 100,000 Population 710 744
(240) (235)
Larceny per 100,000 Population 2,334 2,328
(533) (532)
Motor Theft per 100,000 Population 331 381
(178) (174)
Proportion of Robberies in Which a Gun Was Used 0.35 0.37
(0.13) (0.13)
 Step one: Falsification test

Cheng and Hoekstra (2013) present falsification first to show the reader that
they find no association within region over time in the passage of these laws
and either larceny rate or motor vehicle theft rate
The idea here is to immediately address concerns that what they show you later is
due to generic crime trends in those states that pass the laws
It’s a useful way to assuage doubt people may have, as remember, policy-makers
are not just randomly flipping coins when passing laws, but presumably do so
because of things they observe on the ground
Results will be presented separately under six different specifications
Each new specification adds more controls
What should you expect to find on key variables of interest?
No statistically significant association between the CDL passage and the
placebos; small magnitudes preferably too
No association on the one-year lead either
How do you interpret coefficients?
His model is “log outcomes” regressed onto a dummy variable (level), so these
are semi-elasticities and approximate percentage changes – but you should
transform them by taking the exponential of each coefficient and then
differencing it from one to find the actual percentage change
Ex: CDL = -0.0137 (column 12, Table 3, “Log (larceny rate)” outcome.)
Exp(-0.0137) = 0.986, and so 1-0.986 = 1.4. Thus, CDL reduced larceny rates
by 1.4 percent, which is not statistically significant.
 Results – Falsification Exercise

Table 3: Placebo Tests

OLS - Unweighted
7 8 9 10 11 12
Panel A: Larceny Log (Larceny Rate)
Castle Doctrine Law 0.00745 0.00145 -0.00188 -0.00445 -0.00361 -0.0137
(0.0227) (0.0205) (0.0210) (0.0226) (0.0201) (0.0228)

One Year Before Adoption of -0.0103

Castle Doctrine Law (0.0114)

Observation 550 550 550 550 550 550

Panel B: Motor Vehicle Theft Log (Motor Vehicle Theft Rate)

Castle Doctrine Law 0.0767* 0.0138 0.00814 0.00775 0.00977 -0.00373

(0.0413) (0.0444) (0.0407) (0.0462) (0.0391) (0.0361)

One Year Before Adoption of -0.00155

Castle Doctrine Law (0.0287)

Observation 550 550 550 550 550 550

State and Year Fixed Effects Yes Yes Yes Yes Yes Yes
Region-by-Year Fixed Effects Yes Yes Yes Yes Yes
Time-Varying Controls Yes Yes Yes Yes
Controls for Larceny or Motor Theft Yes
State-Specific Linear Time Trends Yes
Notes: Each column in each panel represents a separate regression. The unit of observation is state-year. Robust standard errors are
clustered at the state level. Time-varying controls include policing and incarceration rates, welfare and public assistance spending, median
income, poverty rate, unemployment rate, and demographics.
 Step two: testing the deterrence hypothesis

Having found no effect on their placebos, Cheng and Hoekstra

(2013) examine the effect of CDL on three deterrence
outcomes: burglary, robbery and aggravated assault
They will, again, have six specifications per outcome in the
“weighted” regression, and then another five for the
“unweighted” regression
What does deterrence look like?
Negative signs on the CDL variable is consistent with
deterrence – these crimes were “deterred”, in other words
Bounds on the magnitudes from the standard errors are used
to provide some confidence about the estimates as well
 Results – Deterrence

OLS - Weighted by State Population OLS - Unweighted

1 2 3 4 5 6 7 8 9 10 11 12
Panel A: Burglary Log (Burglary Rate) Log (Burglary Rate)
Castle Doctrine Law 0.0780*** 0.0290 0.0223 0.0164 0.0327* 0.0237 0.0572** 0.00961 0.00663 0.00277 0.00683 0.0207
(0.0255) (0.0236) (0.0223) (0.0247) (0.0165) (0.0207) (0.0272) (0.0291) (0.0268) (0.0304) (0.0222) (0.0259)

One Year Before Adoption of -0.0201 -0.0154

Castle Doctrine Law (0.0139) (0.0214)
Panel B: Robbery Log (Robbery Rate) Log (Robbery Rate)
Castle Doctrine Law 0.0408 0.0344 0.0262 0.0216 0.0376** 0.0515* 0.0448 0.0320 0.00839 0.00552 0.00874 0.0267
(0.0254) (0.0224) (0.0229) (0.0246) (0.0181) (0.0274) (0.0331) (0.0421) (0.0387) (0.0437) (0.0339) (0.0299)

One Year Before Adoption of -0.0156 -0.0115

Castle Doctrine Law (0.0167) (0.0283)
Panel C: Aggravated Assault Log (Aggravated Assault Rate) Log (Aggravated Assault Rate)
Castle Doctrine Law 0.0434 0.0397 0.0372 0.0362 0.0424 0.0414 0.0555 0.0698 0.0343 0.0305 0.0341 0.0317
(0.0387) (0.0407) (0.0319) (0.0349) (0.0291) (0.0285) (0.0604) (0.0630) (0.0433) (0.0478) (0.0405) (0.0380)

One Year Before Adoption of -0.00343 -0.0150

Castle Doctrine Law (0.0161) (0.0251)
Observations 550 550 550 550 550 550 550 550 550 550 550 550
State and Year Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Region-by-Year Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Time-Varying Controls Yes Yes Yes Yes Yes Yes Yes Yes
Contemporaneous Crime Rates Yes Yes
State-Specific Linear Time Trends Yes Yes
 Conclusion

“In short, these estimates provide strong evidence against the

possibility that castle doctrine laws cause economically
meaningful deterrence effects” (p. 17)
Translation: They can’t find evidence of large deterrence
effects
“Thus, while castle doctrine law may well have benefits to
those legally justified in protecting themselves in self-defense,
there is no evidence that the law provides positive spillovers by
deterring crime more generally” (p. 17)
They note in footnote 24 that they cannot measure the
benefits to victims whose crimes were deterred, or the benefits
from lower legal costs; their focus is limited to whether it
deterred the crimes, not whether the net benefits from the
laws were positive
Obviously, if there is no deterrence, though, then the net
benefits are lower from CDL than they would be if they did
deter
 Step 3: Homicides

The key finding in this study is the very large effect that CDL
had on homicides and non-negligent manslaughter
As the effects are quite large, their strategy is first to present
pictures
The pictures are a bit tricky, though, since they’re going to
also present pictures for the control and treatment group units
This is going to be useful for eye-balling the parallel trends
pre-treatment.
Remember, though – he needs parallel trends within-region –
these figures don’t show that
But you should start with pictures; don’t fetishize regression
 Log Homicide Rates – 2005 Adopter = Florida

2
1.9
1.8
Log Homicide Rate

1.7
1.6
1.5
1.4
1.3
1.2
1.1
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year

Treatment: Florida (law enacted in October 2005)

Control: States that did not enact a law 2000 - 2010
 Log Homicide Rates – 2006 Adopter (13 states)

2
1.9
1.8
Log Homicide Rate

1.7
1.6
1.5
1.4
1.3
1.2
1.1
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year

Treatment: States that enacted the law in 2006

Control: States that did not enact a law 2000 - 2010
 Log Homicide Rates – 2007 Adopter (4 states)

2
1.9
1.8
Log Homicide Rate

1.7
1.6
1.5
1.4
1.3
1.2
1.1
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year

Treatment: States that enacted the law in 2007

Control: States that did not enact a law 2000 - 2010
 Log Homicide Rates – 2008 Adopter (2 states)

2
1.9
1.8
Log Homicide Rate

1.7
1.6
1.5
1.4
1.3
1.2
1.1
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year

Treatment: States that enacted the law in 2008

Control: States that did not enact a law 2000 - 2010
 Log Homicide Rates – 2009 Adopter = Montana

1.5
1.4
1.3
Log Homicide Rate

1.2
1.1
1
.9
.8
.7
.6
.5
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year

Treatment: State that enacted the law in 2009 (Montana)

Control: States that did not enact a law 2000 - 2010
 Residual log homicide rates

.1
.08
Residual Log Homicide Rate

.06
.04
.02
0
-.02
-.04
-.06
-.08
-.1
-6 -5 -4 -3 -2 -1 0 1 2 3 4
Years after passing castle doctrine/stand-your-ground

Treatment: Castle Doctrine/Stand-Your-Ground States

Control: Other States
 Estimation results

Before going into the estimation results, here?s what you are
looking for
This second hypothesis wherein reductions in the expected
penalties and costs associated with self-defense due to CDL
causes lethal violence to increase (non-deterrence escalation of
violence) should exhibit a positive association on the DD
variable
It should be different from zero statistically and economically
meaningful
He will estimate the model using panel fixed effects estimation
and “negative binomial count models”
Because of the smaller number of annual homicides each year
in a state, he moves away from homicide rates in some
specifications and looks at “count” outcomes
He uses a class of estimators more appropriate for “counts”
called “count models”, like the negative binomial estimated
with maximum likelihood
Results are robust to least squares and count models
 Homicide – OLS

1 2 3 4 5 6
Panel A: Log Homicide Rate (OLS - Weighted)

Castle Doctrine Law 0.0801** 0.0946*** 0.0937*** 0.0875** 0.0985*** 0.100**

(0.0342) (0.0279) (0.0290) (0.0337) (0.0299) (0.0388)
One Year Before Adoption of Castle Doctrine -0.0212
Law (0.0246)
Observations 550 550 550 550 550 550
Panel B: Log Homicide Rate (OLS - Unweighted)

Castle Doctrine Law 0.0877 0.0811 0.0600 0.0461 0.0580 0.0672

(0.0638) (0.0769) (0.0684) (0.0764) (0.0662) (0.0450)
One Year Before Adoption of Castle Doctrine -0.0557
Law (0.0494)
Observations 550 550 550 550 550 550

State and Year Fixed Effects Yes Yes Yes Yes Yes Yes
Region-by-Year Fixed Effects Yes Yes Yes Yes Yes
Time-Varying Controls Yes Yes Yes Yes
Contemporaneous Crime Rates Yes
State-Specific Linear Time Trends Yes
 Homicide – Negative Binomial; Murder – OLS

1 2 3 4 5 6
Panel C: Homicide (Negative Binomial - Unweighted)

Castle Doctrine Law 0.0565* 0.0734** 0.0879*** 0.0783** 0.0937*** 0.108***

(0.0331) (0.0305) (0.0313) (0.0355) (0.0302) (0.0346)

One Year Before Adoption of Castle Doctrine -0.0352

Law (0.0260)

Observations 550 550 550 550 550 550

Panel D: Log Murder Rate (OLS - Weighted)

Castle Doctrine Law 0.0906** 0.0955** 0.0916** 0.0884** 0.0981** 0.0813

(0.0424) (0.0389) (0.0382) (0.0404) (0.0391) (0.0520)
One Year Before Adoption of Castle Doctrine -0.0110
Law (0.0230)
Observations 550 550 550 550 550 550

State and Year Fixed Effects Yes Yes Yes Yes Yes Yes
Region-by-Year Fixed Effects Yes Yes Yes Yes Yes
Time-Varying Controls Yes Yes Yes Yes
Contemporaneous Crime Rates Yes
State-Specific Linear Time Trends Yes
 Homicide – identification test

Question: Did homicide rates of adopting states show a general historical

tendency to increase over time relative to other non-adopting states from the
same region?
Method: Move the 11-year panel back one year at a time (covering 1960-2009)
and estimate 40 placebo “effects” of passing CDL 1 to 40 years earlier
Findings

Method Average estimate Estimates larger than actual estimate

Weighted OLS * –0.003 0/40
Unweighted OLS 0.001 1/40
Negative binomial 0.001 0/40
 Interpretation

No evidence that Castle Doctrine/Stand Your Ground Laws

deter violent crimes such as burglary, robbery and aggravated
assault
These laws do lead to an 8% net increase in homicide rates,
translating to around 600 additional homicides per year across
the 21 adopting states
Unlikely that all of the additional homicides were legally
justified
Economics of crime and behavior: incentives matter in some
contexts but not others
Differences-in-differences: Card and Krueger (1994)

Suppose you are interested in the effect of minimum wages on

employment (a classic and controversial question in labor
economics).
In a competitive labor market, increases in the minimum wage
would move us up a downward sloping labor demand curve →
employment would fall
 DD: Card and Krueger (1994)

Card and Krueger (1994) analyzed the effect of a minimum

wage increase in New Jersey using a differences-in-differences
(DD) methodology
In February 1992, New Jersey increased the state minimum
wage from $4.25 to $5.05. Pennsylvania’s minimum wage
stayed at $4.25.
Locations of Restaurants (Card and Krueger 2000)

J. Hainmueller (MIT) 5 / 50

They surveyed about 400 fast food stores both in New Jersey
and Pennsylvania before and after the minimum wage increase
in New Jersey
 DD Strategy

DD is a version of fixed effects estimation. To see this

formally:
1
Yist : employment at restaurant i, state s, time t with a high w min
0
Yist : employment at restaurant i, state s, time t with a low w min
In practice of course we only see one or the other. We then
assume that:
0
E [Yits |s, t] = γs + λt
In the absence of a minimum wage change, employment is
determined by the sum of a time-invariant state effect, γs and
a year effect, λt that is common across states
Let Dst be a dummy for high-minimum wage states and
periods
Assuming E [Yits1 − Y 0 |s, t] = δ is the treatment effect,
its
observed employment can be written:
Yits = γs + λt + δDst + εits
Wages After Rise in Minimum Wage

J. Hainmueller (MIT) 7 / 50
 DD Strategy II

In New Jersey
Employment in February is
E (Yist |s = NJ, t = Feb) = γNJ + λFeb
Employment in November is:
E (Yist |s = NJ, t = Nov ) = γNJ + λNov + δ
Difference between November and February
E (Yist |s = NJ, t = Nov )−E (Yist |s = NJ, t = Feb) = λN −λF +δ
In Pennsylvania
Employment in February is
E (Yist |s = PA, t = Feb) = γPA + λFeb
Employment in November is:
E (Yist |s = PA, t = Nov ) = γPA + λNov
Difference between November and February
E (Yist |s = PA, t = Nov ) − E (Yist |s = PA, t = Feb) = λN − λF
 DD Strategy III

The DD strategy amounts to comparing the change in

employment in NJ to the change in employment in PA.
The population DD are:
 
E (Yist |s = NJ, t = Nov ) − E (Yist |s = NJ, t = Feb)
 
− E (Yist |s = PA, t = Nov ) − E (Yist |s = PA, t = Feb) = δ

This is estimated using the sample analog of the population

means
 TABLE3-AVERAGE EMPLOYMENT PER S
IN NEW JERSEYMI

Stores by state
Difference,
PA NJ NJ-PA
Variable (i) (ii) (iii)
1. FTE employment before,
all available observations
2. FTE employment after,
all available observations
3. Change in mean FTE
employment
4. Change in mean FTE
employment, balanced
sample of storesC
J. Hainmueller (MIT) 17
Surprisingly, employment rose in NJ relative to PA after the
5. Change in mean FTE
minimum wage changesetting
employment,
 Regression DD

We can estimate the DD estimator in a regression framework

Advantages:
It’s easy to calculate the standard errors
We can control for other variables which may reduce the
residual variance (lead to smaller standard errors)
It’s easy to include multiple periods
We can study treatments with different treatment intensity.
(e.g., varying increases in the minimum wage for different
states)
The typical regression model we estimate is

Outcomeit = β1 + β2 Treati + β3 Postt + β4 (Treat × Post)it + εit

where Treat is a dummy if the observation is in the treatment

group and Post is a post treatment dummy
 Regression DD - Card and Krueger

In the Card and Krueger case, the equivalent regression would

be:
Yits = α + γNJs + λdt + δ(NJ × d)st + εits

NJ is a dummy equal to 1 if the observation is from NJ

d is a dummy equal to 1 if the observation is from November
(the post period)
This equation takes the following values
PA Pre: α
PA Post: α + λ
NJ Pre: α + γ
NJ Post: α + γ + λ + δ
DD estimate: (NJ Post - NJ Pre) - (PA Post - PA Pre) = δ
Graph - Observed Data

Waldinger (Warwick) 19 / 55
Graph - DD

YistY= =+
ist α γNJ
a+ s +
gNJ s +λd
ldt t+ d(NJs× d)
+δ(NJ dt )st++#ist
εist

Waldinger (Warwick) 20 / 55
Graph - DD

YistYist
== α+
a+γNJ
gNJs s++λd
ldtt + d(NJs ×
+ δ(NJ  dd) ++
t )st #istεist

Waldinger (Warwick) 21 / 55
Graph - DD

YYistist ==αa + gNJ +ld

γNJss + NJs ×
λdt t++d(δ(NJ dt d) + εist
) +st#ist

Waldinger (Warwick) 22 / 55
Graph - DD

Yist
Yist==αa+ gNJss + ld
+γNJ λdtt + (NJs ×dtd)
+dδ(NJ )+st #+
ist εist

Waldinger (Warwick) 23 / 55
Key assumption of any DD strategy: Parallel trends

The key assumption for any DD strategy is that the outcome

in treatment and control group would follow the same time
trend in the absence of the treatment
This doesn’t mean that they have to have the same mean of
the outcome
Parallel trends are difficult to verify because technically one of
the parallel trends is an unobserved counterfactual
But one often will check
Key Assumption of Any using pre-treatment
DD Strategy: Common Trends data to show

that the trends

The are the same
key assumption for any DD strategy is that the outcome in
treatment and control group would follow the same time trend in the

Even if pre-trends are the same one still has to worry about
absence of the treatment.
This does not mean that they have to have the same mean of the
other policiesoutcome!
changing at the same time (omitted variable
Common trend assumption is di¢cult to verify but one often uses
bias) pre-treatment data to show that the trends are the same.
Even if pre-trends are the same one still has to worry about other
policies changing at the same time.

Waldinger (Warwick) 24 / 55
 Regression DD Including Leads and Lags

Including leads into the DD model is an easy way to analyze

pre-treatment trends
Lags can be included to analyze whether the treatment effect
changes over time after assignment
The estimated regression would be:
−1
X m
X
Yits = γs + λt + γτ Dsτ + δτ Dsτ + xist + εist
τ =−q τ =0

Treatment occurs in year 0

Includes q leads or anticipatory effects
Includes m leads or post treatment effects
 Study including leads and lags – Autor (2003)

Autor (2003) includes both leads and lags in a DD model

analyzing the effect of increased employment protection on
the firm’s use of temporary help workers
In the US employers can usually hire and fire workers at will
Some states courts have made some exceptions to this
employment at will rule and have thus increased employment
protection
The standard thing to do is normalize the adoption year to 0
Autor then analyzes the effect of these exemptions on the use
of temporary help workers.
Results Results

The leads are very close to 0. ! no evidence for anticipatory e§ects

(good news for the common trends assumption).
TheTheleads arethat
lags show verytheclose
e§ect to 0. →during
increases no evidence forof anticipatory
the first years the
effects (good
treatment news
and then for the
remains parallel
relatively trends assumption).
constant.

The lags show the effect increases during the first years of the
treatment and then remains relatively constant.
Waldinger (Warwick) 27 / 55
 Standard errors in DD strategies

Many paper using DD strategies use data from many years –

not just 1 pre and 1 post period
The variables of interest in many of these setups only vary at
a group level (say a state level) and outcome variables are
often serially correlated
In the Card and Krueger study, it is very likely that
employment in each state is not only correlated within the
state but also serially correlated
As Bertrand, Duflo and Mullainathan (2004) point out,
conventional standard errors often severely understate the
standard deviation of the estimators – standard errors are
biased downward
 Standard errors in DD – practical solutions

Bertrand, Duflo and Mullainathan propose the following

solutions:
1 Block bootstrapping standard errors (if you analyze states the
block should be the states and you would sample whole states
with replacement for bootstrapping)
2 Clustering standard errors at the group level (in STATA one
would simply add , cluster(state) to the regression
equation if one analyzes state level variation)
3 Aggregating the data into one pre and one post period.
Literally works if there is only one treatment data. With
staggered treatment dates one should adopt the following
procedure:
Regress Yst onto state FE, year FE and relevant covariates
Obtain residuals from the treatment states only and divide
them into 2 groups: pre and post treatment
Then regress the two groups of residuals onto a post dummy
Correct treatment of standard errors sometimes makes the
number of groups very small: in the Card and Krueger study
the number of groups is only 2.
 Threats to validity

1 Non-parallel trends
2 Compositional differences
3 Long-term effects vs. reliability
4 Functional form dependence
 Non-parallel trends

Often policymakers will select the treatment and controls

based on pre-existing differences in outcomes – practically
guaranteeing the parallel trends assumption will be violated.
“Ashenfelter dip”
Named after Orley Ashenfelter, labor economist at Princeton
Participants in job trainings program often experience a “dip”
in earnings just prior to entering the program
Since wages have a natural tendency to mean reversion,
comparing wages of participants and non-participants using
DD leads to an upward biased estimate of the program effect.
Regional targeting. NGOs may target villages that appear
most promising, or worse off, which is a form of selection bias
and violates parallel trends
 Checks for DD Design

Very common for readers and others to request a variety of

“robustness checks” from a DD design
Think of these as along the same lines as the leads and lags
we already discussed
Falsification test using data for prior periods (already
discussed)
Falsification test using data for alternative control group
Falsification test using alternative “placebo” outcome that
should not be affected by the treatment
 Di↵erence-in-Di↵erences: Threats to Validity
Alternative control group – DDD
Triple DDD: Mandated Maternity Benefits (Gruber, 1994
 DDD in Regression

Wijt = α + β1 Xijt + β2 τt + β3 δj + β4 Di + β5 (δ × τ )jt

+ β6 (τ × D)ti + β7 (δ × D)ij + β8 (δ × τ × D)ijt + εijt

The DDD estimate is the difference between the DD of

interest and a placebo DD (which is supposed to be zero)
If the placebo DD is non-zero, it might be difficult to convince
the reviewer that the DDD removed all the bias
If the placebo DD is zero, then DD and DDD give the same
results but DD is preferable because standard errors are
smaller for DD than DDD
 Falsification test with placebo outcome

Cheng and Hoekstra (2013) examine the effect of castle

doctrine gun laws on homicides. They investigate the effect of
the laws on non-gun related offenses like grand theft auto and
find no evidence of an effect (“second order outcomes”)
Auld and Grootendorst (2004) estimated standard “rational
addiction” models (Becker and Murphy 1988) on data with
milk, eggs, oranges and apples. They find these plausibly
non-addictive goods are addictive, which casts doubt on the
research design of all the rational addiction models.
Several studies found significant network effects on outcomes
like obesity, smoking, alcohol use and happiness. Cohen-Cole
and Fletcher (2008) use similar models and data and find
similar network “effects” for things that aren’t contagious like
acne, height and headaches
 Threats to validity – compositional differences

One of the risks of a repeated cross-section is that the

composition of the sample may have changed between the pre
and post period
Hong (2011) uses repeated cross-sectional data from the
Consumer Expenditure Survey (CEX) containing music
expenditure and internet use for a random sample of
households
Study exploits the emergence of Napter (first file sharing
software widely used by Internet users) in June 1999 as a
natural experiment
Study compares internet users and internet non-users before
and after emergence of Napster
Compositional di↵erences?

Figure 1: Internet Di↵usion and Average Quarterly Music Expenditure in the CEX
40 100

Non-user Group Internet User Group Internet Diffusion 90

35

% of HHs w/Internet connection

80
Average Music Expenditure

30
70
(in 1998 dollars)

25
60

20 50

40
15

30
10
20

5
10

0 0
1996 1997 1998 1999 2000 2001
Year

J. Hainmueller (MIT) 41 / 50
 Table 1: Descriptive Statistics for Internet User and Non-user Groupsa

Year 1997 1998 1999 2000

Internet User Non-user Internet User Non-user Internet User Non-user Internet User
Average Expenditure
Recorded Music $25.73 $10.90 $24.18 $9.97 $20.92 $9.37 $17.42
Entertainment $195.03 $96.71 $193.38 $84.92 $182.42 $80.19 $164.88
Zero Expenditure
Recorded Music .56 .79 .60 .80 .64 .81 .68
Entertainment .08 .32 .09 .35 .14 .39 .17
Demographics
Age 40.2 49.0 42.3 49.0 44.1 49.4 44.3
Income $52,887 $30,459 $51,995 $28,169 $49,970 $26,649 $47,510
High School Grad. .18 .31 .17 .32 .21 .32 .22
Some College .37 .28 .35 .27 .34 .27 .36
College Grad. .43 .21 .45 .21 .42 .20 .37
40

Manager .16 .08 .16 .08 .14 .08 .14

Professional .23 .11 .22 .10 .21 .10 .19
Living in a Dorm .12 0 .08 0 .05 0 .05
Di↵usion of the internet changes samples (e.g. younger music fans are
Urban .93 .87 .93 .86 .91 .87 .89
Diffusion
early of the Internet changes samples (e.g., younger music fans are early
adopters)
Inside a MSA .84 .78 .83 .78 .83 .78 .81
Pop. Size > 4 million .34 .26 .30 .26 .31 .25 .28
adopters)
Appliance Ownership
Computer .79 .27 .81 .28 .80 .28 .81
Sound System .81 .57 .79 .58 .78 .56 .76
J. Hainmueller (MIT) 42 / 50
VCR .83 .72 .86 .74 .86 .72 .85
Total Households
(in million) 15 91 22 86 28 80 34
Observations 3,163 19,052 5,624 21,550 8,191 22,810 9,606
a
All the statistics are weighted using the weights provided by the CEX. Years refer to the period from June of the year to May of the next y
households are computed by summing the CEX weights.
 Comparative case studies

In qualitative comparative case studies, the goal is to reason

inductively the causal effects of events or characteristics of a
single unit on some outcome, but oftentimes through logic
and historical analysis.
May not answer the causal questions at all because there is
rarely a counterfactual, or if so, it’s ad hoc.
Classic example of comparative case study approach is Alexis
de Toqueville’s Democracy in America
Quantitative comparative case studies are more explicitly
causal designs. Usually a natural experiment applied to a
single aggregate unit (e.g., city, school, firm, state, country)
Comparative case studies: compare the evolution of an
aggregate outcome for the unit affected by the intervention to
the evolution of the same aggregate for some control group
(Card 1990; Card and Krueger 1994; Abadie and Gardeazabal
2003)
 Motivating example: The Mariel Boatlift

How do inflows
Motivating of immigrants
Example: The Marielaffect the wages and
Boatlift
employment of natives in local labor markets?
How do inflows of immigrants a↵ect the wages and
Card (1990) usesofthe
employment Mariel
natives boatlift
in local of 1980 as a natural
labor markets?
experiment to measure
Card (1990) the effect
uses the Mariel Boatliftofof a1980
sudden influx of
as a natural
immigrants on unemployment
experiment to measure the e↵ectamong less-skilled
of a sudden influx of natives
immigrants on unemployment among less-skilled natives
 Motivating example: The Mariel Boatlift

How do inflows of immigrants affect the wages and

employment of natives in local labor markets?
Card (1990) uses the Mariel boatlift of 1980 as a natural
experiment to measure the effect of a sudden influx of
immigrants on unemployment among less-skilled natives
The Mariel Boatlift increased the Miami labor force by 7%
Individual-level data on unemployment from the Current
Population Survey (CPS) for Miami and four comparison
cities (Atlanta, Los Angeles, Houston, Tampa-St. Petersburg)
( ~\ i
-..~
0\
---
V Vii'}
--~
1J')NOO
~
V'
- ~ci- N"':N
Motivating example: the Mariel Boatlift

"'"'" "'"'"
- I / C'i~~
-0- C1C1o.
-N-
~
O\C;:) I I I I
,
vi
u
~
u
.r:
c
~
00
~ - r;::; ~ ~ ~
O\MO\ 000\00.
ft ddd "';:dN c
'-' '-' '-" ,-
O\MV \0\00 C
~
M~d o\NM ~
Motivating Example: The Mariel Boatlift

E -
~ OO~ - 0
-00. --
O\N I I.r: ~
u
a ~
d
U, ~
C
L..
='
0
t:
C
0
',----M- --~
r-..000\"d
0u
.-0
c:
- 0 ... - - 0 ... - ~C
K ~ . -vr-..
-""""" MMO
\~ '/ .-00 ~ 0\ . V .. 0
1J') 00'" 0 N g
au t--
~c.s
- C/)
.-
e >c I """
\0
10..,
"d
~ ~
~uU M
rl
C c: :0
U
0 0 ~
.I::
...
10.., .~.~~
0.
~
0..
~
L.
0
0
a E 0\
--
'a
0 0 0\
~
~ U
I Y
~
';j
II)
u
'-.-
u 6
.~
U

(I) .
,-
~

u
~
.~
u'-

~
~
~
'-'
a
r \
'-'
~ 0.2-: c.2-:
OGJ
a
0
g °u
~
U '~ U
C
~
.~ U
C ~
.L..
~F;0..L.. U
I£: -
eU~o..~
L..
u,.. II) .- U ~ ,;... U
"0
~ R
=' ,~ ~ ,- a ~ u ;a a ~
0 ~ ~ .- 0 ~ 0..
t!: (,5 ~~UQ ~~UQ ~
<
';,
,.
: GJ
II)
II)
; U
C U
0
' U -0
Z
'I!~
'
,
.-
~
GJ
, ---N ---" ,
.
or
l~ Q ~_c ~~~
,
\
:)
 Comparative case studies

Advantages:
Policy interventions often take place at an aggregate level
Aggregate/macro data are often available
Problems:
Selection of control group is often ambiguous
Standard errors do not reflect uncertainty about the ability of
the control group to reproduce the counterfactual of interest
 Synthetic Control method

A distinctive feature of comparative case studies is that units

of analysis are usually aggregate entities, like countries or
regions, for which suitable single comparisons often do not
exist
The synthetic control method is based on the observation
that, when the units of analysis are a few aggregate entities, a
combination of comparison units (a “synthetic control”) often
does a better job reproducing the characteristics of a treated
unit than any single comparison unit alone.
Motivated by this consideration, the comparison unit in the
synthetic control method is selected as the weighted average
of all potential comparison units that best resembles the
characteristics of the treated unit(s)
 Synthetic control method: advantages

Precludes extrapolation
Does not require access to post-treatment outcomes in the
“design” phase of the study, when synthetic controls are
calculated
Makes explicit the contribution of each comparison unit to the
counterfactual of interest
Allows researchers to use quantitative and qualitative
techniques to analyze the similarities and differences between
the units representing the case of interest and the synthetic
control
Formalizing the way comparison units are chosen not only
represents a way of systemizing comparative case studies, it
also has direct implications for inference
 Synthetic control method: estimation

Suppose that we observe J + 1 units in periods 1, 2, . . . , T

Unit “one” is exposed to the intervention of interest (that is,
“treated”) during periods T0 + 1, . . . , T
The remaining J are an untreated reservoir of potential
controls (a “donor pool”)
Let YitN be the outcome that would be observed for unit i at
time t in the absence of the intervention
Let YitI be the outcome that would be observed for unit i at
time t if unit i is exposed to the intervention in periods
T0 + 1 to T .
We aim to estimate the effect of the intervention on the
treated unit (α1T0 +1 , . . . , α1T ) where

α1t = Y1tI − Y1tN = Y1t − Y1tN

for t > T0 and Y1t is the outcome for unit one at time t
 Synthetic control method: implementation

Let W = (w2 , . . . , wJ+1 )0 with wj ≥ 0 for j = 2, . . . , J + 1

and w2 + · · · + wj+1 = 1. Each value of W represents a
potential synthetic control
Let X1 be a (k × 1) vector of pre-intervention characteristics
for the treated unit. Similarly, let X0 be a (k × J) matrix
which contains the same variables for the unaffected units.
The vector W ∗ = (w2∗ , . . . , wJ+1
∗ )0 is chosen to minimize

||X1 − X0 W ||, subject to our weight constraints

Let Yjt be the value of the outcome for unit j at time t. For a
post-intervention period t (with t ≥ T0 ) the synthetic control
estimator is:
J+1
X
b1t = Y1t −
α wj∗ Yjt
j=2
 Synthetic control method: estimation

That is to reproduce the counterfactual YitN we find the

combination of untreated units that best resembles the
treated unit before the intervention in terms of the values of k
relevant covariates (predictors of the outcome of interest)
Example: 1988 California’s tobacco control program
(Proposition 99):
Treated unit: California
Outcome of interest: tobacco consumption in California after
1988
Potential controls: other US states
Covariates: predictors of state-level tobacco consumption
measured before 1988
 Synthetic control method: estimation

Abadie, et al. consider

p
||X1 − X0 W ||= (X1 − X0 W )0 V (X1 − X0 W ) where V is
some (k × k) symmetric and positive semidefinite matrix
Let Xjm be the value of the m-th covariates for unit j
Typically, V is diagonal with main diagonal v1 , . . . , vk . Then,
the synthetic control weights w2∗ , . . . , wJ+1
∗ minimize:

k
X  J+1
X 2
vm X1m − wj Xjm
m=1 j=2

where vm is a weight that reflects the relative importance that

we assign to the m-th variable when we measure the
discrepancy between the treated unit and the synthetic
controls
 Synthetic control estimation: estimation

The choice of V is important

W ∗ depends on the choice of V
The synthetic control W ∗ (V ) is meant to reproduce the
behavior of the outcome variable for the treated unit in the
absence of the treatment
Therefore, the weights v1 , . . . , vk should reflect the predictive
value of the covariates
 Synthetic control estimation: estimation

Choice of v1 , . . . , vk can be based on

Subjective assessment of the predictive power of each of the
covariates, or calibration inspecting how different values for
v1 , . . . , vk affect the discrepancies between the treated unit
and the synthetic control
Use regression to assess the predictive power of the covariates
Minimize mean square prediction error (MSPE):
T0 
X J
X 2
Y1t − wj∗ (V )Yjt
t=1 j=2

Cross-validation
Divide the pre-treatment period into an initial training period
and a subsequent validation period
For any given V , calculate W ∗ (V ) in the training period.
Minimize the MSPE of W ∗ (V ) in the validation period
 What about unobserved factors?

Comparative case studies are complicated by unmeasured

factors affecting the outcome variables as well as
heterogeneity in the effect of observed and unobserved factors
However, as we will see, if the number of pre-intervention
periods in the data is large, matching on pre-intervention
outcomes allows us to control for heterogeneous responses to
multiple unobserved factors
Intuition: only units that are alike in observed and unobserved
determinants of the outcome variable as well as in the effect
of those determinants on the outcome variable should produce
similar trajectories of the outcome variable over extended
periods of time
 The Application: California’s Proposition 99

In 1988, California first passed comprehensive tobacco control

legislation:
increased cigarette tax by 25 cents/pack
earmarked tax revenues to health and anti-smoking budgets
funded anti-smoking media campaigns
spurred clean-air ordinances throughout the state
produced more than $100 million per year in anti-tobacco
projects
Other states that subsequently passed control programs are
excluded from donor pool of controls (AK, AZ, FL, HI, MA,
MD, MI, NJ, OR, WA, DC)
Cigarette Consumption: CA and the Rest of the US

Cigarette Consumption: CA and the Rest of the U.S.

140
California
rest of the U.S.

120
per−capita cigarette sales (in packs)

100
80
60
40

Passage of Proposition 99
20
0

1970 1975 1980 1985 1990 1995 2000

year
 Cigarette Consumption: CA and synthetic CA
Cigarette Consumption: CA and synthetic CA

140
California
synthetic California

120
per−capita cigarette sales (in packs)

100
80
60
40

Passage of Proposition 99
20
0

1970 1975 1980 1985 1990 1995 2000

year
Predictor Means:
Predictor ActualActual
Means: vs. Synthetic CaliforniaCalifornia
vs. Synthetic

California Average of
Variables Real Synthetic 38 control states
Ln(GDP per capita) 10.08 9.86 9.86
Percent aged 15-24 17.40 17.40 17.29
Retail price 89.42 89.41 87.27
Beer consumption per capita 24.28 24.20 23.75
Cigarette sales per capita 1988 90.10 91.62 114.20
Cigarette sales per capita 1980 120.20 120.43 136.58
Cigarette sales per capita 1975 127.10 126.99 132.81
Note: All variables except lagged cigarette sales are averaged for the 1980-
1988 period (beer consumption is averaged 1984-1988).
 Smoking Gap between CA and synthetic CA
Smoking Gap Between CA and synthetic CA

30
gap in per−capita cigarette sales (in packs)

20
10
0
−10
−20

Passage of Proposition 99
−30

1970 1975 1980 1985 1990 1995 2000

year
 Inference

To evaluate the significance of our results, we conduct a series

of placebo experiments where we reassign the tobacco control
program to states other than California
We proceed as follows:
Iteratively apply the synthetic method to each country/state in
the donor pool and obtain a distribution of placebo effects
Compare the gap (RMSPE) for California to the distribution of
the placebo gaps. For example the post-Prop. 99 RMSPE is:
 T  J+1 2  12
1 X X
∗
RMSPE = Y1t − wj Yjt
T − T0
t=T0 +1 j=2

Question is whether the effect estimated by the synthetic

control for California is large relative to the effect estimated for
a state chosen at random
 Smoking Gap for CA and 38 control states

Smoking Gap for CA and 38 control states

(All States in Donor Pool)
California

30
control states
gap in per−capita cigarette sales (in packs)

20
10
0
−10
−20

Passage of Proposition 99
−30

1970 1975 1980 1985 1990 1995 2000

year
 Smoking Gap for CA and 34 control states

Smoking Gap for CA and 34 control states

(Pre-Prop. 99 MSPE  20 Times Pre-Prop. 99 MSPE for CA)
California

30
control states
gap in per−capita cigarette sales (in packs)

20
10
0
−10
−20

Passage of Proposition 99
−30

1970 1975 1980 1985 1990 1995 2000

year
 Smoking Gap for CA and 29 control states

Smoking Gap for CA and 29 control states

(Pre-Prop. 99 MSPE  5 Times Pre-Prop. 99 MSPE for CA)
California

30
control states
gap in per−capita cigarette sales (in packs)

20
10
0
−10
−20

Passage of Proposition 99
−30

1970 1975 1980 1985 1990 1995 2000

year
 Smoking Gap for CA and 19 control states

Smoking Gap for CA and 19 control states

(Pre-Prop. 99 MSPE  2 Times Pre-Prop. 99 MSPE for CA)
California

30
control states
gap in per−capita cigarette sales (in packs)

20
10
0
−10
−20

Passage of Proposition 99
−30

1970 1975 1980 1985 1990 1995 2000

year
Ratio Post-Prop. 99 RMSPE to Pre-Prop. 99 RMSPE

Ratio Post-Prop. 99 MSPE to Pre-Prop. 99 MSPE

(All 38 States in Donor Pool)

5
4
3
frequency

California
2
1
0

0 20 40 60 80 100 120

post/pre−Proposition 99 mean squared prediction error

 The Application: The 1990 German Reunification

Cross-country regressions are often criticized because they put

side-by-side countries of very different characteristics
“What do Thailand, the Dominican Republic, Zimbabwe,
Greece and Bolivia have in common that merits their being
put in the same regression analysis? Answer: For most
purposes, nothing at all.” (Harberger 1987)
The synthetic control method provides a data-driven
procedure to select a comparison unit
Application: the economic impact of the 1990 German
reunification in West Germany
Donor pool is restricted to 16 OECD countries
 West Germany and the OECD sample

West Germany and the OECD sample

10000 15000 20000 25000 30000

per−capita GDP (PPP, 2002 USD)

reunification
5000

West Germany
rest of the OECD sample
0

1960 1970 1980 1990 2000

year
Covariate Averages before
Covariate 1990
averages before 1990

West Synthetic OECD

Germany West Germany Sample
GDP per-capita 15808.9 15800.9 8021.1
Trade openness 56.8 56.9 31.9
Inflation rate 2.6 3.5 7.4
Industry share 34.5 34.4 34.2
Schooling 55.5 55.2 44.1
Investment rate 27.0 27.0 25.9
 West Germany and synthetic West Germany

West Germany and synthetic West Germany

10000 15000 20000 25000 30000

per−capita GDP (PPP, 2002 USD)

reunification
5000

West Germany
synthetic West Germany
0

1960 1970 1980 1990 2000

year
Country Weights in the Synthetic West Germany
Country Weights in the Synthetic West Germany

Country Weight Country Weight

Australia 0 Netherlands 0.10
Austria 0.42 New Zealand 0
Belgium 0 Norway 0
Denmark 0 Portugal 0
France 0 Spain 0
Greece 0 Switzerland 0.11
Italy 0 United Kingdom 0
Japan 0.16 United States 0.22
Post-Reunification to Pre-Reunification RMSPE Ratio

Post-Reunification to Pre-Reunification RMSE Ratio

West Germany ●

Italy ●

Australia ●

Norway ●

Greece ●

Netherlands ●

USA ●

New Zealand ●

Spain ●

Belgium ●

UK ●

Switzerland ●

Japan ●

France ●

Denmark ●

Austria ●

Portugal ●

5 10 15

Post−Period RMSE / Pre−Period RMSE

 Placebo Reunification: 1975
Placebo Reunification: 1975

10000 15000 20000 25000 30000

per−capita GDP (PPP, 2002 USD)

placebo reunification
5000

West Germany
synthetic West Germany
0

1960 1965 1970 1975 1980 1985 1990

year
 Comparison to Regression

Constructing a synthetic comparison as a linear combination

of the untreated units with coefficients that sum to one may
appear unusual
However in fact a regression-based approach does exactly this,
albeit in an implicit way
In contrast to the synthetic control method, the regression
approach does not restrict the coefficients of the linear
combination that define the comparison unit to be in between
zero and one, therefore allowing extrapolation outside the
support of the data
 Comparison to regression

Let T1 be the number of post-intervention periods and:

X1 : (k × 1)-matrix of covariates for treated unit

X0 : (k × J)-matrix of covariates for control units
Y0 : (T1 × J)-matrix of post-intervention outcomes for control units

Let
Bb = (X0 X00 )−1 X0 Y00
be the (k × T1 ) matrix of regression coefficients of Y0 on X0
That is, each column of Bb contains the regression coefficients of Y0 on X0 for a
post-intervention period.
A regression-based counterfactual of the outcome for the treated unit in
absence of the treatment is given by the (T1 × 1) vector Bb0 X1
 Comparison to regression

Notice that
Bb0 X1 = Y0 W reg
where
W reg = X00 (X0 X00 )−1 X1
As a result, the regression-based estimate of the
counterfactual of interest is a linear combination of
post-treatment outcomes for the untreated units, with weights
W reg
Let ι be a (J × 1)vector of ones. The sum of the regression
weights is ι0 W reg . It can be proven that

ι0 W reg = 1
 Synthetic vs. Regression Weights

Synthetic Regression Synthetic Regression

Country Control Weight Weight Country Control Weight Weight
Australia 0 0.12 Netherlands 0.10 0.14
Austria 0.42 0.26 New Zealand 0 0.12
Belgium 0 0 Norway 0 0.04
Denmark 0 0.08 Portugal 0 -0.08
France 0 0.04 Spain 0 -0.01
Greece 0 -0.09 Switzerland 0.11 0.05
Italy 0 -0.05 UK 0 0.06
Japan 0.16 0.19 USA 0.22 0.13
 Overview
Part 1: The Core
Part 2: Selection on observables
Part 3: Selection on unobservables Concluding remarks
Part 4: Advanced material
Conclusion
References

Final thoughts

A good research design is one you are excited to tell people about – that’s
basically what characterizes all research designs, whether propensity score
matching or regression discontinuity designs, in some respects
Most important thing is to be honest with yourself and the reader
Always check for covariate balance in everything that you do
Causality is easy and hard. Don’t get confused which is the hard part and
which is the easy part. Don’t get enamored by statistical modeling that
obscures the identification problem from plain sight. Always understand what
assumptions you must make, be clear which parameters you are and are not
identifying, and don’t be afraid of your answers.

Cunningham Causal Inference

 Overview
Part 1: The Core
Part 2: Selection on observables
Part 3: Selection on unobservables
Part 4: Advanced material
Conclusion
References

Articles and Books on the Syllabus I

Cunningham Causal Inference