ECON 7310 Elements of Econometrics

Lecture 4: Linear Regression with Multiple Regressors

Fu Ouyang

University of Queensland

1 / 30
Outline

▶ Omitted variable bias

▶ Causality and regression analysis
▶ Multiple regression and OLS
▶ Measures of fit
▶ Sampling distribution of the OLS estimator

2 / 30
Omitted Variable Bias (SW Section 6.1)

▶ The error u arises because of factors, or variables, that influence Y but

are not included in the regression function. There are always omitted
variables.
▶ Sometimes, the omission of those variables can lead to bias in the OLS
estimator.
▶ The bias in the OLS estimator that occurs as a result of an omitted
factor, or variable, is called omitted variable bias
▶ For omitted variable bias to occur, the omitted variable Z must satisfy the
following two conditions:
1. Z is a determinant of Y (i.e. Z is part of u); and
2. Z is correlated with the regressor X (i.e., corr (Z , X ) ̸= 0)
▶ Both conditions must hold for the omission of Z to result in omitted
variable bias, i.e., OLS estimators are biased and inconsistent.

3 / 30
In the test score example:

1. English language ability (whether the student has English as a second

language) plausibly affects standardized test scores: Z is a determinant
of Y , i.e., Z is part of u.
2. Immigrant communities tend to be less affluent and thus have smaller
school budgets and higher STR: Z is correlated with X .
Accordingly, βb1 is biased. What is the direction of this bias?

4 / 30
In the test score example:

Suppose that the true model is given as

TestScorei = β0 + β1 STRi + β2 Zi + ei

where Zi is the proportion of ESL students in district i. Now, let’s assume that
the following stories are reasonable;
▶ STR ↑⇒ TestScore ↓. That is, β1 < 0
▶ Zi ↑⇒ English skill ↓ ⇒ TestScorei ↓. So, Zi is part of ui . Indeed β2 < 0
▶ Zi ↑⇒ Educ Budget ↓ ⇒ STRi ↑. So, corr (Zi , STRi ) > 0
Hence, if the equation without Zi is estimated,

TestScorei = β0 + β1 STRi + ui ,
|{z}
=β2 Zi +ei

the effect of Zi on TestScore will be partially absorbed into the effect of STR
on TestScore.
That is, OLS estimate for β1 will overestimate the effect of STR on TestScore.

5 / 30
What does the sample say about this?

▶ Districts with fewer English Learners have higher test scores

▶ Districts with lower percent EL (PctEL) have smaller classes
▶ Among districts with comparable PctEL, the effect of class size is small
(recall overall “test score gap” = 7.4)

6 / 30
Causality and regression analysis

▶ This example (test score/STR/fraction English Learners) shows that, if

an omitted variable satisfies the two conditions for omitted variable bias,
then the OLS estimator in the regression omitting that variable is biased
and inconsistent. So, even if n is large, βb1 will not be close to β1 .
▶ This raises a deeper question: how do we define β1 ? That is, what
precisely do we want to estimate when we run a regression?

7 / 30
What precisely do we want to estimate when we run a regression?

There are (at least) three possible answers to this question:

1. We want to estimate the slope of a line through a scatter plot as a simple
summary of the data to which we attach no substantive meaning.
▶ This can be useful at times, but isn’t interesting intellectually and isn’t what
this course is about.
2. We want to make forecasts, or predictions, of the value of Y for an entity
not in the data set, for which we know the value of X .
▶ Forecasting is an important job for economists, and can be done by
regression methods without considering causal effects.
3. We want to estimate the causal effect on Y of a change in X.
▶ This is why we are interested in the class size effect. Suppose the school
decided to cut class size by 2 students. What would be the effect on test
scores? This is a causal question (what is the causal effect of STR on test
scores?).
▶ Except when we discuss forecasting, the aim of this course is the estimation
of causal effects using regression methods.

8 / 30
What is a causal effect?

▶ “Causality” is a complex concept! In this course, we take a practical

approach to defining causality:
▶ A causal effect is defined to be the effect measured in an ideal
randomized controlled experiment.
▶ Ideal: subjects all follow the treatment protocol – perfect compliance, no
errors in reporting, etc.!
▶ Randomized: subjects from the population of interest are randomly
assigned to a treatment or control group (no confounding factors)
▶ Controlled: having a control group permits measuring the differential effect
of the treatment
▶ Experiment: the treatment is assigned as part of the experiment: the
subjects have no choice, so there is no “reverse causality” in which subjects
choose the treatment they think will work best.

9 / 30
Back to class size

Imagine an ideal randomized controlled experiment for measuring the effect

on Test Score of reducing STR.
▶ In that experiment, students would be randomly assigned to classes,
which would have different sizes.
▶ Because they are randomly assigned, all student characteristics (and
thus ui ) would be distributed independently of STRi .
▶ Thus, E(ui |STRi ) = 0, that is, LSA #1 holds in a randomized controlled
experiment.

10 / 30
How does our observational data differ from this ideal?

▶ The treatment is often not randomly assigned

▶ Consider PctEL – percent English learners – in the district. It plausibly
satisfies the two criteria for omitted variable bias: Z = PctEL is:
1. a determinant of Y ; and
2. correlated with the regressor X .
▶ Thus, the “control” and “treatment” groups differ in a systematic way, so
corr (STR, PctEL) ̸= 0.
▶ This means that E(ui |STRi ) ̸= 0 because PctEL is included in u and
LSA #1 is violated.

11 / 30
▶ (Randomization + control group) ⇒ any differences between the
treatment and control groups are random – not systematically related to
the treatment
▶ We can eliminate the difference in PctEL between the large (control) and
small (treatment) groups by examining the effect of class size among
districts with the same PctEL.
▶ If the only systematic difference between the large and small class size
groups is in PctEL, then we are back to the randomized controlled
experiment – within each PctEL group.
▶ This is one way to control for the effect of PctEL when estimating the effect
of STR.

12 / 30
Return to omitted variable bias

Three ways to overcome omitted variable bias;

1. Run a randomized controlled experiment in which treatment (STR) is
randomly assigned: then PctEL is still a determinant of TestScore, but
PctEL is uncorrelated with STR. (This solution to Omitted Variable bias
is rarely feasible.)
2. Adopt the “cross tabulation” approach, with finer gradations of STR and
PctEL – within each group, all classes have the same PctEL, so we
control for PctEL (But soon you will run out of data, and what about other
determinants like family income and parental education?)
3. Use a regression in which the omitted variable (PctEL) is no longer
omitted: include PctEL as an additional regressor in a multiple
regression.

13 / 30
The Population Multiple Regression Model (SW Section 6.2)

▶ Consider the case of two regressors:

Yi = β0 + β1 X1i + β2 X2i + ui , i = 1, . . . , n
▶ Y is the dependent variable (or LHS variable)
▶ X1 , X2 are the two independent variables (regressors, RHS variables)
▶ (Yi , X1i , X2i ) denote the i th observation on Y , X1 , and X2 .
▶ β0 = unknown population intercept
▶ β1 = effect on Y of a change in X1 , holding X2 constant
▶ β2 = effect on Y of a change in X2 , holding X1 constant
▶ ui = the regression error (omitted factors)

14 / 30
Interpretation of coefficients in multiple regression

Yi = β0 + β1 X1i + β2 X2i + ui , i = 1, . . . , n

▶ Consider changing X1 by ∆X1 while holding X2 constant:

▶ Population regression line before the change:

Y = β0 + β1 X1 + β2 X2
▶ Population regression line after the change:

Y + ∆Y = β0 + β1 (X1 + ∆X1 ) + β2 X2
▶ Difference: ∆Y = β1 ∆X1 . So,

β1 = ∆Y /∆X1 holding X2 constant,

β2 = ∆Y /∆X2 holding X1 constant,
β0 = predicted value of Y when X1 = X2 = 0.

15 / 30
The OLS Estimator in Multiple Regression (SW Section 6.3)

▶ With two regressors, the OLS estimator solves:

n
X
min [Yi − (b0 + b1 X1i + b2 X2i )]2
b0 ,b1 ,b2
i=1

▶ The OLS estimator minimizes the average squared difference between

the actual values of Yi and the prediction (predicted value) based on the
estimated line.
▶ This minimization problem can be solved using calculus
▶ This yields the OLS estimators of (β0 , β1 , β2 ).

16 / 30
Example: the California test score data

▶ Regression of TestScore against STR:

TestScore = 698.9 − 2.28 × STR

▶ Now include percent English Learners in the district (PctEL):

TestScore = 686.0 − 1.10 × STR − 0.65 × PctEL

▶ What happens to the coefficient on STR?

17 / 30
Multiple regression in R

18 / 30
Measures of Fit for Multiple Regression (SW Section 6.4)

▶ Actual = predicted + residual: Yi = Ybi + ubi

▶ SER = standard deviation of ubi (with d.f. correction)
▶ RMSE = standard deviation of u bi (without d.f. correction)
▶ R 2 = fraction of variance of Y explained by X
2
▶ R = “adjusted R 2 ” = R 2 with a degrees-of-freedom correction that
2
adjusts for estimation uncertainty; R < R 2

19 / 30
SER and RMSE

▶ As in regression with a single regressor, the SER and the RMSE are
measures of the spread of the Y ’s around the regression line:
v
u n
u 1 X
SER = t bi2
u
n−k −1
i=1
v
u n
u1 X
RMSE = t bi2
u
n
i=1

20 / 30
R 2 and adjusted R 2

▶ The R 2 is the fraction of the variance explained – same definition as in

regression with a single regressor:
ESS SSR
R2 = =1−
TSS TSS

where ESS = i=1 (Yi − Y ) , TSS = i=1 (Yi − Y )2 , SSR = ni=1 u

Pn b 2 Pn
bi2
P

▶ The R 2 always increases when you add another regressor – a bit of a

problem for a measure of “fit”
2
▶ The R (the “adjusted R 2 ”) corrects this problem by “penalizing” you for
2
including another regressor – the R does not necessarily increase
when you add another regressor.
2 n − 1 SSR
R =1−
n − k − 1 TSS
2
Note that R ≤ R 2 , however if n is large the two will be very close.

21 / 30
Measures of fit (continued)

Test score example:

1. TestScore = 698.9 − 2.28 × STR with R 2 = 0.05 and SER = 18.6
2. TestScore = 686.0 − 1.10 × STR − 0.65 × PctEL
2
with R 2 = 0.426, R = 0.424, and SER = 14.5
▶ Including PctEL substantially improves the goodness of fit.
▶ SER reduces (unit of SER = unit of TestScore)
▶ R 2 substantially increases.
▶ Note:R 2 ≈ R 2 because n is large.
2
▶ Question: how to choose a variable – should we maximize R ?
Chapter 7 will discuss about how to choose a variable for a regression
analysis.

22 / 30
The Least Squares Assumptions for Multiple Regression (SW Section 6.5)

Yi = β0 + β1 X1i + β2 X2i + · · · + βk Xki + ui , i = 1, . . . , n

1. The conditional distribution of u given X has mean zero, that is,

E(ui |X1i = x1 , . . . , Xki = xk ) = 0.
2. (X1i , . . . , Xki , Yi ), i = 1, . . . , n, are i.i.d.
3. Large outliers are unlikely: X1 , . . . , Xk , and Y have four moments:
E(X1i4 ) < ∞, . . . , E(Xki4 ) < ∞, E(Yi4 ) < ∞
4. There is no perfect multicollinearity.

23 / 30
The Least Squares Assumptions for Multiple Regression (SW Section 6.5)

Assumption #1: the conditional mean of u given the included X ’s is zero.

E(ui |X1i = x1 , . . . , Xki = xk ) = 0

▶ This has the same interpretation as in regression with a single regressor.

▶ This condition fails when there exists an omitted variable, i.e.,
1. belongs in the equation (so is in u) and
2. is correlated with an included X
▶ then this condition fails and there is Omitted Variable bias.
▶ The best solution, if possible, is to include the omitted variable in the
regression.
▶ A second, related solution is to include a variable that controls for the
omitted variable (discussed in Ch. 7)

24 / 30
The Least Squares Assumptions for Multiple Regression (SW Section 6.5)

Assumption #2: (X1i , . . . , Xki , Yi ), i = 1, . . . , n, are i.i.d.

▶ This is satisfied automatically if the data are collected by simple random
sampling.
Assumption #3: large outliers are rare (finite fourth moments)
▶ This is the same assumption as we had before for a single regressor. As
in the case of a single regressor, OLS can be sensitive to large outliers,
so you need to check your data (scatter plots!) to make sure there are no
crazy values (typos or coding errors).
Assumption #4: There is no perfect multicollinearity
▶ Perfect multicollinearity is when one of the regressors is an exact linear
function of the other regressors. Most statistical software automatically
drop out the problematic variables.

25 / 30
The Sampling Distribution of the OLS Estimator (SW Section 6.6)

Under the four Least Squares Assumptions,

▶ E[βbj ] = βj for j = 0, 1, . . . , k , i.e., OLS estimators are unbiased
▶ V (βbj ) is inversely proportional to n
▶ For n large,
βbj − βj approx
∼ N (0, 1)
SE(βbj )
▶ Conceptually, there is nothing new here! The way we test a simple
hypothesis such as H0 : βj = βj0 is the same. When α = 0.05, Reject H0
b −β
β j j
1. if | | > 1.96
SE(β
b)
j
2. if p-value is smaller than 0.05
3. if βj0 is outside the 95% confidence interval, βbj ± 1.96SE(βbj )

26 / 30
Multicollinearity, Perfect and Imperfect (SW Section 6.7)

▶ Perfect multicollinearity is when one of the regressors is an exact

linear function of the other regressors.
▶ Some more examples of perfect multicollinearity
1. Include the same variable twice, i.e., X1 = X2 .
2. Regress TestScore on a constant, D, and B, where D is dummy for
STR ≤ 20 and B is dummy for STR > 20. So, B = 1 − D.
▶ 2 above is an example of ‘dummy variable trap’. More explicitly, suppose
you have a set of multiple binary (dummy) variables, which are mutually
exclusive and exhaustive; that is, there are multiple categories and every
observation falls in one and only one category. If you include all these
dummy variables and a constant, you will have perfect multicollinearity.
(Why?)
▶ Solutions: (1) omit one of the groups or (2) omit the intercept. The
interpretation of the coefficients is different between (1) and (2)!!

27 / 30
Perfect multicollinearity (continued)

▶ Perfect multicollinearity usually reflects a mistake in the definitions of the

regressors, or an oddity in the data
▶ If you have perfect multicollinearity, your statistical software will let you
know – either by crashing or returning an error message or by “dropping”
one of the variables arbitrarily
▶ The solution to perfect multicollinearity is to modify your list of regressors
so that you no longer have perfect multicollinearity.

28 / 30
Imperfect multicollinearity

▶ Imperfect and perfect multicollinearity are quite different despite the

similarity of the names.
▶ Imperfect multicollinearity occurs when two or more regressors are
highly correlated.
▶ Why the term “multicollinearity”? If two regressors are highly correlated,
then their scatterplot will pretty much look like a straight line – they are
“co-linear” – but unless the correlation is exactly ±1, that collinearity is
imperfect.

29 / 30
Imperfect multicollinearity, ctd.

▶ Imperfect multicollinearity implies that one or more of the regression

coefficients will be imprecisely estimated (large standard errors).
▶ The idea: the coefficient on X1 is the effect of X1 holding X2 constant; but
if X1 and X2 are highly correlated, there is very little variation in X1 once
X2 is held constant.
▶ So the data don’t contain much information about what happens when
X1 changes but X2 doesn’t. If so, the variance of the OLS estimator of
the coefficient on X1 will be large.
▶ Example: X1 is dummy for a woman and X2 is dummy for a lipstick user.
▶ Having high standard errors is a natural result: when X1 and X2 are
highly correlated, it is hard to disentangle the effect of X1 on Y from the
effect of X2 on Y . So, the estimates naturally have a lot of uncertainty.

30 / 30