Lecture 3.

Multiple
Regression Analysis:
Estimation
Prepared by Quanquan Liu
Fall 2024
Motivation

 The Model with Two Independent Variables

 Example. Wage and Education
.
 Because this model contains experience explicitly, we will be able to measure the effect of
education on wage, holding experience fixed.
 Zero conditional mean assumption:
 This implies that other factors affecting wage are not related on average to educ and
exper. If we think innate ability is part of u, we will need average ability levels to be
the same across all combinations of education and experience in the working
population.
Motivation

 The Model with Two Independent Variables

 Example. Consumption and Income
.
 Multiple regression analysis is also useful for generalizing functional relationships between
variables.
 – the marginal effect of income on consumption depends on as well as on and the level
of income.
Motivation

 The Model with k Independent Variables

 The general multiple linear regression (MLR) model can be written in the population as
,
 is the intercept; is the parameter associated with ; is the parameter associated with , and
so on.
 u is the error term; it contains factors other than that affect y.
 Zero conditional mean assumption: .
 All factors in the unobserved error term must be uncorrelated with the explanatory
variables.
 We have correctly accounted for the functional relationships between the explained
and explanatory variables.
Motivation

 The Model with k Independent Variables

 Example. CEO salary, sales and CEO tenure
.
 Model assumes a constant elasticity relationship between CEO salary and the sales of his
or her firm.
 The parameter is the ceteris paribus elasticity of salary with respect to sales.
 Model assumes a quadratic relationship between CEO salary and his or her tenure with
the firm.
 The term “linear” means that the model is linear in the parameters, , not in the variables.
Ordinary Least Squares

 In the general case with k independent variables, we seek estimates, in the equation

 Given n observations the OLS estimates, k + 1 of them, are chosen to minimize the
sum of squared residuals:

 First order conditions:

Ordinary Least Squares

 The OLS regression line is

.
 Written in terms of changes,
.
 The coefficient on measures the change in due to a one-unit increase in , holding all other
independent variables fixed – partial effect. That is,
,
holding fixed.
 partial effect: The effect of an explanatory variable on the dependent variable, holding other
factors in the regression model fixed.
Ordinary Least Squares

 Example. Determinants of College GPA

.
 There is a positive partial relationship between colGPA and hsGPA: holding ACT fixed,
another point on hsGPA is associated with .453 of a point on the college GPA.
 If we choose two “typical” students, A and B, and these students have the same ACT score, but
the high school GPA of Student A is one point higher than the high school GPA of Student B, then
we predict Student A to have a college GPA .453 higher than that of Student B.
 The sign on ACT implies that, while holding hsGPA fixed, a change in the ACT score of 10
points – a very large change – affects colGPA by less than one-tenth of a point.
Ordinary Least Squares

 For observation i, the fitted value

.
 The residual for observation i is
.
 Properties:
Ordinary Least Squares

 Define the total sum of squares (SST), the explained sum of squares (SSE), and the residual
sum of squares or sum of squared residuals (SSR) as

 Using the same argument as in the simple regression case, we can show that
.
 Just as in the simple regression case, the R-squared is defined to be
.
Ordinary Least Squares

 never decreases, and it usually increases, when another independent variable is added to
a regression and the same set of observations is used for both regressions.
 The sum of squared residuals never increases when additional regressors are added
to the model.
 Missing data can be an important practical issue.
 Adjusted : A goodness-of-fit measure in multiple regression analysis that penalizes
additional explanatory variables by using a degrees of freedom adjustment in estimating
the error variance.

 Adjusted can go up or down when a new independent variable is added to a regression.

 : For small n and large k, adjusted can be substantially below . In fact, if the usual is small, and is
also small, adjusted can actually be negative!
The Expected Value of the OLS
Estimators
 Assumption MLR.1. Linear in Parameters
 The model in the population can be written as
,
where are the unknown parameters (constants) of interest and u is an unobserved
random error or disturbance term.
 Assumption MLR.2. Random Sampling
 We have a random sample of n observations, , following the population model in
Assumption MLR.1.
 For a randomly drawn observation i from the population, we have
.
The Expected Value of the OLS
Estimators
 Assumption MLR.3. No Perfect Collinearity
 In the sample (and therefore in the population), none of the independent variables is
constant, and there are no exact linear relationships among the independent variables.
 If an independent variable is an exact linear combination of the other independent variables, then
we say the model suffers from perfect collinearity, and it cannot be estimated by OLS.
 In the general regression model, there are parameters, and MLR.3 fails if .
 Assumption MLR.4. Zero Conditional Mean
 The error u has an expected value of zero given any values of the independent variables,
.
 Fail if the functional relationship between the explained and explanatory variables is mis-specified.
 Fail if an important factor that is correlated with any of is omitted.
 Fail if there is measurement error in an explanatory variable.
The Expected Value of the OLS
Estimators
 When Assumption MLR.4 holds, we often say that we have exogenous explanatory variables.
 exogenous explanatory variables: An explanatory variable that is uncorrelated with the error term.
 If is correlated with u for any reason, then is said to be an endogenous explanatory variable.
 endogenous explanatory variable: An explanatory variable in a multiple regression model that is
correlated with the error term, either because of model misspecification, omitted variables, measurement
error, or simultaneity.
 Theorem 1. Unbiasedness of OLS
 Under Assumptions MLR.1, MLR.2, MLR.3 and MLR.4,
,
for any values of the population parameter . In other words, the OLS estimators are unbiased
estimators of the population parameters.
The Expected Value of the OLS
Estimators
 Including one or more irrelevant variables in a multiple regression model, or
overspecifying the model, does not affect the unbiasedness of the OLS estimators.
 However, including irrelevant variables may increase sampling variance.
 Excluding one or more relevant variables in a multiple regression model, or
underspecifying the model, generally causes the OLS estimators to be biased.
 Example. Omitted Variable Bias
 Suppose the true population model has two explanatory variables and an error term:
.
Estimated equation:
 However, due to our ignorance or data unavailability, we estimate the model by excluding
and perform a simple regression of y on only, obtaining the equation

.
The Expected Value of the OLS
Estimators
 To derive the bias, run a simple regression of on :
.
Estimated equation:
 Plug into the true model:

 Conclusion: All estimated coefficients will be biased.

The Expected Value of the OLS
Estimators
 We have the algebraic relationship
,
,
which implies the bias in is
.
 If – so that does not appear in the true model – then is unbiased.
 If – so that and are uncorrelated in the sample – then is unbiased for , even if .
The Expected Value of the OLS
Estimators
 Table. Summary of Bias in When Is Omitted in Estimating Equation
~ ~
Bias 𝛿 1 >0 𝛿 1 <0

 Example. Omitting ability in a wage equation

The Variance of the OLS Estimators

 Assumption MLR.5. Homoskedasticity

 The error u has the same variance given any value of the explanatory variables. In other
words, .
 Theorem 2. Sampling Variances of the OLS Slope Estimators
 Under Assumptions MLR.1, MLR.2, MLR.3, MLR.4 and MLR.5, conditional on the sample
values of the independent variables,

for , where is the total sample variation in , and is the R-squared from regressing on all
other independent variables (including an intercept).
The Variance of the OLS Estimators

 The Error Variance, .

 A high error variance increases the sampling variance because there is more “noise” in the
equation.
 The error variance does not decrease with sample size.
 The Total Sample Variation in , .
 The larger the total variation in is, the smaller is .

 Total sample variation automatically increases with the sample size.

 Increasing the sample size is thus a way to get more precise estimates.
The Variance of the OLS Estimators

 The Linear Relationships among the Independent Variables, .

 will be higher when can be better explained by the other independent variables.
 The sampling variance of the slope estimator for will be larger when gets larger: as .
 High (but not perfect) correlation between two or more independent variables is called
multicollinearity.
 Everything else being equal, for estimating , it is better to have less correlation between
and the other independent variables.
 Only the sampling variance of the variables involved in multicollinearity will be inflated;
the estimates of other effects may be very precise.
The Variance of the OLS Estimators

 True population model:

Estimated model 1:
Estimated model 2:
 Case 1. and are uncorrelated in the sample.
The Variance of the OLS Estimators

 Case 2. and are correlated in the sample, and .

 is preferred.
 Do not include irrelevant regressors!
 Case 3. and are correlated in the sample, and .

 Trade off bias and variance!

 In large samples, we would prefer .
The Variance of the OLS Estimators

 Given the OLS residuals:

,
The unbiased estimator of in the general multiple regression case is

where degrees of freedom .

 Theorem 3. Unbiased Estimation of
 Under the Gauss-Markov assumptions MLR.1, MLR.2, MLR.3, MLR.4 and MLR.5,
.
The Variance of the OLS Estimators

 - the standard error of the regression (SER), which is an estimator of the standard
deviation of the error term.

 It can either decrease or increase when another independent variable is added to a regression.
Efficiency of OLS

 Theorem 4. Gauss-Markov Theorem

 Under Assumptions MLR.1, MLR.2, MLR.3, MLR.4 and MLR.5, are the best linear unbiased
estimators (BLUEs) of , respectively.
 Estimator: It is a rule that can be applied to any sample of data to produce an estimate.
 Unbiased: In the current context, an estimator of is an unbiased estimator of if for any .
 Linear: In the current context, an estimator of is linear if, and only if, it can be expressed as a
linear function of the data on the dependent variable:
,
where each can be a function of the sample values of all the independent variables.
 Best: For the current theorem, best is defined as having the smallest variance: For any estimator
that is linear and unbiased, , and the inequality is usually strict.
Summary

 Multiple Regression Analysis: Estimation

 Motivation for Multiple Regression
 Mechanics and Interpretation of Ordinary Least Squares
 The Expected Value of the OLS Estimators
 Omitted Variable Bias
 The Variance of the OLS Estimators
 Multicollinearity
 Variances in Mis-specified Models
 Estimating σ2: Standard Errors of the OLS Estimators
 Efficiency of OLS: The Gauss-Markov Theorem