AIL 7310: MACHINE LEARNING FOR ECONOMICS

Lecture 8

11th August, 2023

AIL 7310: ML for Econ Lecture 8 1 / 13

Simple Linear Regression Model

In the last lecture, we derived the sample estimates of βˆ1 and βˆ0 as.
Pn
(x − x̄)(yi − ȳ )
ˆ
β1 = i=1 Pn i 2
(1)
i=1 (xi − x̄)

AIL 7310: ML for Econ Lecture 8 2 / 13

Simple Linear Regression Model

In the last lecture, we derived the sample estimates of βˆ1 and βˆ0 as.
Pn
(x − x̄)(yi − ȳ )
ˆ
β1 = i=1 Pn i 2
(1)
i=1 (xi − x̄)

βˆ0 = ȳ − βˆ1 x̄ (2)

They are called the ordinary least squares estimates of βˆ1 and βˆ0 .

AIL 7310: ML for Econ Lecture 8 2 / 13

Simple Linear Regression Model

To justify this name, for any βˆ1 and βˆ0 define a fitted value for y when
x = xi as
yˆi = βˆ0 + β1ˆxi
This is the value we predict for y when x = xi for the given intercept and
slope. There is a fitted value for each observation in the sample. (Note:
We use the term fitted value here and not predicted value).The residual for
observation i is the difference between the actual yi and its fitted value:

ˆ = yi − βˆ0 − β1ˆxi
ûi = yi − yi

There are n such residuals. (Note: These are not the same as errors in eq.
(1))

AIL 7310: ML for Econ Lecture 8 3 / 13

Simple Linear Regression Model

Now,
Pn suppose we choose βˆ1 and βˆ0 to make the sum of squared residuals
2
i=1 ûi as small as possible. Mathematically,

n
X
min (yi − β0 − β1 xi )2 (3)
β0 ,β1
i=1

The first-order conditions are

n
X
−2 (yi − β0 − β1 xi ) = 0 (4)
i=1

n
X
−2 xi (yi − β0 − β1 xi ) = 0 (5)
i=1

AIL 7310: ML for Econ Lecture 8 4 / 13

Simple Linear Regression Model

Equations (4) and (5) are basically the same equations as eqs. (11) and
(12) from last class which can be solved to give us the ordinary least
squares estimates.

AIL 7310: ML for Econ Lecture 8 5 / 13

Simple Linear Regression Model

Equations (4) and (5) are basically the same equations as eqs. (11) and
(12) from last class which can be solved to give us the ordinary least
squares estimates.

They are called ”ordinary least squares” since they minimize the sum of
squared residuals.

AIL 7310: ML for Econ Lecture 8 5 / 13

Simple Linear Regression Model

Equations (4) and (5) are basically the same equations as eqs. (11) and
(12) from last class which can be solved to give us the ordinary least
squares estimates.

They are called ”ordinary least squares” since they minimize the sum of
squared residuals.

Once we have determined the OLS intercept and slope estimates, we form
the OLS regression line.

AIL 7310: ML for Econ Lecture 8 5 / 13

Simple Linear Regression Model

Equations (4) and (5) are basically the same equations as eqs. (11) and
(12) from last class which can be solved to give us the ordinary least
squares estimates.

They are called ”ordinary least squares” since they minimize the sum of
squared residuals.

Once we have determined the OLS intercept and slope estimates, we form
the OLS regression line.

ŷ = βˆ0 + βˆ1 x (6)

The notation ŷ emphasizes that the predicted values are estimates. The
intercept βˆ0 is the fitted value of y when x = 0.

AIL 7310: ML for Econ Lecture 8 5 / 13

Simple Linear Regression Model

Note: This equation (6) above is the sample regression function (SRF).
This is an estimated version of the population regression function
E (y |x) = β0 + β1 x we has stated earlier. Remember the population
regression line is something fixed. But the SRF is obtained for a given
sample of data, a new sample will generate a different slope and intercept
in eq. (6).

AIL 7310: ML for Econ Lecture 8 6 / 13

Simple Linear Regression Model

Note: This equation (6) above is the sample regression function (SRF).
This is an estimated version of the population regression function
E (y |x) = β0 + β1 x we has stated earlier. Remember the population
regression line is something fixed. But the SRF is obtained for a given
sample of data, a new sample will generate a different slope and intercept
in eq. (6).

The slope estimate can be written as

βˆ1 = ∆ŷ /∆x

This is of primary interest; it tells us the amount by which ŷ changes when

x increases by one unit.

AIL 7310: ML for Econ Lecture 8 6 / 13

Simple Linear Regression:Example
Suppose you take a dataset of 500 people and their data on wage (hourly
in INR) and education (in number of years in school). You fit a simple
linear regression model and obtain the following sample regression
function.

wage
[ = 90 + 154education

AIL 7310: ML for Econ Lecture 8 7 / 13

Simple Linear Regression:Example
Suppose you take a dataset of 500 people and their data on wage (hourly
in INR) and education (in number of years in school). You fit a simple
linear regression model and obtain the following sample regression
function.

wage
[ = 90 + 154education

How do you interpret 90 and 154?

AIL 7310: ML for Econ Lecture 8 7 / 13

Simple Linear Regression:Example
Suppose you take a dataset of 500 people and their data on wage (hourly
in INR) and education (in number of years in school). You fit a simple
linear regression model and obtain the following sample regression
function.

wage
[ = 90 + 154education

How do you interpret 90 and 154?

AIL 7310: ML for Econ Lecture 8 7 / 13

Simple Linear Regression:Example
Suppose you take a dataset of 500 people and their data on wage (hourly
in INR) and education (in number of years in school). You fit a simple
linear regression model and obtain the following sample regression
function.

wage
[ = 90 + 154education

How do you interpret 90 and 154?

Note: In our study so far the slope parameter simply measures the
association between y and x. It says nothing about causality. Even
through we often casually use causal language to talk about β1 (e.g. does
a decrease in price increase sales?) it is important to understand that
causality can only be determined after imposing other conditions.

AIL 7310: ML for Econ Lecture 8 7 / 13

The meaning of ”linear” regression

What does ’linear’ mean in this model?

y = β0 + β1 x + u

AIL 7310: ML for Econ Lecture 8 8 / 13

The meaning of ”linear” regression

What does ’linear’ mean in this model?

y = β0 + β1 x + u

They key here is that this model is linear in the parameters β0 and β1 .

AIL 7310: ML for Econ Lecture 8 8 / 13

The meaning of ”linear” regression

What does ’linear’ mean in this model?

y = β0 + β1 x + u

They key here is that this model is linear in the parameters β0 and β1 .

We can easily use simple linear regression to estimate a √

model where
wage 2
[ = β0 + β1 (education) or consumption = β0 + β1 income
\

AIL 7310: ML for Econ Lecture 8 8 / 13

The meaning of ”linear” regression

What does ’linear’ mean in this model?

y = β0 + β1 x + u

They key here is that this model is linear in the parameters β0 and β1 .

We can easily use simple linear regression to estimate a √

model where
wage 2
[ = β0 + β1 (education) or consumption = β0 + β1 income
\

Infact, we often use this to our advantage to estimate a wide range of

models using a simple linear regression framework.

AIL 7310: ML for Econ Lecture 8 8 / 13

Different functional forms

In the previous example we saw that each additional year of education

increases hourly wage by 154 Rupees. Here the rate of change is same for
the first year of education or the twelfth year. This may not be entirely
realistic. Perhaps a better characterization would be increases in a
percentage.A model that gives (approximately) a constant percentage
effect is

AIL 7310: ML for Econ Lecture 8 9 / 13

Different functional forms

In the previous example we saw that each additional year of education

increases hourly wage by 154 Rupees. Here the rate of change is same for
the first year of education or the twelfth year. This may not be entirely
realistic. Perhaps a better characterization would be increases in a
percentage.A model that gives (approximately) a constant percentage
effect is

log (wage) = β0 + β1 education + u

where log (.) denotes the natural logarithm.

AIL 7310: ML for Econ Lecture 8 9 / 13

Different functional forms

In the previous example we saw that each additional year of education

increases hourly wage by 154 Rupees. Here the rate of change is same for
the first year of education or the twelfth year. This may not be entirely
realistic. Perhaps a better characterization would be increases in a
percentage.A model that gives (approximately) a constant percentage
effect is

log (wage) = β0 + β1 education + u

where log (.) denotes the natural logarithm.

Then,
%∆wage ≈ (100 ∗ β1 )∆education

AIL 7310: ML for Econ Lecture 8 9 / 13

Different functional forms

There are four commonly used functional forms in econometrics.

AIL 7310: ML for Econ Lecture 8 10 / 13

Different functional forms

There are four commonly used functional forms in econometrics.

Model Dependent Var. Independent Var. Interpretation

Level-level y x ∆y = β1 ∆x
Level -log y log(x) ∆y = (β1 /100)%∆x
Log - level log(y) x %∆y = (100β1 )∆x
Log-log log(y) log(x) %∆y = β1 %∆x

AIL 7310: ML for Econ Lecture 8 10 / 13

Different functional forms

There are four commonly used functional forms in econometrics.

Model Dependent Var. Independent Var. Interpretation

Level-level y x ∆y = β1 ∆x
Level -log y log(x) ∆y = (β1 /100)%∆x
Log - level log(y) x %∆y = (100β1 )∆x
Log-log log(y) log(x) %∆y = β1 %∆x

In the log-level model, 100.β1 is sometimes called the semi-elasticity of y

with respect to x.

AIL 7310: ML for Econ Lecture 8 10 / 13

Different functional forms

There are four commonly used functional forms in econometrics.

Model Dependent Var. Independent Var. Interpretation

Level-level y x ∆y = β1 ∆x
Level -log y log(x) ∆y = (β1 /100)%∆x
Log - level log(y) x %∆y = (100β1 )∆x
Log-log log(y) log(x) %∆y = β1 %∆x

In the log-level model, 100.β1 is sometimes called the semi-elasticity of y

with respect to x.

In the log-log model, β1 is called the elasticity of y with respect to x.

AIL 7310: ML for Econ Lecture 8 10 / 13

Multiple Linear Regression

Now, we move on to more realistic models - the multiple linear regression

model. Here, we incorporate multiple (k) explanatory variables.

AIL 7310: ML for Econ Lecture 8 11 / 13

Multiple Linear Regression

Now, we move on to more realistic models - the multiple linear regression

model. Here, we incorporate multiple (k) explanatory variables.

y = β0 + β1 x1 + β2 x2 + ...βk xk + u (7)

AIL 7310: ML for Econ Lecture 8 11 / 13

Multiple Linear Regression

Now, we move on to more realistic models - the multiple linear regression

model. Here, we incorporate multiple (k) explanatory variables.

y = β0 + β1 x1 + β2 x2 + ...βk xk + u (7)

Here, β0 is the intercept, β1 is the change in y with respect to x1 holding

other factors fixed, β2 is the change in y with respect to x2 holding other
factors fixed and so on.

AIL 7310: ML for Econ Lecture 8 11 / 13

Multiple Linear Regression

The key assumption here is

E (u|x1 , x2 , ..., xk ) = 0 (8)

AIL 7310: ML for Econ Lecture 8 12 / 13

Multiple Linear Regression

The key assumption here is

E (u|x1 , x2 , ..., xk ) = 0 (8)

which is the multiple linear regression counterpart of the zero conditional

mean assumption. This assumption implies that all factors in the
unobserved error term are uncorrelated with the explanatory variables.

AIL 7310: ML for Econ Lecture 8 12 / 13

Obtaining the OLS Estimates

Similar to the simple linear regression case, the OLS estimates (k+1) of
them are chosen to minimize the sum of squared residuals:
n
X
(yi − β0 − β1 xi1 − ...βk xik )2 (9)
i=1

The first-order conditions will be a set of k+1 linear equations in k+1

unknowns which can be solved to get the OLS estimates βˆ0 , βˆ1 , ...βˆk .

AIL 7310: ML for Econ Lecture 8 13 / 13