As of Sep 16, 2020: Seppo Pynn Onen Econometrics I

The Simple Regression Model
Part II
As of Sep 16, 2020

Seppo Pynnönen Econometrics I
Definition
1 The Simple Regression Model
Definition
Estimation of the model, OLS
OLS Statistics
Algebraic properties
Goodness-of-Fit, the R-square

Units of Measurement and Functional Form
Scaling and translation
Nonlinearities
Expected Values and Variances of the OLS Estimators

Unbiasedness
Variances
Estimating the Error Variance
Standard Errors of β̂0 and β̂1
Definition
Two (observable) variables ”y ” and ”x”.
y = β0 + β1 x + u. (1)
Equation (1) defines the simple regression model.

Terminology:
y x
Dependent variable Independent variable
Explained variable Explanatory variable
Response variable Control variable
Predicted variable Predictor
Regressand Regressor

Definition
The variable u, called the error term or disturbance, accounts

factors other than x that affect y .
β0 is the intercept (called also the constant term) and β1 is the
slope of the regression line β0 + β1 x. That is, if the other factors
(in u) are held fixed, so that the change in u is zero,
∆y = β1 ∆x, if ∆u = 0.
Thus, the (expected) change in y is β1 times the change in x.

In particular if x changes by one unit, the (expected) change in y
is β1 units.

Definition
Error term ui is a combination of a number of effects, like:

1 Omitted variables: Accounts the effects of variables omitted
from the model
2 Nonlinearities: Captures the effects of nonlinearities between
y and x. Thus, if the true model is yi = β0 + β1 xi + γxi2 + vi ,
and we assume that it is yi = β0 + β1 x + ui , then the effect of
xi2 is absorbed to ui . In fact, ui = γxi2 + vi .
3 Measurement errors: Errors in measuring y and x are
absorbed in ui .
4 Unpredictable effects: ui includes also inherently unpredictable
random effects.


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
Given observations (xi , yi ), i = 1, . . . , n, the task for an

econometrician is to estimate the population parameters β0
(intercept parameter) and β1 (slope parameter) of (1) in an
optimal way by utilizing the sample information.
Assumptions (classical assumptions)
1 Zero average error: E[ui ] = 0 for all i.
2 Homoskedasticity: var[ui ] = σ 2 for all i.
3 Uncorrelated errors: cov[ui , uj ] = 0, for all i 6= j.
Pn 2
i=1 (xi − x̄) > 0.
4
5 cov[ui , xi ] = 0.

Remark 1
Assumptions 1 and 2 are related to distributional properties of
the error term of which Assumption 1 can always be fulfilled
when the intercept term β0 is included to the regression in
(1). We discuss Assumption 2 later in more detail.
Assumption 3. is mostly relevant in time series data or
”clustered” data.
Assumption 5 is the key assumption how x and u are related,
expressed sometimes in terms the stronger assumption of
mean independence, E[ui |xi ] = E[ui ], which implies
cov[ui , xi ] = 0 when E[ui ] = 0.

Exploring further 2.1, Wooldridge 5e, p.23

Suppose that a score on a final exam, score, depends on classes
attended (attend) and unobserved factors that affect exam performance
(such as student ability). Then
score = β0 + β1 attend + u. (2)
When would you expect this model to satisfy E[u|attend] = E[u]?

The goal in the estimation is to find values for β0 and β1 of model

(1) such that the error terms is as small as possible (in a suitable
sense).
Under the above classical assumptions, the Ordinary Least Squares
(OLS) minimizes the residual sum of squares of the error terms
ui = yi − β0 − β1 xi , which produces optimal estimates for the
parameters (the optimality criteria are discussed later).
Denote the sum of squares as
n
X
f (β0 , β1 ) = (yi − β0 − β1 xi )2 . (3)
i=1
The first order conditions (foc) for the minimum are found by
setting the partial derivatives equal to zero. Denote by β̂0 and β̂1
the values satisfying the foc.

First order conditions:

n
∂f (β̂0 , β̂1 ) X
= −2 (yi − β̂0 − β̂1 xi ) = 0 (4)
∂β0
i=1
n
∂f (β̂0 , β̂1 ) X
= −2 xi (yi − β̂0 − β̂1 xi ) = 0 (5)
∂β1
i=1
These yield so called normal equations

P P
nβ̂0 + β̂1 xi = yi
(6)
β̂0 xi + β̂1 xi2 =
P P P
xi yi ,
where the summation is from 1 to n.

The explicit solutions for β̂0 and β̂1 are (OLS estimators of β0 and
β1 ) Pn
(x − x̄)(yi − ȳ )
β̂1 = i=1 Pn i 2
(7)
i=1 (xi − x̄)
β̂0 = ȳ − β̂1 x̄, (8)

where
n n
1X 1X
x̄ = xi and ȳ = yi
n n
i=1 i=1
are the sample means.
In the solutions (7) and (8) we have used the properties
n
X n
X
(xi − x̄)(yi − ȳ ) = xi yi − nx̄ ȳ (9)
i=1 i=1
and
n
X n
X
(xi − x̄)2 = xi2 − nx̄ 2 . (10)
i=1Seppo Pynnönen Econometrics I
i=1
Fitted regression line:

ŷ = β̂0 + β̂1 x. (11)
Residuals:
ûi = yi − ŷi
(12)
= (β0 − β̂0 ) + (β1 − β̂1 )xi + ui .
Thus the residual component ûi consist of the pure error term ui
and the sample errors due to the estimation of the parameters β0
and β1 .

Fitted regression line ŷ = β̂0 + β̂1 x in (11) is called also as the

sample regression function (SRF) while E[y |x] = β0 + β1 x (recall
E[u|x] = 0) is called the population regression function (PRF).
It is notable that PRF is something fixed, but unknown, in the
population.
SRF is obtained for a given sample of data, a new sample will
generate new a slope and intercept in (11), and therefore different
lines (that vary around PRF).

Example 1
Suppose the population regression is y = β0 + β1 x + u with u ∼ N(0, σ 2 ) in
which β0 = 5, β1 = 1.2, and σ 2 = 4.
For a sample of n = 50 observations (x generated from the normal distribution
N(5, 3)), we have the sample regression line shown in the top right hand plot.
Repeating sampling 5 times, the respective SRFs are shown in the bottom left
figure along with the PRF.
PRF and a SDF form a sample
Data
of n = 50 observations
Obs x y
10 12 14
1 5.03 10.24
2 4.68 9.95
3 2.62 10.89
4 3.96 14.03
5 5.51 12.62
. . .
y
. . .
8
. . .
6
PRF
SRF
4
1 2 3 4 5 6 7
Population regression line and

five OLS sample regression lines
PRF
10 12 14
SRFs
y
8
6
4
1 2 3 4 5 6 7

Remark 2
The slope coefficient β̂1 in terms of sample covariance of x and y and
variance of x.
Sample covariance:
n
1 X
sxy = (xi − x̄)(yi − ȳ ) (13)
n−1
i=1
Sample variance:
n
1 X
sx2 = (xi − x̄)2 . (14)
n−1
i=1
Thus
sxy
β̂1 = . (15)
sx2

Remark 3
The slope coefficient β̂1 in terms of sample correlation and standard
deviations of x and y .
Sample correlation:
Pn
i=1 (xi − x̄)(yi − ȳ ) sxy
rxy = qP = , (16)
n
(x − x̄)2
P n
(y − ȳ ) 2 sx sy
i=1 i i=1 i
p q
where sx = sx2 and sy = sy2 are the sample standard deviations of x
and y , respectively.
Thus we can also write the slope coefficient in terms of sample standard
deviations and correlation as
sy
β̂1 = rxy . (17)
sx

Example 2
Relationship between wage and eduction.
wage = average hourly earnings
educ = years of education
Data is collected in 1976, n = 526
SAS commands for reading data, printing a few lines, sample statistics,
generating a scatter plot, and estimating the regression.
SAS Studio excerpt:

Scatter plot of the data with regression line.

/* Scatter plot with regression line imposed */
proc sgplot data = a;
reg x = educ y = wage / lineattrs = (color = "red"); * generates scatter plot with regression line;
xaxis label = "Education (years)"; * x-axis label;
yaxis label = "Hourly wage"; * y-axis label
run;
Figure 2.2: Wages and education.

Sample statistics:
/* Some sample statistics */
proc means data = a min max mean std skew kurt maxdec = 2; * stats with output rounded to two decimals;
var wage educ;
run;

SAS PROC REG results:
The estimated model is

ŷ = −0.905 + 0.541x.
Thus, the model predicts that an additional year increases hourly wage on
average by 0.54 dollars.
Using (17) you can verify the OLS estimate for β1 can be computed using the
correlation (rwage,educ = 0.406) and standard deviations in the above sample
statistics table. After that, applying (8) you get OLS estimate for the intercept.
Thus in all, the estimates can be derived from the basic sample statistics.

OLS Statistics
Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
OLS Statistics
n
X
ûi = 0. (18)
i=1
n
X
xi ûi = 0. (19)
i=1
ȳ = β̂0 + β̂1 x̄. (20)
n
X
SST = (yi − ȳ )2 . (21)
i=1
Xn
SSE = (ŷi − ȳ )2 . (22)
i=1
Xn
SSR = (yi − ŷi )2 . (23)
i=1
OLS Statistics
It can be shown that

n
X n
X n
X
(yi − ȳ )2 = (ŷi − ȳ )2 + (yi − ŷi )2 , (24)
i=1 i=1 i=1
that is
SST = SSE + SSR. (25)
Prove this!
Remark 4
It is unfortunate that different books and different statistical packages
use different definitions, particularly for SSR and SSE. In many the
former means Regression sum of squares and the latter Error sum of
squares. I.e., just the opposite we have here!


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
R-square (coefficient of determination)
SSE SSR
R2 = =1− . (26)
SST SST
The positive square root of R 2 , denoted as R, is called the

multiple correlation.
Remark 5
Here in the case of simple regression R 2 = rxy
2
, i.e. R = |rxy |. These do
not hold in the general case (multiple regression)!
Prove Remark 5 by yourself.

Remark 6
Generally it holds for the OLS estimation, however, that R = ry ŷ , i.e.
correlation between the observed and fitted (or predicted) values.
Remark 7
It is obvious that 0 ≤ R 2 ≤ 1 with R 2 = 0 representing no linear relation
between x and y and R 2 = 1 representing a perfect fit.
Adjusted R-square:
su2
R̄ 2 = 1 − , (27)
sy2
where
n
1 X
su2 = (yi − ŷi )2 (28)
n−2
i=1
is an estimate of the residual variance
σu2 = var[u].
We find easily that
n−2
R̄ 2 = 1 − (1 − R 2 ). (29)
n−1
Seppo Pynnönen
2 Econometrics
2 I
Example 3
In the previous example R 2 = 0.1648 and adjusted R-squared,
R̄ 2 = 0.1632. The R 2 tells that about 16.5 percent of the variation in the
hourly earnings can be explained by education. However, the rest 83.5
percent is not accounted by the model.


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
Consider the simple regression model
yi = β0 + β1 xi + ui (30)
with σu2 = var[ui ].

Let yi∗ = a0 + a1 yi and xi∗ = b0 + b1 xi , a1 6= 0 and b1 6= 0. Then
(30) becomes
yi∗ = β0∗ + β1∗ xi∗ + ui∗ , (31)
where
a1
β0∗ = a1 β0 + a0 − β1 b0 , (32)
b1
a1
β1∗ = β1 , (33)
b1
and
σu2∗ = a12 σu2 . (34)

In short the effects on the regression coefficients are:

The slope coefficient (β1 ) is affected only by scaling of the
measurements.
Scaling y scales the slope coefficient by the same amount.
Scaling x scales the slope coefficient inversely.
The intercept is affected by both the scaling and origin shift.

Remark 8
Coefficients a1 and b1 scale the measurements and a0 and b0 shift the
measurements origin.
For example, if y is temperature measured in Celsius, then

9
y ∗ = 32 + y
5
gives temperature in Fahrenheit.

Example 4
Suppose an econometrician has estimated the following wage equation
wage = 8 + 0.8 educ,
where wage is wage per hour in US dollars and educ is education in

number of years.
How much an additional year of education increases wage in euros if one
euro is the exchange rate is 1.10 (i.e. one euro is worth of 1.10 dollars)?
Because wagee = wage$ /1.10, β̂e = β̂$ /1.10 = 0.8/1.10 ≈ 0.73.

Example 5
Let the estimated model be
ŷi = β̂0 + β̂1 xi .
”Demeaned” observations:
yi∗ = yi − ȳ and xi − x̄. So a0 = −ȳ , b0 = −x̄, and a1 = b1 = 1.

Because β̂0 = ȳ − β̂1 x̄, we obtain from (32)
β̂0∗ = β̂0 − (ȳ − β̂1 x̄) = 0.
So
ŷ ∗ = β̂1 x ∗ .
(Note β̂1 remains unchanged).

If we further define a1 = 1/sy and b1 = 1/sx , where sy and sx are the

sample standard deviations of y and x, respectively. Applying the
transformation yields standardized observations
yi − ȳ xi − x̄
yi∗ = and xi∗ = .
sy sx
Then again β̂0 = 0. The slope coefficient becomes

sx
β̂1∗ = β̂1 ,
sy
which is called standardized regression coefficient.
As an exercise show that in this case β̂1∗ = rxy , the correlation coefficient
of x and y .


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
Logarithmic transformation is one of the most applied

transformation for economic variables.
Table 2.1 Functional forms including log-transformations
Dependent Independent Interpretation

Model variable variable of β1
level-level y x ∆y = β1 ∆x
level-log y log(x) ∆y = (β1 /100)%∆x
log-level log(y ) x %∆y = (100β1 )∆x
log-log log(y ) log(x) %∆y = β1 %∆x
∆x, ∆y are changes in x andy , and %∆x = 100∆x/x, %∆y = 100∆y /y are
percentage changes.
Can you find the rationale for the interpretations?

Remark 9
Log-transformation can be only applied to variables that assume strictly
positive values!
Example 6
Consider again the wage example.
Suppose we believe that instead of the absolute change a better choice is

to consider the percentage change of wage (y ) as a function of education
(x). Then we would consider the model
log(y ) = β0 + β1 x + u.
Estimation of this model yields

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.583773 0.097336 5.998 3.74e-09 ***
educ 0.082744 0.007567 10.935 < 2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.4801 on 524 degrees of freedom

Multiple R-squared: 0.1858,Adjusted R-squared: 0.1843
F-statistic: 119.6 on 1 and 524 DF, p-value: < 2.2e-16

That is
\) = 0.584 + 0.083x
log(y (35)
2
n = 526, R = 0.186. Note that R-squares of this model and the
level-level model are not comparable.
The model predicts that an additional year of education increases on

average hourly earnings by 8.3%.

Remark 10
Typically all models where transformations on y and x are functions of
these variables alone can be cast to the form of linear model. That is, if
have generally
g (y ) = β0 + β1 h(x) + u, (36)
where g and h are functions, then defining y ∗ = g (y ) and x ∗ = h(x) we
have a linear model
y ∗ = β0 + β1 x ∗ + u.
Note, however, that all models cannot be cast to a linear form. An
example is
1
cons = + u.
β0 + β1 income


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
We say generally
h i that an estimator of θ̂ of a parameter θ is
unbiased if E θ̂ = θ.
Theorem 1
Under the classical assumptions 1–5
h i h i
E β̂0 = β0 and E β̂1 = β1 . (37)
Proof: Given observations x1 , . . . , xn the expectations are conditional on

the given xi -values.
We prove first the unbiasedness of β̂1 . Now

P P
β̂1 = Pi −x̄)(yi −ȳ
(x )
=P(xi −x̄)y2i
(xi −x̄)2 (xi −x̄)
P P P
= P(xi −x̄)β20 + β1 P(x i −x̄)xi P(xi −x̄)u2i (38)
(xi −x̄) (xi −x̄)2 + (xi −x̄)
P 1
Seppo Pynnönen
PEconometrics I
I.e.,
1 X
β̂1 = β1 + P (xi − x̄)ui . (39)
(xi − x̄)2
Because xi s fixed we get
h i 1 X
E β̂1 = β1 + P 2
(xi − x̄)E[ui ] = β1 (40)
(xi − x̄)
Because E[ui ] = 0 by assumption 1. Thus β̂1 is unbiased.
Proof of unbiasedness of β̂0 is left to students.


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
Theorem 2
Under the classical assumptions 1 through 5 and given x1 , . . . , xn
h i σu2
var β̂1 = Pn 2
, (41)
i=1 (xi − x̄)
h i 1 x̄ 2

var β̂0 = +P σu2 . (42)
n (xi − x̄)2
and for ŷ = β̂0 + β̂1 x with given x
(x − x̄)2

1
var[ŷ ] = +P σu2 . (43)
n (xi − x̄)2

Proof: Again we prove as an example only (41). Using (39) and the
properties of variance with x1 , . . . , xn given
h i h i
= var β1 + P(xi1−x̄)2 (xi − x̄)ui
P
var β̂1
2 P
= P 1 (xi − x̄)2 var[ui ]
(xi −x̄)2
2 P
= P 1 (xi − x̄)2 σu2 (44)
(xi −x̄)2
σu2 (xi −x̄)2

P
= ( (xi −x̄)2 )2
P
2
= P σu 2 .
(xi −x̄)

Remark 11
(42) can be written equivalently as
σu2 xi2
h i P
var β̂0 = P . (45)
n (xi − x̄)2


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
Recalling from equation (12) the residual ûi = yi − ŷi is of the form
ûi = ui − (β̂0 − β0 ) − (β̂1 − β1 )xi . (46)
This reminds us about the difference between the error term ui and
the residual term ûi .
An unbiased estimator of the error variance σu2 = var[ui ] is
n
1 X 2
σ̂u2 = ûi . (47)
n−2
i=1
Taking the (positive) square

p root gives an estimator for the error
standard deviation σu = σu2 , called usually the standard error of
regression r
1 X 2
σ̂u = ûi . (48)
n−2

Theorem 3
Under the assumption (1)–(5)
E σ̂u2 = σu2 ,

i.e., σ̂ 2 is an unbiased estimator of σu2 .
Proof: Omitted.


Definition
OLS Statistics

Nonlinearities

Unbiasedness
Variances
Replacing in (41) and (42) σu2 by σ̂u2 and taking square roots give
the standard error of β̂1 and β̂0
σ̂u
se(β̂1 ) = pP (49)
(xi − x̄)2
and s
1 x̄ 2
se(β̂0 ) = σ̂u +P . (50)
n (xi − x̄)2
These belong to the standard output of regression estimation, see

computer print-outs above.

As of Sep 16, 2020: Seppo Pynn Onen Econometrics I

Uploaded by

Copyright:

Available Formats

As of Sep 16, 2020: Seppo Pynn Onen Econometrics I

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

As of Sep 16, 2020: Seppo Pynn Onen Econometrics I

Uploaded by

Copyright:

Available Formats

The Simple Regression Model

The Simple Regression Model

As of Sep 16, 2020

Goodness-of-Fit, the R-square

Expected Values and Variances of the OLS Estimators

Two (observable) variables ”y ” and ”x”.

Equation (1) defines the simple regression model.

Seppo Pynnönen Econometrics I

The variable u, called the error term or disturbance, accounts

Thus, the (expected) change in y is β1 times the change in x.

Seppo Pynnönen Econometrics I

Error term ui is a combination of a number of effects, like:

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Goodness-of-Fit, the R-square

Expected Values and Variances of the OLS Estimators

Estimation of the model, OLS

Given observations (xi , yi ), i = 1, . . . , n, the task for an

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Exploring further 2.1, Wooldridge 5e, p.23

score = β0 + β1 attend + u. (2)

When would you expect this model to satisfy E[u|attend] = E[u]?

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

The goal in the estimation is to find values for β0 and β1 of model

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

First order conditions:

These yield so called normal equations

where the summation is from 1 to n.

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

β̂0 = ȳ − β̂1 x̄, (8)

Estimation of the model, OLS

Fitted regression line:

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Fitted regression line ŷ = β̂0 + β̂1 x in (11) is called also as the

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Population regression line and

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

Scatter plot of the data with regression line.

Figure 2.2: Wages and education.

Estimation of the model, OLS

Seppo Pynnönen Econometrics I

Estimation of the model, OLS

SAS PROC REG results:

The estimated model is