1 The Econometrics of the Simple

Regression Model
Multiple regression model with I explanatory vari-
ables:
Y
i
= c + o
1
A
1i
+ o
2
A
2i
+ .. + o
I
A
Ii
+ .
i
,
where i subscripts to denote individual observations and
we have i = 1, .., . observations.
In econometrics, lots of uncertainty.
Uncertain what the regression coecients, c, o
1
, .., o
I
are (and, hence, have to estimate them).
Uncertain whether a hypothesis (e.g. o
)
= 0) is
true (and, hence, have to derive hypothesis testing
procedures).
We are uncertain about what future values of Y
might be (and, hence, have to derive procedures for
forecasting).
Probability provides us with a language and a formal
structure for dealing with uncertainty.
In this chapter, we will use probability to do some
key statistical derivations.
To keep formulae simple, will work with simple re-
gression model (i.e. regression model with one ex-
planatory variable) with no intercept:
j
i
= oA
i
+ .
i
where i = 1, .., . and A
i
is a scalar.
Derivations for multiple regression model are con-
ceptually similar but formulae get complicated (use
of matrix algebra usually involved)
1.1 A Review of Basic Concepts in Proba-
bility in the Context of the Regression
Model
See Appendix B for details, here we present basic
ideas informally.
Assume Y is a random variable.
Regression model provides description about what
probable values for the dependent variable are.
E.g. Y is the price of a house and A is a size of
house.
What if you knew that A = 5000 square feet (a
typical value in our data set), but did not know Y
A house with A = 5000 might sell for roughly
$70, 000 or $60, 000 or $50, 000 (which are typical
values in our data set), but it will not sell for $1, 000
(far too cheap) or $1, 000, 000 (far too expensive).
Econometricians use probability density functions (p.d.f.)
to summarize which are plausible and which are im-
plausible values for the house.
Figure 3.1 is example of a p.d.f.: tells you range of
plausible values which Y might take when A =
5, 000.
Figure 3.1 a Normal distribution
Bell-shaped curve. The curve is highest for the most
plausible values that the house price might take.
Chapter 1 introduced the ideas of a mean (or ex-
pected value) and variance.
The mean is the "average" or "typical" value of a
variable
Variance as being a measure of how dispersed a vari-
able is.
The exact shape of any Normal distribution depends
on its mean and its variance.
"Y is a random variable which has a Normal p.d.f.
with mean j and variance o
2
" is written:
Y .
_
j, o
2
_
Figure 3.1 has j = 61.153 $61, 153 is the mean,
or average, value for a house with a lot size of 5, 000
square feet.
o
2
= 683.812 (not much intuitive interpretation
other than it reects dispersion range of plausible
values)
P.d.f.s measure uncertainty about a random variable
since areas under the curve dened by the p.d.f. are
probabilities.
E.g. Figure 3.2. The area under the curve between
the points 60 and 100 is shaded in.
Shaded area is probability that that the house is
worth between $60, 000 and $100, 000.
This probability is 45% and can be written as:
Pr (60 Y 100) = 0.45.
Normal probabilities can be calculated using statis-
tical tables (or econometrics software packages).
By denition, the entire area under any p.d.f. is 1.
1.1.1 Expected Value and Variance
In this book, we are repeatedly using expected value
and variance operators.
Best way to learn these is through reading/doing
problem sets.
The expected value of A, denoted 1 (A), can be
interpreted as an average or typical value that might
occur.
Expected value is also called the mean, often denoted
by the symbol j. Thus, j = 1 (A).
The variance
ov (A) = 1
_
(A j)
2
_
= 1
_
A
2
_
j
2
The standard deviation is the square root of the vari-
ance.
Variance and standard deviation are commonly-used
measures of dispersion of a random variable.
Covariance between two random variables, A and
Y , dened as:
cc (A, Y ) = 1 (AY ) 1 (A) 1 (Y ) .
Covariance best motivated through correlation be-
tween A and Y :
ccvv (A, Y ) =
cc (A, Y )
_
ov (A) ov (Y )
.
Correlation is degree of association between two ran-
dom variables. It satises 1 ccvv (A, Y ) 1
with larger positive/negative values indicating stronger
positive/negative relationships between A and Y .
If A and Y are independent, then ccvv (A, Y ) = 0.
1.1.2 Properties of Expected Values and Variance
If A and Y are two random variables and o and b are
constants, then:
1. 1 (oA + bY ) = o1 (A) + b1 (Y )
2. ov (Y ) = 1 (Y ) [1 (Y )]
2
3. ov (oY ) = o
2
ov (Y )
4. ov (o + Y ) = ov (Y )
5. cc (A, Y ) = 1 (AY ) 1 (A) 1 (Y )
6. ov (oA + bY ) = o
2
ov (A)+b
2
ov (Y )+2obcc (A, Y )
7. 1 (AY ) ,= 1 (A) 1 (Y ) unless cc (A, Y ) = 0.
8. If A and Y Normally distributed then o.
i
+ b.
2
is
also Normal. "Linear combinations of Normals are
Normal".
These properties generalize to the case of many random
variables
1.1.3 Using Normal Statistical Tables
Table for standard Normal distribution i.e. . (0, 1)
is in textbook.
Can use . (0, 1) tables to gure out probabilities for
the .
_
j, o
2
_
for any j and o
2
.
If Y .
_
j, o
2
_
, what are mean and variance of
Z =
Y j
o
.
As an example of a proof using properties of expected
value operator:
1 (Z) = 1
_
Y j
o
_
=
1 (Y j)
o
=
1 (Y ) j
o
=
j j
o
= 0.
As an example of a proof using properties of variance:
ov (Z) = ov
_
Y j
o
_
=
ov (Y j)
o
2
=
ov (Y )
o
2
=
o
2
o
2
= 1.
Thus, Z is . (0, 1) and we can use our statistical
tables
Z is often referred to as a Z-score.
For any random variable, if you subtract o its mean
and divide by standard deviation always get a new
random variable with mean zero and variance one
Example: In Figure 3.2 how did we obtain
Pr (60 Y 100) = 0.45
Remember Figure 3.2 has Y . (61.153, 683.812).
Pr (60 Y 100)
= Pr
_
60j
o

Y j
o

100j
o
_
= Pr
_
6061.153
_
683.812

Y 61.153
_
683.812

10061.153
_
683.812
_
= Pr (0.04 Z 1.49)
.
Now probability involves the standard Normal distri-
bution
Normal statistical tables say Pr (0.04 Z 1.49) =
0.45.
Details: break into two parts as
Pr (0.04 Z 1.49)
= Pr (0.04 Z 0) + Pr (0 Z 1.49)
From table Pr (0 Z 1.49) = 0.4319.
But since the Normal is symmetric Pr (0.04 Z 0) =
Pr (0 Z 0.04)=0.0160.
Adding these two probabilities together gives 0.4479
1.2 The Classical Assumptions for the Re-
gression Model
Now let us return to the regression model.
We need to make some assumptions to do any statistical
derivations and start with the classical assumptions
1. 1 (Y
i
) = oA
i
.
2. ov (Y
i
) = o
2
.
3. cc
_
Y
i
, Y
)
_
= 0 for i ,= ).
4. Y
i
is Normally distributed
5. A
i
is xed. It is not a random variable.
Compact notation: Y
i
are independent .
_
oA
i
, o
2
_
.
An equivalent way of writing the classical assumptions is:
1. 1 (.
i
) = 0 mean zero errors.
2. ov (.
i
) = 1
_
.
2
i
_
= o
2
constant variance errors
(homoskedasticity).
3. cc
_
.
i
.
)
_
= 0 for i ,= ).
4. .
i
is Normally distributed
5. A
i
is xed. It is not a random variable.
1.3 Motivation for Classical Assumptions
Regression model ts a straight-line through an XY-
plot.
1 (Y
i
) = oA
i
is the linearity assumption.
Second assumption: all observations have the same
variance (homoskedasticity).
Ex. where this might not be a good assumption.
House price data. Small houses all the same. Big
houses more diverse. If so, house prices might be
more diverse for big houses (heteroskedasticity).
Third assumption: observations uncorrelated with one
another.
This assumption is usually reasonable with cross-
sectional data (e.g. in a survey, response of person1
and person 2 are unrelated).
For time series data not a good assumption (e.g.
interest rate now and last month are correlated with
one another)
Fourth assumption (Y is Normal), harder to moti-
vate.
In many empirical applications, Normality is reason-
able.
Asymptotic theory can be used to relax this assump-
tion. We will not cover this in this course (but see
Appendix C and Appendices at end of several chap-
ters)
Fifth assumption (explanatory variable not a random
variable) is good in experimental sciences, but maybe
not in social sciences.
We will talk about relaxing these assumptions in later
chapters.
1.4 The Ordinary Least Squares (OLS) Es-
timator of o
j
i
= oA
i
+ .
i
OLS estimator is chosen to minimize:
.

i=1
.
2
i
This can be done using calculus.

o =
.

i=1
A
i
j
i
.

i=1
A
2
i
1.4.1 Properties of OLS Estimator

o =

A
i
j
i

A
2
i
=

A
i
(A
i
o + .
i
)

A
2
i
= o +

A
i
.
i

A
2
i
(*)
Property 1: OLS is unbiased under the classical as-
sumptions
1
_

o
_
= o
Proof:
1
_

o
_
= 1
_
_
o +

A
i
.
i

A
2
i
_
_
= o + 1
_
_

A
i
.
i

A
2
i
_
_
= o +
1

A
2
i
1
_

A
i
.
i
_
= o +
1

A
2
i

A
i
1 (.
i
)
= o
Use equation (*) and properties of expected value opera-
tor. Remember A
i
is not random (hence can be treated
as a constant).
Property 2: Variance of OLS estimator under the
classical assumptions
ov
_

o
_
=
o
2

A
2
i
Proof:
ov
_

o
_
= ov
_
_
o +

A
i
.
i

A
2
i
_
_
= ov
_
_

A
i
.
i

A
2
i
_
_
=
_
_
1

A
2
i
_
_
2
ov
_

A
i
.
i
_
=
_
_
1

A
2
i
_
_
2

A
2
i
ov (.
i
)
=
_
_
1

A
2
i
_
_
2
o
2

A
2
i
=
o
2

A
2
i
Use equation (*) and properties of variance operator. Re-
member A
i
is not random (hence can be treated as a
constant).
Property 3: Distribution of OLS estimator under
classical assumptions

o is .
_
_
o,
o
2

A
2
i
_
_
Proof: Properties 1 and 2 plus "linear combinations of
Normals are Normal" theorem.
Property 3 is important since it can be used to derive
condence intervals and hypothesis tests.
The OLS estimator is a random variable and has a p.d.f.
Ex. Figure 3.3 if

o is . (2, 1)
Want your estimator to be unbiased and want it to have
as small a variance as possible.
An unbiased estimator is said to be ecient relative to
another if it has a smaller variance.
Property 4: The Gauss-Markov Theorem
If the classical assumptions hold, then OLS is the best,
linear unbiased estimator,
where best = minimum variance
linear = linear in y.
Short form: "OLS is BLUE"
Note: the assumption of Normal errors is NOT required
to prove Gauss-Markov theorem. Hence, OLS is BLUE
even if errors are not Normal.
Property 5: Under the Classical Assumptions, OLS
is the maximum likelihood estimator
Maximum likelihood is another statistical principal for
choosing estimators.
Textbook has a discussion of this topic, but I do not have
time to cover in lectures.
1.4.2 Deriving a Condence Interval for o
Assume o
2
known (discuss relaxing this assumption later).
Use Property 3 to obtain:
Z =

o o
_
o
2

A
2
i
is . (0, 1)
Then can use statistical tables for the Normal distribution
to make probability statements. For instance,
Pr [1.96 Z 1.96] = 0.95
To get 95% condence interval, rearrange the inequalities
to put o in the middle:
Pr
_

_
1.96

o o
_
o
2

A
2
i
1.96
_

_
= 0.95
rearranging:
Pr
_
_

o 1.96

_
o
2

A
2
i
o

o + 1.96

_
o
2

A
2
i
_
_
= 0.95
Note:

o is a random variable, o is not a random variable.
Hence, we do not say "probability interval" but rather
"condence interval".
95% condence interval is
_
_

o 1.96

_
o
2

A
2
i
o

o + 1.96

_
o
2

A
2
i
_
_
commonly written as:

o 1.96

_
o
2

A
2
i
Other condence levels can be handled by getting dif-
ferent number from Normal tables. For instance, 90%
condence interval would replace "1.96)" by "1.64" in
previous equations.
1.4.3 Hypothesis tests about o
Assume o
2
known (discuss relaxing this assumption later).
Basic idea in testing any hypothesis, 1
0
:
The econometrician accepts 1
0
if the calculated value
of the test statistic is consistent with what could plausibly
happen if 1
0
is true.
List the general steps in hypothesis testing along with the
specic steps for this case.
Step1: Specify a hypothesis, 1
0
.
1
0
: o = o
0
(where o
0
is known, usually o
0
= 0)
Step 2: Specify a test statistic
Z =

o o
_
o
2

A
2
i
Step 3: Figure out distribution of test statistic assuming
1
0
is true.
Z =

o o
0
_
o
2

A
2
i
is . (0, 1) .
Step 4: Choose a level of signicance (usually 5%).
0.05
Step 5: Use Steps 3 and 4 to get a critical value.
Critical value = 1.96 (from Normal statistical tables)
Step 6: Calculate your test statistic and compare to crit-
ical value. Reject 1
0
if absolute value of test statistic is
greater than critical value (else accept H
0
).
Reject if [Z[ 1.96.
1.5 Modications when o
2
is unknown
o
2
appears in previous formula for condence interval and
test statistic. What to do when it is unknown?
1.5.1 Estimation of o
2
Residuals:

.
i
= j
i

oA
i
Unbiased estimator of o
2
is
c
2
=

.
2
i
. 1
Property (not proved in this course):
1
_
c
2
_
= o
2
.
Note: The . 1 in denominator becomes . I 1
in multiple regression where I is number of explanatory
variables.
1.5.2 Condence interval for o when o
2
unknown
Replace o
2
by c
2
in equations from earlier section "De-
riving a Condence Interval for o".
Nothing changes, except:
Z =

o o
_
o
2

A
2
i
is . (0, 1)
is replaced by:
Z =

o o
_
c
2

A
2
i
is t (. 1) ,
where t (. 1) is the Student-t distribution with N-1
degrees of freedom. And must use Student-t statistical
tables instead of Normal.
See Appendix B for instruction in using Student-t tables.
Example:
Suppose we have . = 21.
Before (with o
2
known) we derived

o 1.96

_
o
2

A
2
i
.
Now we have to look in t(20) row of Student-t statistical
tables and obtain:

o 2.08

_
c
2

A
2
i
.
1.5.3 Hypothesis testing about o when o
2
unknown
Replace o
2
by c
2
in equations from earlier section "Hy-
pothesis tests about o".
Nothing changes, except test statistic:
Z =

o o
0
_
o
2

A
2
i
is . (0, 1)
is replaced by:
Z =

o o
0
_
c
2

A
2
i
is t (. 1) ,
where t (. 1) is the Student-t distribution with N-1
degrees of freedom.
Must use Student-t statistical tables instead of Normal
to get critical value.
1.5.4 Note on P-values
All relevant computer packages now present P-values for
hypothesis tests. This means you do not need to look
up critical values in statistical tables (so no emphasis on
tables in this course).
Useful (but not quite correct) intuition: "P-value is the
probability that 1
0
is true"
A correct interpretation: "P-value equals the smallest level
of signicance at which you can reject 1
0
"
Example: If P-value is .04 you can reject 1
0
at 5% level
of signicance or 10% or 20% (or any number above 4%).
You cannot reject 1
0
at 1% level of signicance.
Common rule of thumb:
Reject 1
0
if P-value less than .05.
1.6 Chapter Summary
The major points and derivations covered in this chapter
include:
1. The manner in which the Normal distribution (which
is characterized by a mean and variance) is used in
the context of the simple regression model.
2. The introduction of the classical assumptions, from
which all else in this chapter is derived.
3. The properties of the OLS estimator, including a
proof that it is unbiased and a derivation of its dis-
tribution (i.e.

o is .
_
_
o,
o
2

A
2
i
_
_
).
4. The Gauss-Markov theorem which says OLS is BLUE
under the classical assumptions.
5. The derivation of a condence interval for o (assum-
ing o
2
is known).
6. The derivation of a test of the hypothesis that o = 0
(assuming o
2
is known).
7. The OLS estimator of o
2
.
8. How the condence interval and hypothesis test are
modied when o
2
is unknown.