3.

1 Least squares in matrix form

E
Uses Appendix A.2A.4, A.6, A.7.
3.1.1 Introduction
More than one explanatory variable
In the foregoing chapter we considered the simple regression model where
the dependent variable is related to one explanatory variable. In practice the
situation is often more involved in the sense that there exists more than one
variable that inuences the dependent variable.
As an illustration we consider again the salaries of 474 employees at a
US bank (see Example 2.2 (p. 77) on bank wages). In Chapter 2 the vari-
ations in salaries (measured in logarithms) were explained by variations in
education of the employees. As can be observed from the scatter diagram in
Exhibit 2.5(a) (p. 85) and the regression results in Exhibit 2.6 (p. 86), around
half of the variance can be explained in this way. Of course, the salary of an
employee is not only determined by the number of years of education because
many other variables also play a role. Apart from salary and education, the
following data are available for each employee: begin or starting salary (the
salary that the individual earned at his or her rst position at this bank),
gender (with value zero for females and one for males), ethnic minority (with
value zero for non-minorities and value one for minorities), and job category
(category 1 consists of administrative jobs, category 2 of custodial jobs,
and category 3 of management jobs). The begin salary can be seen as an
indication of the qualities of the employee that, apart from education, are
determined by previous experience, personal characteristics, and so on. The
other variables may also affect the earned salary.
Simple regression may be misleading
Of course, the effect of each variable could be estimated by a simple regres-
sion of salaries on each explanatory variable separately. For the explanatory
variables education, begin salary, and gender, the scatter diagrams with
regression lines are shown in Exhibit 3.1 (ac). However, these results may
be misleading, as the explanatory variables are mutually related. For
118 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 118
9.5
10.0
10.5
11.0
11.5
12.0
9.0 9.5 10.0 10.5 11.0 11.5
LOGSALBEGIN
L
O
G
S
A
L
LOGSAL vs. LOGSALBEGIN
9.5
10.0
10.5
11.0
11.5
12.0
0.5 0.0 0.5 1.0 1.5
GENDER
L
O
G
S
A
L
LOGSAL vs. GENDER
9.0
9.5
10.0
10.5
11.0
11.5
0.5 0.0 0.5 1.0 1.5
GENDER
L
O
G
S
A
L
B
E
G
I
N
LOGSALBEGIN vs. GENDER
9.5
10.0
10.5
11.0
11.5
12.0
5 10 15 20 25
EDUC
L
O
G
S
A
L
LOGSAL vs. EDUC
9.0
9.5
10.0
10.5
11.0
11.5
5 10 15 20 25
EDUC
L
O
G
S
A
L
B
E
G
I
N
LOGSALBEGIN vs. EDUC
5
10
15
20
25
0.5 0.0 0.5 1.0 1.5
GENDER
E
D
U
C
EDUC vs. GENDER
(a) (b)
(c) (d)
(e)
(f )
Exhibit 3.1 Scatter diagrams of Bank Wage data
Scatter diagrams with regression lines for several bivariate relations between the variables
LOGSAL (logarithm of yearly salary in dollars), EDUC (nished years of education),
LOGSALBEGIN (logarithm of yearly salary when employee entered the rm) and GENDER
(0 for females, 1 for males), for 474 employees of a US bank.
3.1 Least squares in matrix form 119
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 119
example, the gender effect on salaries (c) is partly caused by the gender effect
on education (e). Similar relations between the explanatory variables are
shown in (d) and (f ). This mutual dependence is taken into account by
formulating a multiple regression model that contains more than one ex-
planatory variable.
3.1.2 Least squares
E
Uses Appendix A.7.
Regression model in matrix form
The linear model with several explanatory variables is given by the equation
y
i
= b
1
b
2
x
2i
b
3
x
3i
b
k
x
ki
e
i
(i = 1, , n): (3:1)
From now on we follow the convention that the constant term is denoted by
b
1
rather than a. The rst explanatory variable x
1
is dened by x
1i
= 1 for
every i = 1, , n, and for simplicity of notation we write b
1
instead of b
1
x
1i
.
For purposes of analysis it is convenient to express the model (3.1) in matrix
form. Let
y =
y
1
.
.
.
y
n
0
B
@
1
C
A, X =
1 x
21
x
k1
.
.
.
.
.
.
.
.
.
1 x
2n
x
kn
0
B
@
1
C
A, b =
b
1
.
.
.
b
k
0
B
@
1
C
A, e =
e
1
.
.
.
e
n
0
B
@
1
C
A: (3:2)
Note that in the n k matrix X = (x
ji
) the rst index j (j = 1, , k) refers to
the variable number (in columns) and the second index i (i = 1, , n) refers
to the observation number (in rows). The notation in (3.2) is common in
econometrics (whereas in books on linear algebra the indices i and j are often
reversed). In our notation, we can rewrite (3.1) as
y = Xb e: (3:3)
Here b is a k 1 vector of unknown parameters and e is an n 1 vector of
unobserved disturbances.
Residuals and the least squares criterion
If b is a k 1 vector of estimates of b, then the estimated model may be
written as
120 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 120
y = Xb e: (3:4)
Here e denotes the n 1 vector of residuals, which can be computed fromthe
data and the vector of estimates b by means of
e = y Xb: (3:5)
We denote transposition of matrices by primes (
/
) for instance, the trans-
pose of the residual vector e is the 1 n matrix e
/
= (e
1
, , e
n
). To deter-
mine the least squares estimator, we write the sum of squares of the residuals
(a function of b) as
S(b) =
X
e
2
i
= e
/
e = (y Xb)
/
(y Xb)
= y
/
y y
/
Xb b
/
X
/
y b
/
X
/
Xb:
(3:6)
Derivation of least squares estimator
The minimum of S(b) is obtained by setting the derivatives of S(b) equal to zero.
Note that the function S(b) has scalar values, whereas b is a column vector with k
components. So we have k rst order derivatives and we will followthe convention
to arrange them in a column vector. The second and third terms of the last expres-
sion in (3.6) are equal (a 1 1 matrix is always symmetric) and may be replaced by
2b
/
X
/
y. This is a linear expression in the elements of b and so the vector of
derivatives equals 2X
/
y. The last termof (3.6) is a quadratic formin the elements
of b. The vector of rst order derivatives of this term b
/
X
/
Xb can be written as
2X
/
Xb. The proof of this result is left as anexercise (see Exercise 3.1). Toget the idea
we consider the case k = 2 and we denote the elements of X
/
X by c
ij
, i, j = 1, 2,
with c
12
= c
21
. Then b
/
X
/
Xb = c
11
b
2
1
c
22
b
2
2
2c
12
b
1
b
2
. The derivative with re-
spect to b
1
is 2c
11
b
1
2c
12
b
2
and the derivative with respect to b
2
is
2c
12
b
1
2c
22
b
2
. When we arrange these two partial derivatives in a 2 1 vector,
this can be written as 2X
/
Xb. See Appendix A(especially Examples A.10 and A.11
in Section A.7) for further computational details and illustrations.
The least squares estimator
Combining the above results, we obtain
@S
@b
= 2X
/
y 2X
/
Xb: (3:7)
The least squares estimator is obtained by minimizing S(b). Therefore we set
these derivatives equal to zero, which gives the normal equations
X
/
Xb = X
/
y: (3:8)
T
3.1 Least squares in matrix form 121
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 121
Solving this for b, we obtain
b = (X
/
X)
1
X
/
y (3:9)
provided that the inverse of X
/
X exists, which means that the matrix X
should have rank k. As X is an n k matrix, this requires in particular that
n _ k that is, the number of parameters is smaller than or equal to the
number of observations. In practice we will almost always require that k is
considerably smaller than n.
Proof of minimum
From now on, if we write b, we always mean the expression in (3.9). This is the
classical formula for the least squares estimator in matrix notation. If the matrix X
has rank k, it follows that the Hessian matrix
@
2
S
@b@b
/
= 2X
/
X (3:10)
is a positive denite matrix (see Exercise 3.2). This implies that (3.9) is indeed
the minimum of (3.6). In (3.10) we take the derivatives of a vector
@S
@b

with
respect to another vector (b
/
) and we follow the convention to arrange these
derivatives in a matrix (see Exercise 3.2). An alternative proof that b minimizes
the sum of squares (3.6) that makes no use of rst and second order derivatives is
given in Exercise 3.3.
Summary of computations
The least squares estimates can be computed as follows.
Least squares estimation
v Step 1: Choice of variables. Choose the variable to be explained (y) and the
explanatory variables (x
1
, , x
k
, where x
1
is often the constant that
always takes the value 1).
v Step 2: Collect data. Collect n observations of y and of the related values of
x
1
, , x
k
and store the data of y in an n 1 vector and the data on the
explanatory variables in the n k matrix X.
v Step 3: Compute the estimates. Compute the least squares estimates by the
OLS formula (3.9) by using a regression package.
E
Exercises: T: 3.1, 3.2.
T
122 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 122
3.1.3 Geometric interpretation
E
Uses Sections 1.2.2, 1.2.3; Appendix A.6.
Least squares seen as projection
The least squares method can be given a geometric interpretation, which we
discuss now. Using the expression (3.9) for b, the residuals may be written as
e = y Xb = y X(X
/
X)
1
X
/
y = My (3:11)
where
M= I X(X
/
X)
1
X
/
: (3:12)
The matrix M is symmetric (M
/
= M) and idempotent (M
2
= M). Since it
also has the property MX = 0, it follows from (3.11) that
X
/
e = 0: (3:13)
We may write the explained component ^yy of y as
^yy = Xb = Hy (3:14)
where
H = X(X
/
X)
1
X
/
(3:15)
is called the hat matrix, since it transforms y into ^yy (pronounced: y-hat).
Clearly, there holds H
/
= H, H
2
= H, H M= I and HM= 0. So
y = Hy My = ^yy e
where, because of (3.11) and (3.13), ^yy
/
e = 0, so that the vectors ^yy and e
are orthogonal to each other. Therefore, the least squares method can be
given the following interpretation. The sum of squares e
/
e is the square of
the length of the residual vector e = y Xb. The length of this vector is
minimized by choosing Xb as the orthogonal projection of y onto the space
spanned by the columns of X. This is illustrated in Exhibit 3.2. The projec-
tion is characterized by the property that e = y Xb is orthogonal to
all columns of X, so that 0 = X
/
e = X
/
(y Xb). This gives the normal
equations (3.8).
3.1 Least squares in matrix form 123
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 123
Geometry of least squares
Let S(X) be the space spanned by the columns of X (that is, the set of all n 1
vectors that can be written as Xa for some k 1 vector a) and let S
l
(X) be the space
orthogonal to S(X) (that is, the set of all n 1 vectors z with the property that
X
/
z = 0). The matrix H projects onto S(X) and the matrix M projects onto S
l
(X).
In y = ^yy e, the vector y is decomposed into two orthogonal components, with
^yy S(X) according to (3.14) and e S
l
(X) according to (3.13). The essence of this
decomposition is given in Exhibit 3.3, which can be seen as a two-dimensional
version of the three-dimensional picture in Exhibit 3.2.
Geometric interpretation as a tool in analysis
This geometric interpretation can be helpful to understand some of the algebraic
properties of least squares. As an example we consider the effect of applying linear
transformations on the set of explanatory variables. Suppose that the n k matrix
X is replaced by X
+
= XA where A is a k k invertible matrix. Then the least
squares t (^yy), the residuals (e), and the projection matrices (H and M) remain
unaffected by this transformation. This is immediately evident from the geometric
pictures in Exhibits 3.2 and 3.3, as S(X
+
) = S(X).
T
e=My
y
0
Xb=Hy
X-plane
Exhibit 3.2 Least squares
Three-dimensional geometric impression of least squares, the vector of observations on
the dependent variable y is projected onto the plane of the independent variables X to obtain
the linear combination Xb of the independent variables that is as close as possible to y.
e = My
0
y
Xb = Hy
S(X)
Exhibit 3.3 Least squares
Two-dimensional geometric impression of least squares where the k-dimensional plane S(X) is
represented by the horizontal line, the vector of observations on the dependent variable y is
projected onto the space of the independent variables S(X) to obtain the linear combination Xb
of the independent variables that is as close as possible to y.
T
124 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 124
The properties can also be checked algebraically, by working out the expres-
sions for ^yy, e, H, and M in terms of X
+
. The least squares estimates change after
the transformation, as b
+
= (X
/
+
X
+
)
1
X
/
+
y = A
1
b. For example, suppose that
the variable x
k
is measured in dollars and x
+
k
is the same variable measured in
thousands of dollars. Then x
+
ki
= x
ki
=1000 for i = 1, , n, and X
+
= XA where A
is the diagonal matrix diag(1, , 1, 0:001). The least squares estimates of b
j
for
j ,= k remain unaffected that is, b
+
j
= b
j
for j ,= k, and b
+
k
= 1000b
k
. This also
makes perfect sense, as one unit increase in x
+
k
corresponds to an increase of a
thousand units in x
k
.
E
Exercises: T: 3.3.
3.1.4 Statistical properties
E
Uses Sections 1.2.2, 1.3.2.
Seven assumptions on the multiple regression model
To analyse the statistical properties of least squares estimation, it is conveni-
ent to use as conceptual background again the simulation experiment de-
scribed in Section 2.2.1 (p. 878). We rst restate the seven assumptions of
Section 2.2.3 (p. 92) for the multiple regression model (3.3) and use the
matrix notation introduced in Section 3.1.2.
. Assumption 1: xed regressors. All elements of the n k matrix X con-
taining the observations on the explanatory variables are non-stochastic. It
is assumed that n _ k and that the matrix X has rank k.
. Assumption 2: random disturbances, zero mean. The n 1 vector e con-
sists of random disturbances with zero mean so that E[e] = 0, that is,
E[e
i
] = 0 (i = 1, , n).
. Assumption 3: homoskedasticity. The covariance matrix of the disturb-
ances E[ee
/
] exists and all its diagonal elements are equal to s
2
, that is,
E[e
2
i
] = s
2
(i = 1, , n).
. Assumption 4: no correlation. The off-diagonal elements of the covariance
matrix of the disturbances E[ee
/
] are all equal to zero, that is, E[e
i
e
j
] = 0 for
all i ,= j.
. Assumption 5: constant parameters. The elements of the k 1 vector b
and the scalar s are xed unknown numbers with s > 0.
. Assumption 6: linear model. The data on the explained variable y have
been generated by the data generating process (DGP)
y = Xb e: (3:16)
3.1 Least squares in matrix form 125
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 125
. Assumption 7: normality. The disturbances are jointly normally distrib-
uted.
Assumptions 3 and 4 can be summarized in matrix notation as
E[ee
/
] = s
2
I, (3:17)
where I denotes the n n identity matrix. If in addition Assumption 7 is
satised, then e follows the multivariate normal distribution
e ~ N(0, s
2
I):
Assumptions 4 and 7 imply that the disturbances e
i
, i = 1, , n are mutually
independent.
Least squares is unbiased
The expected value of b is obtained by using Assumptions 1, 2, 5, and 6.
Assumption 6 implies that the least squares estimator b = (X
/
X)
1
X
/
y can be
written as
b = (X
/
X)
1
X
/
(Xb e) = b (X
/
X)
1
X
/
e:
Taking expectations is a linear operationthat is, if z
1
and z
2
are two
random variables and A
1
and A
2
are two non-random matrices of
appropriate dimensions so that z = A
1
z
1
A
2
z
2
is well dened, then
E[z] = A
1
E[z
1
] A
2
E[z
2
]. From Assumptions 1, 2, and 5 we obtain
E[b] = E[b (X
/
X)
1
X
/
e] = b (X
/
X)
1
X
/
E[e] = b: (3:18)
So b is unbiased.
The covariance matrix of b
Using the result (3.18), we obtain that under Assumptions 16 the covariance
matrix of b is given by
var(b) =E[(b b)(b b)
/
] =E[(X
/
X)
1
X
/
ee
/
X(X
/
X)
1
]
=(X
/
X)
1
X
/
E[ee
/
]X(X
/
X)
1
=(X
/
X)
1
X
/
(s
2
I)X(X
/
X)
1
=s
2
(X
/
X)
1
: (3:19)
The diagonal elements of this matrix are the variances of the estimators of
the individual parameters, and the off-diagonal elements are the covariances
between these estimators.
126 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 126
Least squares is best linear unbiased
The GaussMarkov theorem, proved in Section 2.2.5 (p. 978) for the simple
regression model, also holds for the more general model (3.16). It states that,
among all linear unbiased estimators, b has minimal variance that is, b is
the best linear unbiased estimator (BLUE) in the sense that, if
^
bb = Ay with A
a k n non-stochastic matrix and E[
^
bb] = b, then var(
^
bb) var(b) is a positive
semidenite matrix. This means that for every k 1 vector c of constants
there holds c
/
(var(
^
bb) var(b))c _ 0, or, equivalently, var(c
/
b) _ var(c
/
^
bb).
Choosing for c the jth unit vector, this means in particular that for the jth
component var(b
j
) _ var(
^
bb
j
) so that the least squares estimators are efcient.
This result holds true under Assumptions 16, the assumption of normality is
not needed.
Proof of GaussMarkov theorem
To prove the result, rst note that the condition that E[
^
bb] = E[Ay] =
AE[y] = AXb = b for all b implies that AX = I, the k k identity matrix. Now
dene D = A (X
/
X)
1
X
/
, then DX = AX(X
/
X)
1
X
/
X = I I = 0 so that
var(
^
bb) = var(Ay) = var(Ae) = s
2
AA
/
= s
2
DD
/
s
2
(X
/
X)
1
,
where the last equality follows by writing A = D(X
/
X)
1
X
/
and working out
AA
/
. This shows that var(
^
bb) var(b) = s
2
DD
/
, which is positive semidenite, and
zero if and only if D = 0 that is, A = (X
/
X)
1
X
/
. So
^
bb = b gives the minimal
variance.
E
Exercises: T: 3.4.
3.1.5 Estimating the disturbance variance
Derivation of unbiased estimator
Next we consider the estimation of the unknown variance s
2
. As in the previous
chapter we make use of the sumof squared residuals e
/
e. Intuition could suggest to
estimate s
2
= E[e
2
i
] by the sample mean
1
n
P
e
2
i
=
1
n
e
/
e, but this estimator is not
unbiased. It follows from (3.11) and (3.16) and the fact that MX = 0 that
e = My = M(Xb e) = Me. So
E[e] = 0, (3:20)
var(e) = E[ee
/
] = E[Mee
/
M] = ME[ee
/
]M= s
2
M
2
= s
2
M: (3:21)
T
T
3.1 Least squares in matrix form 127
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 127
To evaluate E[e
/
e] it is convenient to use the trace of a square matrix, which is
dened as the sum of the diagonal elements of this matrix. Because the trace and
the expectation operator can be interchanged, we nd, using the property that
tr(AB) = tr(BA), that
E[e
/
e] = E[tr(ee
/
) ] = tr(E[ee
/
] ) = s
2
tr(M):
Using the property that tr(A B) = tr(A) tr(B) we can simplify this as
tr(M) = tr(I
n
X(X
/
X)
1
X
/
) = n tr(X(X
/
X)
1
X
/
)
= n tr(X
/
X(X
/
X)
1
) = n tr(I
k
) = n k,
where the subscripts denote the order of the identity matrices.
The least squares estimator s
2
and standard errors
This shows that E[e
/
e] = (n k)s
2
so that
s
2
=
e
/
e
n k
(3:22)
is an unbiased estimator of s
2
. The square root s of (3.22) is called the
standard error of the regression. If in the expression (3.19) we replace s
2
by s
2
and if we denote the jth diagonal element of (X
/
X)
1
by a
jj
, then s

a
jj
_
is
called the standard error of the estimated coefcient b
j
. This is an estimate of
the standard deviation s

a
jj
_
of b
j
.
Intuition for the factor 1/(n k)
The result in (3.22) can also be given a more intuitive interpretation. Suppose
we would try to explain y by a matrix X with k = n columns and rank k.
Then we would obtain e = 0, a perfect t, but we would not have obtained
any information on s
2
. Of course this is an extreme case. In practice we
conne ourselves to the case k < n. The very fact that we choose b in such a
way that the sum of squared residuals is minimized is the cause of the fact
that the squared residuals are smaller (on average) than the squared disturb-
ances. Let us consider a diagonal element of (3.21),
var(e
i
) = s
2
(1 h
i
), (3:23)
where h
i
is the ith diagonal element of the matrix H = I M in (3.15). As H
is positive semidenite, it follows that h
i
_ 0. If the model contains a con-
stant term (so that the matrix X contains a column of ones), then h
i
> 0 (see
Exercise 3.7). So each single element e
i
of the residual vector has a variance
128 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 128
that is smaller than s
2
, and therefore the sumof squares
P
e
2
i
has an expected
value less than ns
2
. This effect becomes stronger when we have more
parameters to obtain a good t for the data. If one would like to use a
small residual variance as a criterion for a good model, then the denominator
(n k) of the estimator (3.22) gives an automatic penalty for choosing
models with large k.
Intuition for the number of degrees of freedom (n k)
As e = Me, it follows under Assumptions 17 that e
/
e=s
2
= e
/
Me=s
2
follows
the w
2
-distribution with (n k) degrees of freedom. This follows from the
results in Section 1.2.3 (p. 32), using the fact that M is an idempotent matrix
with rank (n k). The term degrees of freedom refers to the restrictions
X
/
e = 0. We may partition this as X
/
1
e
1
X
/
2
e
2
= 0, where X
/
1
is a k (n k)
matrix and X
/
2
a k k matrix. If the matrix X
/
2
has a rank less than k, we may
rearrange the columns of X
/
in such a way that X
/
2
has rank k. The restric-
tions imply that, once we have freely chosen the n k elements of e
1
, the
remaining elements are dictated by e
2
= (X
/
2
)
1
X
/
1
e
1
. This is also clear from
Exhibit 3.3. For given matrix X of explanatory variables, the residual vector
lies in S
l
(X) and this space has dimension (n k). That is, k degrees of
freedom are lost because b has been estimated.
E
Exercises: T: 3.5, 3.7a.
3.1.6 Coefficient of determination
Derivation of R
2
The performance of least squares can be evaluated by the coefcient of determin-
ation R
2
that is, the fraction of the total sample variation
P
(y
i
y)
2
that is
explained by the model.
In matrix notation, the total sample variation can be written as y
/
Ny with
N = I
1
n
ii
/
,
where i = (1, , 1)
/
is the n 1 vector of ones. The matrix N has the property
that it takes deviations from the mean, as the elements of Ny are y
i
y. Note that
N is a special case of an M-matrix (3.12) with X = i, as i
/
i = n. So Ny can be
interpreted as the vector of residuals and y
/
Ny = (Ny)
/
Ny as the residual sum of
squares from a regression where y is explained by X = i. If X in the multiple
regression model (3.3) contains a constant term, then the fact that X
/
e = 0
implies that i
/
e = 0 and hence Ne = e. From y = Xb e we then obtain
Ny = NXb Ne = NXb e = explained residual, and
T
3.1 Least squares in matrix form 129
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 129
y
/
Ny = (Ny)
/
Ny = (NXb e)
/
(NXb e)
= b
/
X
/
NXb e
/
e:
Here the cross term vanishes because b
/
X
/
Ne = 0, as Ne = e and X
/
e = 0. It
follows that the total variation in y (SST) can be decomposed in an explained
part SSE = b
/
X
/
NXb and a residual part SSR = e
/
e.
Coefficient of determination: R
2
Therefore R
2
is given by
R
2
=
SSE
SST
=
b
/
X
/
NXb
y
/
Ny
= 1
e
/
e
y
/
Ny
= 1
SSR
SST
: (3:24)
The third equality in (3.24) holds true if the model contains a constant term.
If this is not the case, then SSR may be larger than SST (see Exercise 3.7) and
R
2
is dened as SSE=SST (and not as 1 SSR=SST). If the model contains a
constant term, then (3.24) shows that 0 _ R
2
_ 1. It is left as an exercise (see
Exercise 3.7) to show that R
2
is the squared sample correlation coefcient
between y and its explained part ^yy = Xb. In geometric terms, R (the square
root of R
2
) is equal to the length of NXb divided by the length of Ny that
is, R is equal to the cosine of the angle between Ny and NXb. This is
illustrated in Exhibit 3.4. A good t is obtained when Ny is close to
NXbthat is, when the angle between these two vectors is small. This
corresponds to a high value of R
2
.
Adjusted R
2
When explanatory variables are added to the model, then R
2
never decreases
(see Exercise 3.6). The wish to penalize models with large k has motivated an
adjusted R
2
dened by adjusting for the degrees of freedom.
Ne
0
Ny
NXb
j
Exhibit 3.4 Geometric picture of R
2
Two-dimensional geometric impression of the coefcient of determination. The dependent
variable and all the independent variables are taken in deviation fromtheir sample means, with
resulting vector of dependent variables Ny and matrix of independent variables NX. The
explained part of Ny is NXb with residuals Ne = e, and the coefcient of determination is
equal to the square of the cosine of the indicated angle j.
130 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 130
R
2
= 1
e
/
e=(n k)
y
/
Ny=(n 1)
= 1
n 1
n k
(1 R
2
): (3:25)
E
Exercises: T: 3.6a, b, 3.7b, c.
3.1.7 Illustration: Bank Wages
To illustrate the foregoing results we consider the data on salary and educa-
tion discussed earlier in Chapter 2 and in Section 3.1.1. We will discuss (i) the
data, (ii) the model, (iii) the normal equations and the least squares estimates,
(iv) the interpretation of the estimates, (v) the sums of squares and R
2
, and
(vi) the orthogonality of residuals and explanatory variables.
(i) Data
The data consist of a cross section of 474 individuals working for a US bank.
For each employee, the information consists of the following variables:
salary (S), education (x
2
), begin salary (B), gender (x
4
= 0 for females,
x
4
= 1 for males), minority (x
5
= 1 if the individual belongs to a minority
group, x
5
= 0 otherwise), job category (x
6
= 1 for clerical jobs, x
6
= 2 for
custodial jobs, and x
6
= 3 for management positions), and some further job-
related variables.
(ii) Model
As a start, we will consider the model with y = log (S) as variable to be
explained and with x
2
and x
3
= log (B) as explanatory variables. That is,
we consider the regression model
y
i
= b
1
b
2
x
2i
b
3
x
3i
e
i
(i = 1, , n):
(iii) Normal equations and least squares estimates
As before, to simplify the notation we dene the rst regressor by x
1i
= 1. The
normal equations (3.8) involve the cross product terms X
/
X and X
/
y. For the
data at hand they are given (after rounding) in Exhibit 3.5, Panel 1. Solving
the normal equations (3.8) gives the least squares estimates
shown in Panel 3 in Exhibit 3.5, so that (after rounding) b
1
= 1:647,
b
2
= 0:023, and b
3
= 0:869. It may be checked from the cross products in
Panel 1 in Exhibit 3.5 that X
/
Xb = X
/
y (apart fromrounding errors) that is,
474 6395 4583
6395 90215 62166
4583 62166 44377
0
@
1
A
1:647
0:023
0:869
0
@
1
A
=
4909
66609
47527
0
@
1
A
:
E
XM301BWA
3.1 Least squares in matrix form 131
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 131
(iv) Interpretation of estimates
A rst thing to note here is that the marginal relative effect of education on
wage (that is,
dS=S
dx
2
=
d log (S)
dx
2
=
dy
dx
2
= b
2
) is estimated now as 0.023, whereas in
Chapter 2 this effect was estimated as 0.096 with a standard error of 0.005
(see Exhibit 2.11, p. 103). This is a substantial difference. That is, an
Panel 1 IOTA LOGSAL EDUC LOGSALBEGIN
IOTA 474
LOGSAL 4909 50917
EDUC 6395 66609 90215
LOGSALBEGIN 4583 47527 62166 44377
Panel 2 LOGSAL EDUC LOGSALBEGIN
LOGSAL 1.000000
EDUC 0.696740 1.000000
LOGSALBEGIN 0.886368 0.685719 1.000000
Panel 3: Dependent Variable: LOGSAL
Method: Least Squares
Sample: 1 474
Included observations: 474
Variable Coefcient Std. Error
C 1.646916 0.274598
EDUC 0.023122 0.003894
LOGSALBEGIN 0.868505 0.031835
R-squared 0.800579
Adjusted R-squared 0.799733
S.E. of regression 0.177812
Sum squared resid 14.89166
Total sum of squares 74.67462
Explained sum of squares 59.78296
Panel 4: Dependent Variable: RESID
Method: Least Squares
Sample: 1 474
Included observations: 474
Variable Coefcient
C 3.10E-11
EDUC 2.47E-13
LOGSALBEGIN 3.55E-12
R-squared 0.000000
Adjusted R-squared 0.004246
S.E. of regression 0.177812
Sum squared resid 14.89166
Exhibit 3.5 Bank Wages (Section 3.1.7)
Panel 1 contains the cross product terms (X
/
X and X
/
y) of the variables (iota denotes the
constant term with all values equal to one), Panel 2 shows the correlations between the
dependent and the two independent variables, and Panel 3 shows the outcomes obtained by
regressing salary (in logarithms) on a constant and the explanatory variables education and the
logarithm of begin salary. The residuals of this regression are denoted by RESID, and Panel 4
shows the result of regressing these residuals on a constant and the two explanatory variables
(3.10E-11 means 3.10+10
11
, and so on; these values are zero up to numerical rounding).
132 3 Multiple Regression
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 132
additional year of education corresponds on average with a 9.6 per cent
increase in salary. But, if the begin salary is kept xed, an additional year of
education gives only a 2.3 per cent increase in salary. The cause of this
difference is that the variable begin salary is strongly related to the variable
education. This is clear from Panel 2 in Exhibit 3.5, which shows that x
2
and x
3
have a correlation of around 69 per cent. We refer also to Exhibit 3.1
(d), which shows a strong positive relation between x
2
and x
3
. This means
that in Chapter 2, where we have excluded the begin salary from the model,
part of the positive association between education and salary is due to a third
variable, begin salary. This explains why the estimated effect in Chapter 2 is
larger.
(v) Sums of squares and R
2
The sums of squares for this model are reported in Panel 3 in Exhibit 3.5,
with values SST = 74:675, SSE = 59:783, and SSR = 14:892, so that
R
2
= 0:801. This is larger than the R
2
= 0:485 in Chapter 2 (see Exhibit
2.6, p. 86). In Section 3.4 we will discuss a method to test whether this is a
signicant increase in the model t. Panel 3 in Exhibit 3.5 also reports the
standard error of the regression s =

SSR=(474 3)
p
= 0:178 and the stand-
ard error of b
2
0.0039.
(vi) Orthogonality of residuals and explanatory variables
Panel 4 in Exhibit 3.5 shows the result of regressing the least squares
residuals on the variables x
1
, x
2
, and x
3
. This gives an R
2
= 0, which is in
accordance with the property that the residuals are uncorrelated with the
explanatory variables in the sense that X
/
e = 0 (see Exhibits 3.2 and 3.4).
3.1 Least squares in matrix form 133
Heij / Econometric Methods with Applications in Business and Economics Final Proof 28.2.2004 3:03pm page 133