Chapter 3

Multiple Linear Regression Model

We consider the problem of regression when study variable depends on more than one explanatory or
independent variables, called as multiple linear regression model. This model generalizes the simple linear
regression in two ways. It allows the mean function E ( y ) to depend on more than one explanatory
variables and to have shapes other than straight lines, although it does not allow for arbitrary shapes.

The linear model:

Let y denotes the dependent (or study) variable that is linearly related to k independent (or explanatory)

variables X 1 , X 2 ,..., X k through the parameters β1 , β 2 ,..., β k and we write

y X 1β1 + X 2 β 2 + ... + X k β k + ε .
=

This is called as the multiple linear regression model. The parameters β1 , β 2 ,..., β k are the regression

coefficients associated with X 1 , X 2 ,..., X k respectively and ε is the random error component reflecting the
difference between the observed and fitted linear relationship. There can be various reasons for such
difference, e.g., joint effect of those variables not included in the model, random factors which can not be
accounted in the model etc.

Note that the j th regression coefficient β j represents the expected change in y per unit change in j th

independent variable X j . Assuming E (ε ) = 0,

∂E ( y )
βj = .
∂X j

Linear model:
∂y ∂E ( y )
A model is said to be linear when it is linear in parameters. In such a case (or equivalently )
∂β j ∂X j

should not depend on any β ' s . For example

i) y β 0 + β1 X is a linear model as it is linear is parameter.
=

ii) y = β 0 X β1 can be written as

log y log β 0 + β1 log X

=
y=
*
β 0* + β1 x*
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
1
 which is linear is parameter β 0* and β1 , but nonlinear is variables
= =
y* log y, x* log x. So it is a
linear model.
iii) y =β 0 + β1 X + β 2 X 2

is linear in parameters β 0 , β1 and β 2 but it is nonlinear is variables X . So it is a linear model

β1
iv) y β0 +
=
X − β2
is nonlinear in parameters and variables both. So it is a nonlinear model.
v) y β 0 + β1 X β2
=
is nonlinear in parameters and variables both. So it is a nonlinear model.
vi) y =β 0 + β1 X + β 2 X 2 + β3 X 3
is a cubic polynomial model which can be written as
y =β 0 + β1 X + β 2 X 2 + β3 X 3

which is linear in parameters β 0 , β1 , β 2 , β3 and linear in variables

= X 1 X=
, X 2 X=
2
, X 3 X 3 . So it is
a linear model.

Example:
The income and education of a person are related. It is expected that, on an average, higher level of
education provides higher income. So a simple linear regression model can be expressed as
β 0 + β1 education + ε .
income =

Not that β1 reflects the change is income with respect to per unit change is education and β 0 reflects the

income when education is zero as it is expected that even an illiterate person can also have some income.

Further this model neglects that most people have higher income when they are older than when they are
young, regardless of education. So β1 will over-state the marginal impact of education. If age and
education are positively correlated, then the regression model will associate all the observed increase in
income with an increase in education. So better model is
β 0 + β1 education + β 2 age + ε .
income =

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

2
 Often it is observed that the income tends to rise less rapidly in the later earning years than is early years. To
accommodate such possibility, we might extend the model to
β 0 + β1education + β 2age + β3age 2 + ε
income =

This is how we proceed for regression modeling in real life situation. One needs to consider the experimental
condition and the phenomenon before taking the decision on how many, why and how to choose the
dependent and independent variables.

Model set up:

Let an experiment be conducted n times and the data is obtained as follows:
Observation Response Explanatory variables
number y X1 X2  Xk

1 y1 x11 x12  x1k

2 y2 x21 x22  x2 k
     
n yn xn1 xn 2  xnk

Assuming that the model is

y = β 0 + β1 X 1 + β 2 X 2 + ... + β k X k + ε ,
the n-tuples of observations are also assumed to follow the same model. Thus they satisfy
y1 = β 0 + β1 x11 + β 2 x12 + ... + β k x1k + ε1
y2 = β 0 + β1 x21 + β 2 x22 + ... + β k x2 k + ε 2
 
yn = β 0 + β1 xn1 + β 2 xn 2 + ... + β k xnk + ε n .

These n equations can be written as

 y1  1 x11 x12  x1k   β 0   ε1 
      
 y2  1 x21 x22  x2 k   β1   ε 2 
= +
          
      
 yn  1 xn1 xn 2  xnk   β k   ε n 

y X β + ε.
or =

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

3
In general, the model with k explanatory variables can be expressed as
y Xβ +ε
=
where y = ( y1 , y2 ,..., yn ) ' is a n × 1 vector of n observation on study variable,

 x11 x12  x1k 

 
x x22  x2 k 
X =  21
    
 
 xn1 xn 2  xnk 

is a n × k matrix of n observations on each of the k explanatory variables, β = ( β1 , β 2 ,..., β k ) ' is a k ×1

vector of regression coefficients and ε = (ε1 , ε 2 ,..., ε n ) ' is a n × 1 vector of random error components or
disturbance term.

If intercept term is present, take first column of X to be (1,1,…,1)’.

Assumptions in multiple linear regression model

Some assumptions are needed in the y X β + ε for drawing the statistical inferences.
model = The
following assumptions are made:
(i) E (ε ) = 0

(ii) E (εε ') = σ 2 I n

(iii) Rank ( X ) = k
(iv) X is a non-stochastic matrix
(v) ε ~ N (0, σ 2 I n ) .

These assumptions are used to study the statistical properties of estimator of regression coefficients. The
following assumption is required to study particularly the large sample properties of the estimators

 X 'X 
(vi) lim   = ∆ exists and is a non-stochastic and nonsingular matrix (with finite elements).
n →∞
 n 

The explanatory variables can also be stochastic in some cases. We assume that X is non-stochastic unless
stated separately.

We consider the problems of estimation and testing of hypothesis on regression coefficient vector under the
stated assumption.
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
4
Estimation of parameters:
A general procedure for the estimation of regression coefficient vector is to minimize
n n

i
=i 1 =i 1
∑ M=
(ε ) ∑ M ( y − x β i i1 1 − xi 2 β 2 − ... − xik β k )

for a suitably chosen function M .

Some examples of choice of M are
M ( x) = x
M ( x) = x 2

M ( x) = x , in general.
p

We consider the principle of least square which is related to M ( x) = x 2 and method of maximum likelihood
estimation for the estimation of parameters.

Principle of ordinary least squares (OLS)

Let B be the set of all possible vectors β . If there is no further information, the B is k -dimensional real
Euclidean space. The object is to find a vector b ' = (b1 , b2 ,..., bk ) from B that minimizes the sum of squared

deviations of ε i ' s, i.e.,

n

∑ ε i2 ==
S (β ) =
i =1
ε ' ε ( y − X β ) '( y − X β )

for given y and X . A minimum will always exist as S ( β ) is a real valued, convex and differentiable
function. Write
S (β ) =
y ' y + β ' X ' X β − 2β ' X ' y .
Differentiate S ( β ) with respect to β
∂S ( β )
= 2X ' X β − 2X ' y
∂β
∂ 2 S (β )
= 2 X ' X (atleast non-negative definite).
∂β 2
The normal equation is
∂S ( β )
=0
∂β
⇒ X ' Xb = X 'y

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

5
where the following result is used:
Result: If f ( z ) = Z ' AZ is a quadratic form, Z is a m ×1 vector and A is any m × m symmetric matrix
∂
then F ( z ) = 2 Az .
∂z

Since it is assumed that rank ( X ) = k (full rank), then X ' X is positive definite and unique solution of
normal equation is
b = ( X ' X ) −1 X ' y
which is termed as ordinary least squares estimator (OLSE) of β .

∂ 2 S (β )
Since is at least non-negative definite, so b minimize S ( β ) .
∂β 2

In case, X is not of full rank, then

=b ( X ' X ) − X ' y +  I − ( X ' X ) − X ' X  ω

where ( X ' X ) − is the generalized inverse of X ' X and ω is an arbitrary vector. The generalized inverse

( X ' X ) − of X ' X satisfies

X ' X ( X ' X )− X ' X = X ' X

X ( X ' X )− X ' X = X
X ' X ( X ' X )− X ' = X '
Theorem:
(i) Let ŷ = Xb be the empirical predictor of y . Then ŷ has the same value for all solutions b of
X ' Xb = X ' y.
(ii) S ( β ) attains the minimum for any solution of X ' Xb = X ' y.
Proof:
(i) Let b be any member in
=b ( X ' X ) − X ' y +  I − ( X ' X ) − X ' X  ω .

Since X ( X ' X ) − X ' X = X , so then

=Xb X ( X ' X ) − X ' y + X  I − ( X ' X ) − X ' X  ω

= X ( X ' X )− X ' y
which is independent of ω . This implies that ŷ has same value for all solution b of X ' Xb = X ' y.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

6
(ii) Note that for any β ,

S ( β ) =[ y − Xb + X (b − β ) ]′ [ y − Xb + X (b − β ) ]
= ( y − Xb)′( y − Xb) + (b − β )′ X ' X (b − β ) + 2(b − β )′ X ′( y − Xb)
= ( y − Xb)′( y − Xb) + (b − β )′ X ' X (b − β ) (Using X ' Xb = X ' y )
≥ ( y − Xb)′( y − Xb) = S (b)
= y ' y − 2 y ' Xb + b ' X ' Xb
= y ' y − b ' X ' Xb
= y ' y − yˆ ' yˆ .

Fitted values:
If β̂ is any estimator of β for the model =
y X β + ε , then the fitted values are defined as

ŷ = X βˆ where β̂ is any estimator of β .

In case of βˆ = b,
yˆ = Xb
= X ( X ' X ) −1 X ' y
= Hy

where H = X ( X ' X ) −1 X ' is termed as Hat matrix which is

(i) symmetric
(ii) idempotent (i.e., HH = H ) and

(iii) H tr X ( X ′X ) −1=
tr= ) −1 tr=
X ' tr X ' X ( X ' X= Ik k .

Residuals
The difference between the observed and fitted values of study variable is called as residual. It is denoted
as
e = y ~ yˆ
= y − yˆ
= y − Xb
= y − Hy
= (I − H ) y
= Hy

where H = I − H .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

7
Note that
(i) H is a symmetric matrix
(ii) H is an idempotent matrix, i.e.,
HH =( I − H )( I − H ) =( I − H ) =H and

(iii) trH =trI n − trH =(n − k ).

Properties of OLSE
(i) Estimation error:
The estimation error of b is
b − β ( X ' X ) −1 X ' y − β
=
= ( X ' X ) −1 X '( X β + ε ) − β
= ( X ' X ) −1 X ' ε

(ii) Bias
Since X is assumed to be nonstochastic and E (ε ) = 0

E (b − β ) = ( X ' X ) −1 X ' E (ε )
= 0.
Thus OLSE is an unbiased estimator of β .

(iii) Covariance matrix

The covariance matrix of b is
V (b) =E (b − β )(b − β ) '
= E ( X ' X ) −1 X ' εε ' X ( X ' X ) −1 
= ( X ' X ) −1 X ' E (εε ') X ( X ' X ) −1
= σ 2 ( X ' X ) −1 X ' IX ( X ' X ) −1
= σ 2 ( X ' X ) −1.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

8
 (iv) Variance
The variance of b can be obtained as the sum of variances of all b1 , b2 ,..., bk which is the trace of covariance

matrix of b . Thus
Var (b) = tr [V (b) ]
k
= ∑ E (b − β )
i =1
i i
2

k
= ∑ Var (bi ).
i =1

Estimation of σ 2
The least squares criterion can not be used to estimate σ 2 because σ 2 does not appear in S ( β ) . Since

E (ε i2 ) = σ 2 , so we attempt with residuals ei to estimate σ 2 as follows:

e= y − yˆ
= y − X ( X ' X ) −1 X ' y
= [ I − X ( X ' X ) −1 X '] y
= Hy.
Consider the residual sum of squares
n
SS r e s = ∑ ei2
i =1

= e 'e
= ( y − Xb) '( y − Xb)
=y '( I − H )( I − H ) y
= y '( I − H ) y
= y ' Hy.
Also
SS r e s =
( y − Xb) '( y − Xb)
=
y ' y − 2b ' X ' y + b ' X ' Xb
=
y ' y − b ' X ' y (Using X ' Xb =
X ' y)

SS r e s = y ' Hy
(X β + ε )'H (X β + ε )
=
= ε=
' H ε (Using HX 0)

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

9
Since ε ~ N (0, σ 2 I ) .

So y ~ N ( X β , σ 2 I ) .

Hence y ' Hy ~ χ 2 (n − k ) .

] (n − k )σ 2
Thus E[ y ' Hy=

 y ' Hy 
or E  =σ2
 n−k 
or E [ MS r e s ] = σ 2

SS r e s
where MS r e s = is the mean sum of squares due to residual.
n−k

Thus an unbiased estimator of σ 2 is

σˆ 2 MS
= = res s 2 (say)
which is a model dependent estimator.

Variance of ŷ
The variance of ŷ is
V ( yˆ ) = V ( Xb)
= XV (b) X '
= σ 2 X ( X ' X ) −1 X '
= σ 2H.

Gauss-Markov Theorem:
The ordinary least squares estimator (OLSE) is the best linear unbiased estimator (BLUE) of β .
Proof: The OLSE of β is

b = ( X ' X ) −1 X ' y
which is a linear function of y . Consider the arbitrary linear estimator

b* = a ' y
of linear parametric function  ' β where the elements of a are arbitrary constants.

Then for b* ,
= (a ' y ) a ' X β
E (b* ) E=

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

10
and so b* is an unbiased estimator of  ' β when

=
E ' X β 'β
(b* ) a=
⇒ a'X =  '.
Since we wish to consider only those estimators that are linear and unbiased, so we restrict ourselves to
those estimators for which a ' X =  '.

Further
=
Var (a ' y ) a= 'Var ( y )a σ 2 a ' a
Var ( ' b) =  'Var (b)
= σ 2 a ' X ( X ' X ) −1 X ' a.

Consider
Var (a ' y ) − Var ( ' b) =σ 2  a ' a − a ' X ( X ' X ) −1 X ' a 
= σ 2 a '  I − X ( X ' X ) −1 X ' a
= σ 2 a '( I − H )a.
Since ( I − H ) is a positive semi-definite matrix, so
Var (a ' y ) − Var ( ' b) ≥ 0 .

This reveals that if b* is any linear unbiased estimator then its variance must be no smaller than that of b .
Consequently b is the best linear unbiased estimator, where ‘best’ refers to the fact that b is efficient
within the class of linear and unbiased estimators.

Maximum likelihood estimation:

y X β + ε , it is assumed that the errors are normally and independently distributed with
In the model=

constant variance σ 2 or ε ~ N (0, σ 2 I ).

The normal density function for the errors is
1  1 
f (ε i ) = exp  − 2 ε i2  i = 1, 2,..., n. .
σ 2π  2σ 
The likelihood function is the joint density of ε1 , ε 2 ,..., ε n given as

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

11
 n
L( β , σ 2 ) = ∏ f (ε i )
i =1

 1 n 2
1
=
(2πσ 2 ) n /2
exp  − 2σ 2 ∑ ε i 
 i =1 
1  1 
= exp  − 2 ε ' ε 
(2πσ )
2 n /2
 2σ 
1  1 
= exp  − 2 ( y − X β ) '( y − X β )  .
(2πσ ) 2 n /2
 2σ 
Since the log transformation is monotonic, so we maximize ln L( β , σ 2 ) instead of L( β , σ 2 ) .
n 1
ln L( β , σ 2 ) =
− ln(2πσ 2 ) − 2 ( y − X β ) '( y − X β ) .
2 2σ
The maximum likelihood estimators (m.l.e.) of β and σ 2 are obtained by equating the first order
derivatives of ln L( β , σ 2 ) with respect to β and σ 2 to zero as follows:

∂ ln L( β , σ 2 ) 1
= 2 X '( y − X
= β) 0
∂β 2σ 2
∂ ln L( β , σ 2 ) n 1
=
− 2+ ( y − X β ) '( y − X β ).
∂σ 2
2σ 2(σ 2 ) 2
The likelihood equations are given by
X 'Xβ = X 'y
1
σ 2 =( y − X β ) '( y − X β ).
n
Since rank( X ) = k , so that the unique mle of β and σ 2 are obtained as

β = ( X ' X ) −1 X ' y
1
σ 2 =( y − X β ) '( y − X β ).
n
Further to verify that these values maximize the likelihood function, we find
∂ 2 ln L( β , σ 2 ) 1
= − 2 X 'X
∂β 2
σ
∂ 2 ln L( β , σ 2 ) n 1
= 4 − 6 ( y − X β ) '( y − X β )
∂ (σ )
2 2 2
2σ σ
∂ 2 ln L( β , σ 2 ) 1
− 4 X '( y − X β ).
=
∂β∂σ 2
σ

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

12
Thus the Hessian matrix of second order partial derivatives of ln L( β , σ 2 ) with respect to β and σ 2 is

 ∂ 2 ln L( β , σ 2 ) ∂ 2 ln L( β , σ 2 ) 
 
 ∂β 2 ∂β∂σ 2 
 ∂ 2 ln L( β , σ 2 ) ∂ 2 ln L( β , σ 2 ) 
 
 ∂σ 2 ∂β ∂ 2 (σ 2 ) 2 

which is negative definite at β = β and σ 2 = σ 2 . This ensures that the likelihood function is maximized at
these values.

Comparing with OLSEs, we find that

(i) OLSE and m.l.e. of β are same. So m.l.e. of β is also an unbiased estimator of β .
n−k 2
(ii) OLSE of σ 2 is s 2 which is related to m.l.e. of σ 2 as σ 2 = s . So m.l.e. of σ 2 is a
n
biased estimator of σ 2 .

Consistency of estimators
(i) Consistency of b :
 X 'X 
Under the assumption that lim   = ∆ exists as a nonstochastic and nonsingular matrix (with finite
n →∞
 n 
elements), we have
−1
1 X 'X 
lim V (b) = σ 2 lim  
n →∞ n →∞ n
 n 
1
= σ 2 lim ∆ −1
n →∞ n

= 0.
This implies that OLSE converges to β in quadratic mean. Thus OLSE is a consistent estimator of β . This
holds true for maximum likelihood estimators also.

Same conclusion can also be proved using the concept of convergence in probability.
An estimator θˆn converges to θ in probability if

lim P  θˆn − θ ≥ δ  =
0 for any δ > 0
n →∞  

and is denoted as plim(θˆn ) = θ .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

13
The consistency of OLSE can be obtained under the weaker assumption that
 X 'X 
plim   = ∆* .
 n 
exists and is a nonsingular and nonstochastic matrix such that
 X 'ε 
plim   = 0.
 n 
Since
b−β =
( X ' X ) −1 X ' ε
−1
 X ' X  X 'ε
=  .
 n  n
So
−1
 X 'X   X 'ε 
plim(b − β ) = plim   plim  
 n   n 
= ∆*−1.0
= 0.
Thus b is a consistent estimator of β . Same is true for m.l.e. also.

(ii) Consistency of s 2
Now we look at the consistency of s 2 as an estimate of σ 2 as
1
s2 = e 'e
n−k
1
= ε ' Hε
n−k
−1
1 k 
1 −  ε ' ε − ε ' X ( X ' X ) X ' ε 
−1
=
n n
 k   ε 'ε ε ' X  X ' X  X 'ε 
−1 −1

=
1 −   −   .
 n   n n  n  n 

ε 'ε 1 n 2
Note that
n
consists of ∑ ε i and {ε i2 , i = 1, 2,..., n} is a sequence of independently and identically
n i =1

distributed random variables with mean σ 2 . Using the law of large numbers

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

14
  ε 'ε 
 =σ
2
plim 
 n 
 ε ' X  X ' X  −1 X ' ε   ε 'X    X ' X  
−1
X 'ε 
plim     =  plim   plim     plim 
 n  n  n   n    n    n 
= 0.∆*−1.0
=0
(1 − 0) −1 σ 2 − 0 
⇒ plim( s 2 ) =
= σ 2.

Thus s 2 is a consistent estimator of σ 2 . Same hold true for m.l.e. also.

Cramer-Rao lower bound

Let θ = ( β , σ 2 ) ' . Assume that both β and σ 2 are unknown. If E (θˆ) = θ , then the Cramer-Rao lower

bound for θˆ is grater than or equal to the matrix inverse of

 ∂ 2 ln L(θ ) 
I (θ ) = − E  
 ∂θ∂θ ' 
  ∂ ln L( β , σ 2 )   ∂ ln L( β , σ 2 )  
− E   −E 
  ∂β 2   ∂β∂σ
2

=
  ∂ ln L( β , σ 2 )   ∂ ln L( β , σ 2 )  
− E   −E 
  ∂σ ∂β   ∂ (σ )  
2 2 2 2

  X 'X   X '( y − X β )  
− E − σ 2  −E
σ  
   
4
= 
 (y − X β )' X   n ( y − X β ) '( y − X β )  
− E   −E 4 −  
  σ4  2σ σ6 
X 'X 
 σ2 0 
= .
 0 n 
 2σ 4 
Then
σ 2 ( X ' X ) −1 0 
[ I (θ )] =  
−1
2σ 4 
0
 n 
is the Cramer-Rao lower bound matrix of β and σ 2 .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

15
 The covariance matrix of OLSEs of β and σ 2 is

σ 2 ( X ' X ) −1 0 
∑ OLS =  0 2σ 4 

 n − k 
which means that the Cramer-Rao have bound is attained for the covariance of b but not for s 2 .

Standardized regression coefficients:

Usually it is difficult to compare the regression coefficients because the magnitude of βˆ j reflects the units

of measurement of j th explanatory variable X j . For example, in the following fitted regression model

yˆ =+
5 X 1 + 1000 X 2 ,

y is measured in liters, X 1 is liters and X 2 in milliliters. Although βˆ2 >> βˆ1 but effect of both explanatory

variables is identical. One liter change in either X 1 and X 2 when other variable is held fixed produces the

same change is ŷ .

Sometimes it is helpful to work with scaled explanatory variables and study variable that produces
dimensionless regression coefficients. These dimensionless regression coefficients are called as
standardized regression coefficients.

There are two popular approaches for scaling which gives standardized regression coefficients. We discuss
them as follows:

1. Unit normal scaling:

Employ unit normal scaling to each explanatory variable and study variable .
So define
xij − x j
=zij = , i 1,=
2,..., n, j 1, 2,..., k
sj
yi − y
yi* =
sy

1 n
=
where s 2j ∑
n − 1 i =1
( xij − x j ) 2

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

16
 1 n
=
and s y2 ∑
n − 1 i =1
( yi − y ) 2

are the sample variances of j th explanatory variable and study variable respectively.

All scaled explanatory variable and scaled study variable have mean zero and sample variance unity, i.e.,
using these new variables, the regression model becomes
yi*= γ 1 zi1 + γ 2 zi 2 + ... + γ k zik + ε i , i= 1, 2,..., n.

Such centering removes the intercept term from the model. The least squares estimate of γ = (γ 1 , γ 2 ,..., γ k ) '
is
γˆ = ( Z ' Z ) −1 Z ' y* .

This scaling has a similarity to standardizing a normal random variable, i.e., observation minus its mean and
divided by its standard deviation. So it is called as a unit normal scaling.

2. Unit length scaling:

In unit length scaling, define
xij − x j
=ωij = , i 1,=
2,..., n; j 1, 2,..., k
S 1/2
jj

yi − y
yi0 =
SST1/2
n
where=
S jj ∑ (x
i =1
ij − x j ) 2 is the corrected sum of squares for j th explanatory variables X j and

n
= =
ST SST ∑ ( y − y)
i =1
i
2
is the total sum of squares. In this scaling, each new explanatory variable W j has

n
1 n
=
mean ωj = ∑ ωij 0 and length
n i =1
∑ (ω
i =1
ij − ω j )2 =
1.

In terms of these variables, the regression model is

y=
o
i δ1ωi1 + δ 2ωi 2 + ... + δ k ωik + ε i , =
i 1, 2,..., n.

The least squares estimate of regression coefficient δ = (δ1 , δ 2 ,..., δ k ) ' is

δˆ = (W 'W ) −1W ' y 0 .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

17
In such a case, the matrix W 'W is in the form of correlation matrix, i.e.,
1 r12 r13  r1k 
 
 r12 1 r23  r2 k 
W 'W =  r13 r23 1  r3k 
 
      
r r3k  1 
 1k r2 k

where
n

∑ (x ui − xi )( xuj − x j )
rij = u =1

( Sii S jj )1/2
Sij
=
( Sii S jj )1/2

is the simple correlation coefficient between the explanatory variables X i and X j . Similarly

W ' y o = (r1 y , r2 y ,..., rky ) '

where
n

∑ (x uj − x j )( yu − y )
rjy = u =1

( S jj SST )1/2
Siy
=
( S jj SST )1/2

is the simple correlation coefficient between j th explanatory variable X j and study variable y .

Note that it is customary to refer rij and rjy as correlation coefficient though X i ' s are not random variable.

If unit normal scaling is used, then

Z ' Z= (n − 1)W 'W .

So the estimates of regression coefficient in unit normal scaling (i.e., γˆ ) and unit length scaling (i.e., δˆ) are

identical. So it does not matter which scaling is used, so γˆ = δˆ .

The regression coefficients obtained after such scaling, viz., γˆ or δˆ usually called standardized regression
coefficients.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

18
 The relationship between the original and standardized regression coefficients is
1/ 2
 SST 
=b j δˆ=
j 
 S 
, j 1, 2,..., k
 jj 
and
k
b0= y − ∑ b j x j
j =1

where b0 is the OLSE of intercept term and b j are the OLSE of slope parameters.

The model in deviation form

The multiple linear regression model can also be expressed in the deviation form.

First all the data is expressed in terms of deviations from sample mean.

The estimation of regression parameters is performed in two steps:

• First step: Estimate the slope parameters.
• Second step : Estimate the intercept term.
The multiple linear regression model in deviation form is expressed as follows:
Let
1
A= I −  '
n
=
where  (1,1,...,1) ' is a n ×1 vector of each element unity. So

1 0  0 1 1  1
0 1  0  1 1
 1  1
=A  − .
     n    
   
0 0  1 1 1  1
Then
 y1 
 
1 1 n y
=y = ∑ y (1,1,...,1)  2 
n i =1
i
n 
 
 yn 
1
= ' y
n
Ay =y − y =( y1 − y , y2 − y ,..., yn − y ) '.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

19
Thus pre-multiplication of any column vector by A produces a vector showing those observations in
deviation form:

Note that
1
A=  −  ' 
n
1
=  − .n
n
= −
=0
and A is symmetric and idempotent matrix.

In the model
y X β +ε,
=
the OLSE of β is

b =(X 'X ) X 'y

−1

and residual vector is

e= y − Xb.

Note that Ae = e.

If the n × k matrix is partitioned as

X =  X 1 X 2* 

=
where X 1 (1,1,...,1) ' is n ×1 vector with all elements unity, X 2* is n × ( k − 1) matrix of observations of

( k − 1) explanatory variables X 2 , X 3 ,..., X k and OLSE b = ( b1 , b2* ') is suitably partitioned with OLSE of

intercept term β1 as b1 and b2 as a ( k − 1) ×1 vector of OLSEs associated with β 2 , β3 ,..., β k .

Then
y = X 1b1 + X 2*b2* + e.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

20
Premultiply by A,

Ay = AX 1b1 + AX 2*b2* + Ae
= AX 2*b2* + e.

Premultiply by X 2* gives

=
X 2* ' Ay X 2* ' AX 2*b2* + X 2* ' e
= X 2* ' AX 2*b2* .

Since A is symmetric and idempotent,

( AX ) ' ( Ay ) = ( AX ) ' ( AX ) b
*
2
*
2
*
2
*
2 ..

y X β + ε . Such a
This equation can be compared with the normal equations X ' y = X ' Xb in the model=
comparison yields following conclusions:
• b2* is the sub vector of OLSE.

• Ay is the study variables vector in deviation form.

• AX 2* is the explanatory variable matrix in deviation form.

• This is normal equation in terms of deviations. Its solution gives OLS of slope coefficients as

b2* = ( AX 2* ) ' ( AX 2* )  ( AX ) ' ( Ay ) .

−1
*
2

The estimate of intercept term is obtained in the second step as follows:

1
Premultiplying =
y Xb + e by  ' gives
n
1 1 1
= ' y  ' Xb +  ' e
n n n
 b1 
 
b
y 1 X 2 X 3 ... X k   2  + 0

 
 bk 
⇒ b1 = y − b2 X 2 − b3 X 3 − ... − bk X k .
Now we explain various sums of squares in terms of this model.

The expression of total sum of squares (TSS) remains same as earlier and is given by
TSS = y ' Ay.
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
21
Since
=
Ay AX 2*b2* + e
=
y ' Ay y ' AX 2*b2* + y ' e

( Xb + e ) ' AX 2*b2* + y ' e

=

= ( X 1b1 + X 2*b2* + e ) ' AX 2*b2* + ( X 1b1 + X 2*b2* + e ) ' e

= b2* ' X 2* ' AX 2*b2* + e ' e

= SS reg + SS res
TSS

where sum of squares due to regression is

SS reg = b2* ' X 2* ' AX 2*b2*

and sum of squares due to residual is

SS res = e ' e .

Testing of hypothesis:
There are several important questions which can be answered through the test of hypothesis concerning the
regression coefficients. For example
1. What is the overall adequacy of the model?
2. Which specific explanatory variables seems to be important?
etc.

In order the answer such questions, we first develop the test of hypothesis for a general framework, viz.,
general linear hypothesis. Then several tests of hypothesis can be derived as its special cases. So first we
discuss the test of a general linear hypothesis.

Test of hypothesis for H 0 : Rβ = r

We consider a general linear hypothesis that the parameters in β are contained in a subspace of parameter
space for which Rβ = r , where R is ( J × k ) matrix of known elements and r is a ( J ×1 ) vector of known
elements.

In general, the null hypothesis

H 0 : Rβ = r
is termed as general linear hypothesis and
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
22
 H1 : R β ≠ r
is the alternative hypothesis.

We assume that rank ( R) = J , i.e., full rank so that there is no linear dependence in the hypothesis.

Some special cases and interesting example of H 0 : R β = r are as follows:

(i) H 0 : βi = 0

Choose=
J 1,=
r 0,=
R [0, 0,..., 0,1, 0,..., 0] where 1 occurs at the i th position is R .
This particular hypothesis explains whether X i has any effect on the linear model or not.

(ii) H 0 : β3 β 4 or H 0=
= : β3 − β 4 0
Choose =
J 1,=
r 0,=
R [0, 0,1, −1, 0,..., 0]

(iii) H 0 : β=
3 β=
4 β5
or H 0 : β3 − β 4= 0, β3 − β5= 0

0 0 1 − 1 0 0 ... 0 
Choose =
J 2,=
r (0, 0) ',=
R  .
0 0 1 0 − 1 0 ... 0 
(iv) H 0 : β 3 +5β 4 =
2

Choose =
J 1,=
r 2,=
R [0, 0,1,5, 0...0]
(v)
H 0 : β 2= β3= ...= β k= 0
J= k − 1
r = (0, 0,..., 0) '
0 1 0 ... 0  0 
0 0 1 ... 0  0 I k −1 
R =   .
       
   
0 0 0 ... 1  ( k −1)×k 0 

This particular hypothesis explains the goodness of fit. It tells whether βi has linear effect or not and are

they of any importance. It also tests that X 2 , X 3 ,..., X k have no influence in the determination of y . Here

β1 = 0 is excluded because this involves additional implication that the mean level of y is zero. Our main
concern is to know whether the explanatory variables helps to explain the variation in y around its mean
value or not.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

23
 We develop the likelihood ratio test for H 0 : R β = r.

Likelihood ratio test:

The likelihood ratio test statistic is

max L( β , σ 2 | y, X ) Lˆ (Ω)
λ =
max L( β , σ | y, X , Rβ = r ) Lˆ (ω )
2

where Ω is the whole parametric space and ω is the sample space.

If both the likelihood are maximized, one constrained and the other unconstrained, then the value of the
unconstrained will not be smaller than the value of the constrained. Hence λ ≥ 1.

First we discus the likelihood ratio test for a simpler case when
=R I=
k and r β 0 ,=
i.e., β β 0 . This will give as better and detailed understanding for the minor details and
then we generalize it for Rβ = r , in general.

Likelihood ratio test for H 0 : β = β 0

Let the null hypothesis related to k ×1 vector β is

H 0 : β = β0

where β 0 is specified by the investigator. The elements of β 0 can take on any value, including zero. The
concerned alternative hypothesis is
H1 : β ≠ β 0 .

Since ε ~ N (0, σ 2 I ) in =
y X β + ε , so y ~ N ( X β , σ 2 I ). Thus the whole parametric space and sample
space are Ω and ω respectively given by

Ω : {( β , σ 2 ) : − ∞ < βi < ∞, σ 2 > 0, i =1, 2,..., k }

ω : {( β , σ =
2
) : β β 0 , σ 2 > 0} .

The unconstrained likelihood under Ω .

1  1 
L( β , σ 2 |=
y, X ) exp  − 2 ( y − X β ) '( y − X β )  .
(2πσ )
2 n /2
 2σ 

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

24
This is maximized over Ω when
β = ( X ' X ) −1 X ' y
1
σ 2 =( y − X β ) '( y − X β ).
n

where β and σ 2 are the maximum likelihood estimates of β and σ 2 which are the values maximizing the
likelihood function.
Lˆ (Ω) =max L ( β , σ 2 | y, X ) )
 
 
1  ( y − X β ) '( y − X β ) 
exp −
n
  2( y − X β ) '( y − X β )  
 2π   2   
 n ( y − X β ) '( y − X β )    n  
 n
n n /2 exp  − 
=  2 .
n
 
(2π ) ( y − X β ) '( y − X β ) 
n /2 2

The constrained likelihood under ω is

1  1 
Lˆ (ω ) =
max L( β , σ 2 | y, X , β =
β 0 ) = 2 n /2 exp  − 2 ( y − X β 0 ) '( y − X β 0 )  .
(2πσ )  2σ 
Since β 0 is known, so the constrained likelihood function has an optimum variance estimator

1
σω2 =( y − X β 0 ) '( y − X β 0 )
n
 n
n n /2 exp  − 
Lˆ (ω ) =  2 .
n /2
(2π ) ( y − X β 0 ) '( y − X β 0) 
n /2

The likelihood ratio is

 
 n n /2 exp(−n / 2) 
 (2π ) ( y − X β ) '( y − X β )  
n /2
   
n /2
Lˆ (Ω) 
=
Lˆ (ω )  
 n n /2 exp(−n / 2) 
 (2π ) n /2 ( y − X β ) '( y − X β )  n /2 
  0 0  
n /2
 ( y − X β 0 ) '( y − X β 0 ) 
= 
 ( y − X β ) '( y − X β ) 
n /2
 σω2 
= = (λ )
n /2
2 
 σ 

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

25
where
( y − X β 0 ) '( y − X β 0 )
λ=
( y − X β ) '( y − X β )
is the ratio the quadratic forms. Now we simplify the numerator in λ as follows:
′
( y − X β 0 ) '( y − X β 0 ) =( y − X β ) + X ( β − β 0 )  ( y − X β ) + X ( β − β 0 ) 
= ( y − X β ) '( y − X β ) + 2 y '  I − X ( X ' X ) −1 X ' X ( β − β 0 ) + ( β − β 0 ) ' X ' X ( β − β 0 )
= ( y − X β ) '( y − X β ) + ( β − β 0 ) ' X ' X ( β − β 0 ).

Thus
( y − X β ) '( y − X β ) + ( β − β 0 ) ' X ' X ( β − β 0 )
λ=
( y − X β ) '( y − X β )
( β − β 0 ) ' X ' X ( β − β 0 )
= 1+
( y − X β ) '( y − X β )
( β − β 0 ) ' X ' X ( β − β 0 )
or λ − 1= λ0 =
( y − X β ) '( y − X β )

where 0 ≤ λ0 < ∞.

Distribution of ratio of quadratic forms

Now we find the distribution of the quadratic forms involved is λ0 to find the distribution of λ0 as follows:

( y − X β ) '( y − X β ) =
e ' e
= y '  I − X ( X ' X ) −1 X ' y
= y ' Hy
(X β + ε )'H(X β + ε )
=
= ε=
' Hε (using HX 0)
= (n − k )σˆ 2

Result: If Z is a n × 1 random vector that is distributed as N (0, σ 2 I n ) and A is any symmetric

Z ' AZ
idempotent n × n matrix of rank p then ~ χ 2 ( p ). If B is another n × n symmetric idempotent
σ2
Z ' BZ
matrix of rank q , then ~ χ 2 (q ) . If AB = 0 then Z ' AZ is distributed independently of Z ' BZ .
σ 2

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

26
 So using this result, we have
y ' Hy (n − k )σˆ 2
= ~ χ 2 (n − k ).
2
σ2
σ
Further, if H 0 is true, then β = β 0 and we have the numerator in λ0 . Rewriting the numerator in λ0 , in
general, we have
( β − β ) ' X ' X ( β − β ) =
ε ' X ( X ' X ) −1 X ' X ( X ' X ) −1 X ' ε
= ε ' X ( X ' X ) −1 X ' ε
= ε ' Hε
where H is an idempotent matrix with rank k .

Thus using this result, we have

ε ' H ε ε ' X '( X ' X ) −1 X ' ε
= ~ χ 2 (k ).
σ 2
σ 2

Furthermore, the product of the quadratic form matrices in the numerator (ε ' H ε ) and denominator (ε ' H ε )

of λ0 is

 I − X ( X ' X ) −1 X ' X ( X ' X ) −1 X ' =−

X ( X ' X ) −1 X ' X ( X ' X ) −1 X ' X ( X ' X ) −1 X ' =
0

and hence the χ 2 random variables in numerator and denominator of λ0 are independent. Dividing each of

the χ 2 random variable by their respective degrees of freedom

 
   
 (β − β0 ) ' X ' X (β − β0 ) 
 σ2 
 k 
λ1 =  
  (n − k )σˆ 2  
 
  σ 2
 
 n−k  
   
   
( β − β 0 ) ' X ' X ( β − β 0 )
=
kσˆ 2
( y − X β 0 ) '( y − X β 0 ) − ( y − X β ) '( y − X β )
=
kσˆ 2
~ F (k , n − k ) under H 0 .
Note that
( y − X β 0 ) '( y − X β 0 ) : Restricted error sum of squares
( y − X β ) '( y − X β ) : Unrestricted error sum of squares
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
27
 Numerator in λ1 : Difference between the restricted and unrestricted error sum of squares.

The decision rule is to reject H 0 : β = β 0 at α level of significance whenever

λ1 ≥ Fα (k , n − k )
where Fα (k , n − k ) is the upper critical points on the central F -distribution with k and n − k degrees of
freedom.

Likelihood ratio test for H 0 : Rβ = r

The same logic and reasons used in the development of likelihood ratio test for H 0 : β = β 0 can be extended

to develop the likelihood ratio test for H 0 : R β = r as follows.

=Ω {(β ,σ 2
) : − ∞ < βi < ∞, σ=
2
> 0, i 1, 2,..., k }
ω
= {(β ,σ 2
Rβ r , σ 2 > 0} .
) : − ∞ < β i < ∞, =

Let β = ( X ' X ) −1 X ' y. .

Then
E ( Rβ ) = Rβ
V ( Rβ )= E  R( β − β )( β − β ) ' R '
= RV ( β ) R '
= σ 2 R ( X ' X ) −1 R '.

Since β ~ N  β , σ 2 ( X ' X ) −1 
so Rβ ~ N  Rβ , σ 2 R( X ' X ) −1 R '
Rβ − r= Rβ − Rβ= R ( β − β ) ~ N 0, σ 2 R ( X ' X ) −1 R ' .

−1
There exists a matrix Q such that  R ( X ' X ) −1 R ' = QQ ' and then

=ξ QR(b − β )  N (0, σ 2 I n ) . Therefore under H 0 : Rβ − r =0, so

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

28
 ξξ ' ( Rβ − r ) ' QQ '( Rβ − r )
=
σ2 σ2
−1
( Rβ − r ) '  R ( X ' X ) −1 R ' ( R β − r )
=
σ2
−1
( β − β ) ' R '  R( X ' X ) −1 R ' R ( β − β )
=
σ2
−1
ε ' X ( X ' X ) −1 R '  R( X ' X ) −1 R ' R ( X ' X ) −1 X ' ε
=
σ2
~ χ 2 ( J ).
−1
which is obtained as X ( X ' X ) −1 R '  R ( X ' X ) −1 R ' R ( X ' X ) −1 X ' is an idempotent matrix and its trace is J

which is the associated degrees of freedom.

Also, irrespective of whether H 0 is true or not,

e ' e ( y − X β ) '( y − X β ) y ' Hy (n − k )σˆ 2

= = = ~ χ 2 (n − k ).
σ 2
σ
2 2
σ 2
σ

Moreover, the product of quadratic form matrices of e ' e and

−1
( β − β ) ' R '  R( X ' X ) −1 R ' R ( β − β ) is zero implying that both the quadratic forms are independent. So in

terms of likelihood ratio test statistic

 ( Rβ − r ) '  R ( X ' X ) −1 R ' −1 ( R β − r ) 
   
 σ 2

 J 
 
λ1 =  
 (n − k )σˆ 2

 
 σ
2

n−k

( ( ))
−1
Rβ − r ) '  R ( X ' X ) −1 R ' R β − r
=
J σˆ 2
~ F ( J , n − k ) under H 0 .

So the decision rule is to reject H 0 whenever

λ1 ≥ Fα ( J , n − k )
where Fα ( J , n − k ) is the upper critical points on the central F distribution with J and (n − k ) degrees of
freedom.
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
29
 Test of significance of regression (Analysis of variance)
= I k −1 ], r 0, then the hypothesis H 0 : R β = r reduces to the following null hypothesis:
If we set R [0=

H 0 : β 2= β3= ...= β k= 0
against the alternative hypothesis
H1 : β j ≠ 0 for at least one j = 2,3,..., k

This hypothesis determines if there is a linear relationship between y and any set of the explanatory
variables X 2 , X 3 ,..., X k . Notice that X 1 corresponds to the intercept term in the model and hence

=xi1 1=
for all i 1, 2,..., n.

This is an overall or global test of model adequacy. Rejection of the null hypothesis indicates that at least
one of the explanatory variables among X 2 , X 3 ,..., X k . contributes significantly to the model. This is called
as analysis of variance.

Since ε ~ N (0, σ 2 I ),

so y ~ N ( X β , σ 2 I )
b = ( X ' X ) −1 X ' y ~ N  β , σ 2 ( X ' X ) −1  .

SS res
Also σˆ 2 =
n−k
( y − yˆ ) '( y − yˆ )
=
n−k
y '  I − X ( X ' X ) −1 X ' y y ' Hy y' y −b' X ' y
= = = .
n−k n−k n−k
Since ( X ' X )-1 X ' H = 0, so b and σˆ 2 are independently distributed.

Since y ' Hy = ε ' H ε and H is an idempotent matrix, so

SS r e s ~ χ (2n − k ) ,

i.e., central χ 2 distribution with (n − k ) degrees of freedom.

Partition X = [ X 1 , X 2* ] where the submatrix X 2* contains the explanatory variables X 2 , X 3 ,..., X k

and partition β = [ β1 , β 2* ] where the subvector β 2* contains the regression coefficients β 2 , β 3 ,..., β k .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

30
Now partition the total sum of squares due to y ' s as
SST = y ' Ay
= SS r e g + SS r e s

where SS r e g = b2* ' X 2* ' AX 2*b2* is the sum of squares due to regression and the sum of squares due to residuals

is given by
SS r e s =
( y − Xb) '( y − Xb)
= y ' Hy
= SST − SS r e g .

Further
SS r e g  β * ' X * ' AX * β *  β * ' X * ' AX * β *
~ χ k2−1  2 2 2 2 2  , i.e., non-central χ 2 distribution with non − centrality parameter 2 2 2 2 2 ,
σ2  2σ  2σ

SST  β 2* ' X 2* ' AX 2* β 2*  β 2* ' X 2* ' AX 2* β 2*

~χ 2
n −1   , i.e., non-central χ distribution with non − centrality parameter
2
.
σ2  2σ 2  2σ 2

Since X 2 H = 0, so SS r e g and SS r e s are independently distributed. The mean squares due to regression is

SS r e g
MS r e g =
k −1
and the mean square due to error is
SS r e s
MS res = .
n−k
Then
MS reg  β * ' X * ' AX * β * 
~ Fk −1,n − k  2 2 2 2 2 
MS res  2σ 
which is a non-central F -distribution with (k − 1, n − k ) degrees of freedom and noncentrality parameter

β 2* ' X 2* ' AX 2* β 2*
.
2σ 2
Under H 0 : β 2= β3= ...= β k ,

MS reg
F= ~ Fk −1,n − k .
MS res
The decision rule is to reject at α level of significance whenever
F ≥ Fα (k − 1, n − k ).

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

31
The calculation of F -statistic can be summarized in the form of an analysis of variance (ANOVA) table
given as follows:
Source of variation Sum of squares Degrees of freedom Mean squares F
Regression SS r e g k −1 =
MS reg SS r e g / k − 1 F
Error n−k
SS r e s =
MS res SS r e s /(n − k )

Total SST n −1

Rejection of H 0 indicates that it is likely that atleast one βi ≠ 0 (i =

1, 2,..., k ).

Test of hypothesis on individual regression coefficients

In case, if the test in analysis of variance is rejected, then another question arises is that which of the
regression coefficients is/are responsible for the rejection of null hypothesis. The explanatory variables
corresponding to such regression coefficients are important for the model.

Adding such explanatory variables also increases the variance of fitted values ŷ , so one need to be careful
that only those regressors are added that are of real value in explaining the response. Adding unimportant
explanatory variables may increase the residual mean square which may decrease the usefulness of the
model.

To test the null hypothesis

H0 : β j = 0

versus the alternative hypothesis

H1 : β j ≠ 0

has already been discussed is the case of simple linear regression model. In present case, if H 0 is accepted,

it implies that the explanatory variable X j can be deleted from the model. The corresponding test statistic

is
bj
=t ~ t (n − k − 1) under H 0
se(b j )

where the standard error of OLSE b j of β j is

se(b j ) = σˆ 2C jj where C jj denotes the j th diagonal element of ( X ' X ) −1 corresponding to b j .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

32
The decision rule is to reject H 0 at α level of significance if

t > tα .
, n − k −1
2

Note that this is only a partial or marginal test because βˆ j depends on all the other explanatory variables

X i (i ≠ j that are in the model. This is a test of the contribution of X j given the other explanatory variables

in the model.

Confidence interval estimation

The confidence intervals in multiple regression model can be constructed for individual regression
coefficients as well as jointly . We consider both of them as follows:

Confidence interval on the individual regression coefficient:

Assuming ε i ' s are identically and independently distributed following N (0, σ 2 ) in=
y X β + ε , we have

y ~ N ( X β ,σ 2 I )
b ~ N ( β , σ 2 ( X ' X ) −1 ).
Thus the marginal distribution of any regression coefficient estimate
b j ~ N ( β j , σ 2C jj )

where C jj is the j th diagonal element of ( X ' X ) −1 .

Thus
bj − β j
tj = ~ t (n − k ) under H 0 , j =1, 2,...
σˆ 2C jj

SS r e s y ' y − b ' X ' y

σˆ 2
where= = .
n−k n−k
So the 100(1 − α )% confidence interval for β j ( j = 1, 2,..., k ) is obtained as follows:

 bj − β j 
P  −tα ≤ ≤ tα  = 1−α
 2 ,n −k σˆ 2C jj ,n −k 
 2

 
P b j − t α σˆ 2C jj ≤ β j ≤ b j + tα σˆ 2C jj  =1 − α .
,n −k ,n −k
 2 2 
So the confidence interval is
 
 b j − tα ,n − k σˆ C jj , b j + tα ,n − k σˆ C jj  .
2 2

 2 2 
Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur
33
Simultaneous confidence intervals on regression coefficients:
A set of confidence intervals that are true simultaneously with probability (1 − α ) are called simultaneous or
joint confidence intervals.
It is relatively easy to define a joint confidence region for β in multiple regression model.
Since
(b − β ) ' X ' X (b − β )
~ Fk ,n − k
k MS r e s
 (b − β ) ' X ' X (b − β ) 
⇒ P 1−α.
≤ Fα (k , n − k )  =
 k MS r e s 
So a 100 (1 − α )% joint confidence region for all of the parameters in β is
(b − β ) ' X ' X (b − β )
≤ Fα (k , n − k )
k MS r e s
which describes an elliptically shaped region.

Coefficient of determination ( R 2 ) and adjusted R 2

Let R be the multiple correlation coefficient between y and X 1 , X 2 ,..., X k . Then square of multiple

correlation coefficient ( R 2 ) is called as coefficient of determination. The value of R 2 commonly describes

that how well the sample regression line fits to the observed data. This is also treated as a measure of
goodness of fit of the model.

Assuming that the intercept term is present in the model as

yi = β1 + β 2 X i 2 + β3 X i 3 + ... + β k X ik + ui , i = 1, 2,..., n
then
e 'e
R2 = 1 − n

∑ ( y − y)
i =1
i
2

SS res
= 1−
SST
SS r e g
=
SST
where

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

34
 SS r e s : sum of squares due to residuals,

SST : total sum of squares

SS r e g : sum of squares due to regression.

R 2 measure the explanatory power of the model which in turn reflects the goodness of fit of the model. It
reflects the model adequacy in the sense that how much is the explanatory power of explanatory variable.
Since
y '  I − X ( X ' X ) −1 X ' y =
e 'e = y ' Hy,
n n

=i 1 =i 1
∑ ( yi − y )2 = ∑y 2
i − ny 2 ,

1 n 1
=
where y = ∑
n i =1
yi
n
' y =
with  (1,1,...,1=
) ', y ( y1 , y2 ,..., yn ) '

Thus
n
 1 
∑ ( y − y)
i =1
i = y ' y − n  2  ' yy '  
2

n 
1
= y ' y − y '  ' y
n
−1
= y ' y − y '  ( '  )  ' y
= y '  I − ( ' ) −1  ' y
= y ' Ay

where A= I − ( ' ) −1  ' .

y ' Hy
So R2 = 1 − .
y ' Ay

The limits of R 2 are 0 and 1, i.e.,

0 ≤ R 2 ≤ 1.
R 2 = 0 indicates the poorest fit of the model.
R 2 = 1 indicates the best fit of the model
R 2 = 0.95 indicates that 95% of the variation in y is explained by R 2 . In simple words, the model is 95%
good.

Similarly any other value of R 2 between 0 and 1 indicates the adequacy of fitted model.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

35
Adjusted R 2
If more explanatory variables are added to the model, then R 2 increases. In case the variables are
irrelevant, then R 2 will still increase and gives an overly optimistic picture.

With a purpose of correction in overly optimistic picture, adjusted R 2 , denoted as R 2 or adj R 2 is used
which is defined as
SS r e s / (n − k )
R2 = 1−
SST / (n − 1)
 n −1 
=
1−   (1 − R ).
2

 n − k 
We will see later that (n − k ) and (n − 1) are the degrees of freedom associated with the distributions of SS res

SS r e s SST
and SST . Moreover , the quantities and are based on the unbiased estimators of respective
n−k n −1
variances of e and y is the context of analysis of variance.

The adjusted R 2 will decline if the addition if an extra variable produces too small a reduction in (1 − R 2 )

 n −1 
to compensate for the increase is  .
n−k 

Another limitation of adjusted R 2 is that it can be negative also. For example if =

k 3,= =
n 10, R 2
0.16,
then
9
R 2 =−
1 × 0.97 =−0.25 < 0
7
which has no interpretation.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

36
Limitations
1. If constant term is absent in the model, then R 2 can not be defined. In such cases, R 2 can be
negative. Some ad-hoc measures based on R 2 for regression line through origin have been proposed
in the literature.

Reason that why R 2 is valid only in linear models with intercept term:
y X β + ε , the ordinary least squares estimator of β is b = ( X ' X ) −1 X ' y . Consider the
In the model =
fitted model as
y = Xb + ( y − Xb)
= Xb + e
where e is the residual. Note that
y − ly = Xb + e − ly
= yˆ + e − ly
where ŷ = Xb is the fitted value and l = (1,1,...,1) ' is a n ×1 vector of elements unity. The total sum of
n
squares =
TSS ∑ ( y − y)
i =1
i
2
is then obtained as

TSS = ( y − ly ) '( y − ly ) =[( yˆ − ly ) + e]'[( yˆ − ly ) + e]

= ( yˆ − ly ) '( yˆ − ly ) + e ' e + 2( yˆ − ly ) ' e
↓ ↓ ↓
= SS reg + SS res + 2( Xb − ly ) ' e =
(because yˆ Xb)
= SS reg + SS res − 2 yl ' e (because X ' e =
0).

The Fisher Cochran theorem requires =

TSS SS reg + SS res to hold true in the context of analysis of
2
variance and further to define the R . In order that =
TSS SS reg + SS res holds true, we need that

l ' e should be zero, i.e. l ' e =l '( y − yˆ ) =

0 which is possible only when there is an intercept term in the
model. We show this claim as follows:

yi β1 xi + ε i , (i = 1, 2,..., n) where the

First we consider a no intercept simple linear regression model =
n

∑x y i i n n n
parameter β1 is estimated as b = *
1
i =1
n
. Then l ' e = ∑ e=
i ∑ ( yi − yˆi =) ∑ ( y − b x ) ≠0, in general.
i
*
1 i

∑x
i =1
2
i
=i 1 =i 1 =i 1

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

37
 y X β + ε , we find that
Similarly, in a no intercept multiple linear regression model=
yˆ ) l '( X β + ε − Xb) = − l ' X (b − β ) + l ' ε ≠ 0 , in general.
l ' e =l '( y −=

Next we consider a simple linear regression model with intercept term yi =β 0 + β1 xi + ε i , (i = 1, 2,..., n)

sxy
where the parameters β 0 and β1 are estimated as b0= y − b1 x and b1 = respectively, where
sxx
n n
1 n 1 n
sxy = ∑ ( xi − x )( yi − y ), =
i =1
sxx ∑ ( xi − x )2 , x =
i =1
∑ i n∑
n i =1
x y =
i =1
yi . We find that

n n

∑ ei
l ' e ==
=i 1 =i 1
∑ ( y − yˆ )
i i

n
= ∑(y −b
i =1
i 0 − b1 xi )
n
= ∑(y − y + b x −b x )
i =1
i 1 1 i

n
= ∑ [( y − y ) − b ( x − x )]
i =1
i 1 i

n n

=i 1 =i 1
= ∑ ( yi − y ) − b1 ∑ ( xi − x )
= 0.

In a multiple linear regression model with an intercept term y =β 0l + X β + ε where the parameters β 0

and β are estimated as βˆ0= y − bx and b = ( X ' X ) −1 X ' y , respectively. We find that

l ' e =l '( y − yˆ )
=l '( y − βˆ0 − Xb)
=l '( y − y + Xb − Xb) ,
=l '( y − y ) + l '( X − X )b
=0.
Thus we conclude that for the Fisher Cochran to hold true in the sense that the total sum of squares can
be divided into two orthogonal components, viz., sum of squares due to regression and sum of squares
due to errors, it is necessary that l ' e =l '( y − yˆ ) =
0 holds and which is possible only when the intercept
term is present in the model.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

38
 2. R 2 is sensitive to extreme values, so R 2 lacks robustness.

3. R 2 always increases with an increase in the number of explanatory variables in the model. The main
drawback of this property is that even when the irrelevant explanatory variables are added in the
model, R 2 still increases. This indicates that the model is getting better which is not really correct.
4. Consider a situation where we have following two models:
yi = β1 + β 2 X i 2 + ... + β k X ik + ui , i = 1, 2,.., n
log yi = γ 1 + γ 2 X i 2 + ... + γ k X ik + vi
The question is now which model is better?
For the first model,
n

∑ ( y − yˆ )
i i
2

R12 = 1 − i =1
n

∑ ( y − y)
i =1
i
2

and for the second model, an option is to define R 2 as

n

∑ (log y − log yˆ )
i i
2

R22 = 1 − i =1
n
.
∑ (log y − log y )
i =1
i
2

As such R12 and R22 are not comparable. If still, the two models are needed to be compared, a better

proposition to define R 2 can be as follows:

n

∑ ( y − anti log yˆ )
i
*
i
R32 = 1 − i =1
n

∑ ( y − y)
i =1
i
2

where  y . Now
yi* = log R12 and R32 on comparison may give an idea about the adequacy of the two
i

models.

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

39
Relationship of analysis of variance test and coefficient of determination
Assuming the β1 to be an intercept term, then for H 0 : β 2= β3= ...= β k= 0, the F − statistic in analysis of
variance test is
MS r e g
F=
MS res
(n − k ) SS r e g
=
(k − 1) SS r e s
 n − k  SS r e g
= 
 k − 1  SST − SSr e g
SS r e g
 n − k  SST
= 
 k − 1  1 − SS r e g
SST
n−k  R
2
= 
 k −1  1− R
2

where R 2 is the coefficient of determination. So F and R 2 are closely related. When R 2 = 0, then F = 0.

In limit, when R 2 = 1, F = ∞ . So both F and R 2 vary directly. Larger R 2 implies greater F value. That is
why the F test under analysis of variance is termed as the measure of overall significance of estimated
regression. It is also a test of significance of R 2 . If F is highly significant, it implies that we can reject
H 0 , i.e. y is linearly related to X ' s.

Prediction of values of study variable

The prediction in multiple regression model has two aspects
1. Prediction of average value of study variable or mean response.
2. Prediction of actual value of study variable.

1. Prediction of average value of y

We need to predict E ( y ) at a given x0 = ( x01 , x02 ,..., x0 k ) '.
The predictor as a point estimate is
=
p x=
'
0b x0' ( X ' X ) −1 X ' y
E ( p ) = x0' β .
So p is an unbiased predictor for E ( y ) .

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

40
Its variance is
E [ p − E ( y )] '[ p − E ( y )]
Var ( p ) =
=σ 2 x0' ( X ' X ) −1 x0
Then
E( = 0β
yˆ 0 ) x=
'
E ( y / x0 )
Var ( yˆ 0 ) = σ 2 x0' ( X ' X ) −1 x0

The confidence interval on the mean response at a particular point, such as x01 , x02 ,..., x0 k can be found as
follows:
Define x0 = ( x01 , x02 ,..., x0 k ) '. The fitted value at x0 is yˆ 0 = x0' b.
Then
 yˆ 0 − E ( y / x0 ) 
P  −tα ≤ ≤ tα  = 1−α
 2 ,n − k σˆ 2 x0' ( X ' X ) −1 x0 2
,n −k

 
P  yˆ 0 − tα ≤ σˆ 2 x0' ( X ' X ) −1 x0 ≤ E ( y / x0 ) ≤ yˆ 0 + tα ≤ σˆ 2 x0' ( X ' X ) −1 x0  =−
1 α.
,n −k ,n −k
 2 2 
The 100 (1 − α )% confidence interval on the mean response at the point x01 , x02 ,..., x0 k , i.e., E ( y / x0 ) is

 
 yˆ 0 − tα ,n − k σˆ x0 ( X ' X ) x0 , yˆ 0 + tα ,n − k ≤ σˆ x0 ( X ' X ) x0
2 ' −1 2 ' −1
.
 2 2 

2. Prediction of actual value of y

We need to predict y at a given x0 = ( x01 , x02 ,..., x0 k ) '.
The predictor as a point estimate is
p f = x0' b
E ( p f ) = x0' β
So p f is an unbiased for y. It's variance is
Var ( p f ) = E ( ( p f − y )( p f − y ) ' )
= σ 2 1 + x0' ( X ' X ) −1 x0  .

The 100 (1 − α )% confidence interval for this future observation is

 
 p f − tα ,n − k σˆ [1 + x0 ( X ' X ) x0 ], p f + tα ,n − k σˆ [1 + x0 ( X ' X ) x0 ]  .
2 ' −1 2 ' −1

 2 2 

Econometrics | Chapter 3 | Multiple Linear Regression Model | Shalabh, IIT Kanpur

41