Simple Regression Model

Regression Analysis
• Regression analysis is used to predict the value of
one variable (the dependent variable) on the basis
of other variables (the independent variables or
explanatory variables).

• Dependent variable: denoted Y

• Independent variables: denoted X2, X3, …, Xk
• If we have only ONE independent variable, the model will
be

• Yi = β1 + β2Xi + ui
• which is referred to as simple linear regression. We
would be interested in estimating β1 and β2 from
the data we collect.
 A HYPOTHETICAL EXAMPLE

• Regression analysis is largely concerned with estimating

and/or predicting the (population) mean value of the
dependent variable on the basis of the known or fixed
values of the explanatory variable(s).
• Look at table 2.1 which refers to a total population of 60
families and their income (X) and consumption
expenditure (Y). The 60 families are divided into 10
income groups.
• There is considerable variation in consumption
expenditure in each income group. But the general picture
that one gets is that, despite the variability of consumption
expenditure within each income bracket, on the average,
consumption expenditure increases as income increases.
 The population regression line

• The dark circled points in Figure 2.1 show the

conditional mean values of Y against the
various X values. If we join these conditional
mean values, we obtain what is known as the
population regression line (PRL), or more
generally, the population regression curve.
More simply, it is the regression of Y on X. The
adjective “population” comes from the fact that
we are dealing in this example with the entire
population of 60 families. Of course, in reality a
population may have many families.
 • The population regression line
• We can express the deviation of an individual Yi around its
expected value as follows:
• ui = Yi − E(Y | Xi)
• or
• Yi = E(Y | Xi) + ui (2.4.1)
• Technically, ui is known as the stochastic disturbance or
stochastic error term.
• How do we interpret (2.4.1)?
• The expenditure of an individual family, Yi, can be
expressed as the sum of two components:
– (1) E(Y | Xi), the mean consumption of all families with the
same level of income. This component is known as the
systematic, or deterministic, component,
• (2) ui, which is the random, or
nonsystematic, component.

• The disturbance term ui is a proxy for

all those variables that are omitted from
the model but that collectively affect Y.
we don’t introduce them into the model
explicitly.
 The population regression line
• Yi = β1 + β2Xi + ui
•
• Yi = E(Y | Xi) + ui

• E(Y | Xi) = β1 + β2Xi

• Ui = Yi − E(Y | Xi)
 • sample regression function (SRF)
• We can develop the concept of the sample regression function
(SRF) to represent the sample regression line.
• Yˆi = βˆ1 + βˆ2Xi
• where Yˆ is read as “Y-hat’’
• Yˆi = estimator of E(Y | Xi)
• βˆ1 = estimator of β1
• βˆ2 = estimator of β2
• uˆi = Yi − Yˆi (1)
• Yi = Yˆi + uˆi (2)
• Yi = βˆ1 + βˆ2Xi +uˆi (3)

• ˆui denotes the (sample) residual term. Conceptually ˆui is

analogous to ui and can be regarded as an estimate of ui. It is
introduced in the SRF for the same reasons as ui was introduced in
the PRF.
• Method of the estimation of the sample regression line
• Ordinary Least Squares method(OLS) minimizes the sum of
squares of residuals (deviations of individual data points
from the regression line)
σ ˆ𝒖𝒊= (Yi − Yˆi)
σ ˆu2i= (Yi − Yˆi)2
= (Yi − βˆ1 − βˆ2Xi)2
Parameter Estimation Solution
Table
Xi Yi Xi2 Yi2 XiYi
1 1 1 1 1
2 1 4 1 2
3 2 9 4 6
4 2 16 4 8
5 4 25 16 20
15 10 55 26 37
EPI 809/Spring 2008 16
   n 
n
  X i   Yi 
n
  i =1  (15)(10 )

i =1
X Y
i i − 37 −
n
ˆ1 = i =1
= 5 = 0.70

n

2
(15)
2

  i X 55 −
n
  5

i =1
X i
2
−
i =1 n

ˆ0 = Y − ˆ1 X = 2 − (0.70)(3) = −0.10

EPI 809/Spring 2008 17
 THE ASSUMPTIONS UNDERLYING THE METHOD OF
LEAST SQUARES

• Look at the PRF: Yi = β1 + β2Xi + ui . It shows

that Yi depends on both Xi and ui . The
assumptions made about the Xi variable(s) and
the error term are extremely critical to the valid
interpretation of the regression estimates.

• classical linear regression model (CLRM), makes

10 assumptions.
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
• Assumption No (9) the regression model is correctly
specified means that:

• 1- It contains the relative variables according to the

economic theory , for example the demand function
must contains the price of the commodity and the price
of other goods. for example, if we ignored the variable
of the price of the other goods, so the model will not
be correctly specified.
2- It uses the correct functional form, for example
Suppose the “true’’ or correct model in a cost-output study
is as follows:
• Marginal costi = β1 + β2 outputi + β3 output2i + ui (1)
• but we fit the following model:
• Marginal costi = α1 + α2 outputi + ui (2)

• So model (2) uses the wrong functional form so

model (2) is not correctly specified, but model (1)
is correctly specified.
 THE CLASSICAL LINEAR REGRESSION MODEL: THE
ASSUMPTIONS UNDERLYING THE METHOD OF LEAST SQUARES
• If there is perfect linear relationship between
(x2 & x3) in this case we can not estimate the
regression model as a whole.
• so the classical linear regression model assumes that
there is no perfect multicollinearity or no perfect
linear relationship between the explanatory variables
to be able to estimate the regression model or to
estimate the parameters
Note that: this assumption related to the linear
relationship bet. explanatory variables
So if there is non linear relationship we can estimate the
regression model and there is no any problem.
 PROPERTIES OF LEAST-SQUARES ESTIMATORS:
THE GAUSS–MARKOV THEOREM

• To understand this theorem, we need to consider the best

linear unbiasedness property of an estimator. An estimator,
say the OLS estimator βˆ2, is said to be a best linear unbiased
estimator (BLUE) of β2 if the following hold:
• 1. It is linear, that is, a linear function of a random variable,
such as the dependent variable Y in the regression model.
• 2. It is unbiased, that is, its average or expected value, E(βˆ2),
is equal to the true value, β2.
• 3. It has minimum variance in the class of all such linear
unbiased estimators; an unbiased estimator with the least
variance is known as an efficient estimator.
 PROPERTIES OF LEAST-SQUARES ESTIMATORS:
THE GAUSS–MARKOV THEOREM

• To understand this theorem, we need to consider the best

linear unbiasedness property of an estimator. An estimator,
say the OLS estimator βˆ2, is said to be a best linear unbiased
estimator (BLUE) of β2 if the following hold:
• 1. It is linear, that is, a linear function of a random variable,
such as the dependent variable Y in the regression model.
• 2. It is unbiased, that is, its average or expected value, E(βˆ2),
is equal to the true value, β2.
• 3. It has minimum variance in the class of all such linear
unbiased estimators; an unbiased estimator with the least
variance is known as an efficient estimator.