Lecture notes on Business Statistics II: simple linear regression and correlation

CHAPTER 9

2. SIMPLE LINEAR REGRESSION AND CORRELATION

Linear regression and correlation is studying and measuring the linear relation ship
among two or more variables. When only two variables are involved, the analysis is
referred to as simple correlation and simple linear regression analysis, and when there are
more than two variables the term multiple regression and partial correlation is used.

Regression Analysis: is a statistical technique that can be used to develop a

mathematical equation showing how variables are related.

Correlation Analysis: deals with the measurement of the closeness of the relation ship
which are described in the regression equation.
We say there is correlation when the two series of items vary together directly or
inversely.

Simple Correlation
Suppose we have two variables and

 When higher values of X are associated with higher values of Y and lower values
of X are associated with lower values of Y, then the correlation is said to be
positive or direct.

Examples:
- Income and expenditure
- Number of hours spent in studying and the score obtained
- Height and weight
- Distance covered and fuel consumed by car.

 When higher values of X are associated with lower values of Y and lower values
of X are associated with higher values of Y, then the correlation is said to be
negative or inverse.

Examples:
- Demand and supply
- Income and the proportion of income spent on food.
The correlation between X and Y may be one of the following
1. Perfect positive (slope=1)
2. Positive (slope between 0 and 1)
3. No correlation (slope=0)
4. Negative (slope between -1 and 0)
5. Perfect negative (slope=-1)
The presence of correlation between two variables may be due to three reasons:

Page 1 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

1. One variable being the cause of the other. The cause is

called “subject” or “independent” variable, while the effect is called
“dependent” variable.
2. Both variables being the result of a common cause. That is,
the correlation that exists between two variables is due to their being related
to some third force.

Example:
Let X1= be ESLCE result
Y1=be rate of surviving in the University
Y2=be the rate of getting a scholar ship.

Both X1&Y1 and X1&Y2 have high positive correlation, likewise

Y1 & Y2 have positive correlation but they are not directly related, but they are
related to each other via X1.

3. Chance:

The correlation that arises by chance is called spurious correlation.

Examples:
 Price of teff in Addis Ababa and grade of students in USA.
 Weight of individuals in Ethiopia and income of individuals in Kenya.

Therefore, while interpreting correlation coefficient, it is necessary to see if there is any

likelihood of any relation ship existing between variables under study.
The correlation coefficient between X and Y denoted by is given by

Remark:

Always this lies between -1 and 1 inclusively and it is also symmetric.

Interpretation of

Page 2 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

1. Perfect positive linear relationship (

2. Some Positive linear relationship ( between 0 and 1)
3. No linear relationship (
4. Some Negative linear relationship ( between -1 and
0)
5. Perfect negative linear relationship (

Examples:

1. Calculate the simple correlation between mid semester and final exam scores of 10
students (both out of 50)

Student Mid Sem.Exam Final Sem.Exam

(X) (Y)
1 31 31
2 23 29
3 41 34
4 32 35
5 29 25
6 33 35
7 28 33
8 31 42
9 31 31
10 33 34
Solution:

This means mid semester exam and final exam scores have a slightly positive correlation.

2. The following data were collected from a certain household on the monthly income
(X) and consumption (Y) for the past 10 months. Compute the simple correlation
coefficient.( Exercise)

X: 650 654 720 456 536 853 735 650 536 666
Y: 450 523 235 398 500 632 500 635 450 360

Page 3 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

The above formula and procedure is only applicable on quantitative data, but when we
have qualitative data like efficiency, honesty, intelligence, etc
We calculate what is called Spearman’s rank correlation coefficient as follows:
Steps
i. Rank the different items in X and Y.
ii. Find the difference of the ranks in a pair , denote them by Di
iii. Use the following formula

Example:
Aster and Almaz were asked to rank 7 different types of lipsticks, see if there is
correlation between the tests of the ladies.
Lipsticks A B C D E F G
Aster 2 1 4 3 5 7 6
Almaz 1 3 2 4 5 6 7
Solution:
X Y R1-R2 D2
(R1) (R2) (D)
2 1 1 1
1 3 -2 4
4 2 2 4
3 4 -1 1
5 5 0 0
7 6 1 1
6 7 -1 1
Total 12

Yes, there is positive correlation.

Simple Linear Regression

- Simple linear regression refers to the linear relation ship between two variables
- We usually denote the dependent variable by Y and the independent variable by X.
- A simple regression line is the line fitted to the points plotted in the scatter diagram,
which would describe the average relation ship between the two variables. Therefore,
to see the type of relation ship, it is advisable to prepare scatter plot before fitting the
model.
- The linear model is:

Page 4 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

- To estimate the parameters ( ) we have several methods:

 The free hand method
 The semi-average method
 The least square method
 The maximum likelihood method
 The method of moments
 Bayesian estimation technique.

- The above model is estimated by:

Where is a constant which gives the value of Y when X=0 .It is called the Y-
intercept. is a constant indicating the slope of the regression line, and it gives a
measure of the change in Y for a unit change in X. It is also regression coefficient of Y
on X.

- and are found by minimizing

And this method is known as OLS (ordinary least square)

- Minimizing gives

Example 1: The following data shows the score of 12 students for Accounting and Statistics
Examinations.

Page 5 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

a) Calculate a simple correlation coefficient

b) Fit a regression line of Statistics on Accounting using least square estimates.
c) Predict the score of Statistics if the score of accounting is 85.

Accounting Statistics
X Y
1 74.00 81.00
2 93.00 86.00
3 55.00 67.00
4 41.00 35.00
5 23.00 30.00
6 92.00 100.00
7 64.00 55.00
8 40.00 52.00
9 71.00 76.00
10 33.00 24.00
11 30.00 48.00
12 71.00 87.00

Scatter Diagram of raw data.

Accounting Statistics
X2 Y2 XY
X Y

Page 6 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

1 74.00 81.00 5476.00 6561.00 5994.00

2 93.00 86.00 8649.00 7396.00 7998.00
3 55.00 67.00 3025.00 4489.00 3685.00
4 41.00 35.00 1681.00 1225.00 1435.00
5 23.00 30.00 529.00 900.00 690.00
6 92.00 100.00 8464.00 10000.00 9200.00
7 64.00 55.00 4096.00 3025.00 3520.00
8 40.00 52.00 1600.00 2704.00 2080.00
9 71.00 76.00 5041.00 5776.00 5396.00
10 33.00 24.00 1089.00 576.00 792.00
11 30.00 48.00 900.00 2304.00 1440.00
12 71.00 87.00 5041.00 7569.00 6177.00
Total 687.00 741.00 45591.00 52525.00 48407.00
Mean 57.25 61.75

a)

The Coefficient of Correlation (r) has a value of 0.92. This indicates that the two
variables are positively correlated (Y increases as X increases).

b)

Using OLS:

Page 7 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

Scatter Diagram and Regression Line

c) Insert X=85 in the estimated regression line.

Example 2:

A car rental agency is interested in studying the relationship between the distance
driven in kilometer (Y) and the maintenance cost for their cars (X in birr). The
following summarized information is given based on samples of size 5. (Exercise)

Page 8 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

, ,

a) Find the least squares regression equation of Y on X

b) Compute the correlation coefficient and interpret it.
c) Estimate the maintenance cost of a car which has been driven for 6 km

- To know how far the regression equation has been able to explain the variation in Y we
use a measure called coefficient of determination ( )

- gives the proportion of the variation in Y explained by the regression of Y on X.

- gives the unexplained proportion and is called coefficient of indetermination.

Example: For the above problem (example 1):

84.53% of the variation in Y is explained and only 15.47% remains

unexplained and it will be accounted by the random term.

o Covariance of X and Y measures the co-variability of X and Y together. It is

denoted by and given by

o Next we will see the relation ship between the coefficients.

i.

ii.

Page 9 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

o When we fit the regression of X on Y , we interchange X and Y in all formulas, i.e.

we fit

Here X is dependent and Y is independent.

Choice of Dependent and Independent variable

- In correlation analysis there is no need of identifying the dependent and independent

variable, because is symmetric. But in regression analysis
If bYX is the regression coefficient of Y on X
bXY is the regression coefficient of X on Y

Then

- Moreover, are completely different numerically as well as

conceptually.

- Let us consider three cases concerning these coefficients.

1. If the correlation is perfect positive, i.e. then the b values reciprocals of each
other.
2. , then irrespective of the value of the b values are equal, i.e.
( but this is unlikely case)
3. The most important case is when , here the b values are not equal
or reciprocals to each other, but rather the two lines differ , intersecting at the
common point ( )

Page 10 of 11
Lecture notes on Business Statistics II: simple linear regression and correlation

 Thus to determine if a regression equation is X on Y or Y on X ,

we have to use the formula
 If
 If
Example: The regression line between height (X) in inches and weight (Y) in lbs of
male students are:

Determine which is regression of Y on X and X on Y

Solution
We will assume one of the equation as regression of X on Y and the other as Y on X
and calculate

Then write these in the standard form.

,
This is impossible (contradiction). Hence our assumption is not correct. Thus

To verify:

Page 11 of 11