Regression Analysis

Linear regression is one of the most widely used tools in statistics.

Suppose we were jobless college students interested in finding out
how big (or small) our salaries would be 20 years from now. There’s
no way to pin down this number for sure, but we know that there are
many factors that contribute to how much money a college graduate
will make. For example, a naive observation (but a good starting
point) is that students with higher GPAs earn more money 20 years
from now. In this case, we assume that there is some true distribution
that governs the behavior of the random variables
.
X = GPA
.
Y = Salary 20 years from now

where X and Y are not independent. In this case, we call X a predictor

of Y. Another way that people refer to X and Y are as independent
and dependent variables (nothing to do with probabilistic indepen-
dence), since Y depends on X. In the following sections, we set up a
linear model to describe the relationship between Y and X, which we
can then use to predict our own future salary, based on some sample
data.

Ordinary Least Squares

The Linear Model

Since X and Y seem to have some relationship, it would be reason-
able to assume that given some value of X, we have a better idea
about what Y is. Intuitively, we would expect students with higher
GPAs to have a larger future salary, so we could model the relation-
ship between X and Y using a line. That is, for some real numbers w0
and w1 ,

Y = w0 + w1 X

This is our familiar y = mx + b relationship from high school algebra,

but with different names for m and b.
56 seeing theory

Note that this is an extremely simple model that is likely to miss

most of the nuances in predicting someone’s salary 20 years from
now. There are in fact many more predictors than someone’s GPA
that affect their future salary. Also notice that we can express the
above relationship using the following vector form.
.
Y = X · w = (1, X ) · (w0 , w1 )

where “·” represents the dot product. This form is why the method is
called linear regression.

Exercise 0.0.5. Verify the function f :2 → defined by

f (w) = X · w

is linear in w.

Solution. Remember that the term linear was used to describe the
“Expectation” operator. The two conditions we need to check are

(a) For any vectors w, v ∈2 , we have

f (u + v) = f (w) + f (v)

(b) For any vector w ∈2 and constant c ∈,

f (cw) = c f (w)

To show (a), we know that w and v are vectors of the form

.
w = ( w0 , w1 )
.
v = ( v0 , v1 )

so that

f (w + v) = f ((w0 , w1 ) + (v0 , v1 ))
= f ((w0 + v0 , w1 + v1 ))
= X · ( w0 + v 0 , w1 + v 1 )
= (1, X ) · (w0 + v0 , w1 + v1 ) (Definition of X)
= ( w0 + v 0 ) + X ( w1 + v 1 ) (Definition of dot product)
= (w0 + Xw1 ) + (v0 + Xv1 ) (Rearranging)
= X·w+X·v
= f (w) + f (v)
 regression analysis 57

For (b), observe that if w ∈2 and c ∈,

f (cw) = X · (cw0 , cw1 )

= (1, X ) · (cw0 , cw1 )
= cw0 + cw1 X
= c ( w0 + w1 X )
= cX · w
= c f (w)

This completes the proof.

The observation that f is linear in w as opposed to linear in X is

an extremely important distinction. Take a moment and let it sink in.
This means that we can transform X in crazy nonlinear ways while
maintaining the linearity of this problem. For example, the proof
above implies that we could replace X with log(X) or sin(X) and we
still have a linear relationship between Y and w.
The above example not realistic in the sense that its extremely un-
likely that if we sampled n college graduates and their actual salaries
20 years after college, all their GPAs fall on a perfect line when plot-
ted against their salaries. That is, if we took n sample points, written

Sample = {( X1 , Y1 ), ( X2 , Y2 ), . . . , ( Xn , Yn )}

and plotted these points in the plane with “GPA” on the x-axis and
“Salary” on the y-axis, the points would almost surely not fall on a
perfect line. As a result, we introduce an error term e, so that

Y = X·w+e (5)

All of this hasn’t yet told us how to predict our salaries 20 years
from now using only our GPA. The subject of the following section
gives a method for determining the best choice for w0 and w1 given
some sample data. Using these values, we could plug in the vector
(1, our GPA) for X in equation (2) and find a corresponding predicted
salary Y (within some error e).

Method of Least Squares

Our current model for X and Y is the relationship

Y = X·w+e

where e is some error term. Suppose we go out and ask a bunch of

50 year olds for their college GPAs and their salaries 20 years out of
college. We can pair these quantities and record this sample data as

Data = {( x1 , y1 ), ( x2 , y2 ), . . . , ( xn , yn )}
58 seeing theory

Remember that we assume these samples come from the relationship

yi = (1, xi ) · (w0 , w1 ) + ei

and we are trying to find w0 and w1 to best fit the data. What do
we mean by “best fit”? The notion we use is to find w0 and w1 that
minimize the sum of squared errors ∑in=1 ei2 . Rearranging the above
equation for ei , we can rewrite this sum of squared errors as
n n
.
E (w) = ∑ ei2 = ∑ (yi − xi · w)2
i =1 i =1

where the vector xi is shorthand for (1, xi ). As we can see above, the
error E is a function of w. In order to minimize the squared error,
we minimize the function E with respect to w. E is a function of
both w0 and w1 . In order to minimize E with respect to these values,
we need to take partial derivatives with respect to w0 and w1 . This
derivation can be tricky in keeping track of all the indices so the
details are omitted. If we differentiate E with respect to w0 and w1 ,
we eventually find that minimizing w can be expressed in matrix
form as
    −1  
1 x1 y1
" # " # " #
1 x y2 
  
w0  1 1 ... 1  2  1 1 ... 1 
=  x x . . . x .
. .  .
w1  1 2 n . .. 
 x1 x2 . . . x n  .. 


1 xn yn
This can be written in the following concise form,

[box = ] align∗wT = (DT D)−1 DT y

where D is the matrix made by stacking the sample vectors xi ,
   
x1 1 x1
x2   1 x 2 
   
. 
D =  .  = .
   .. 
 ..   .. . 


xn 1 xn
and y is the column vector made by stacking the observations yi ,
 
y1
.  y2 
 
y=  .. 

.
yn
A sketch of the derivation using matrices is given in the following
section for those who cringed at the sentence “This derivation can
be tricky in keeping track of all the indices so the details are omit-
ted.” Some familiarity with linear algebra will also be helpful going
through the following derivation.
 regression analysis 59

Linear Algebra Derivation

We can write the error function E as the squared norm of the matrix
difference e = y − DwT .

E(w) = ky − DwT k2 = kDwT − yk2

Differentiating with respect to w, the two comes down from the

exponent by the power rule, and we multiply by DT to account for
the chain rule. We get

∇ E = 2DT (DwT − y)

We set ∇ E = 0 (we use a bold “0” since it is actually a vector of

zeros) so that

2DT (DwT − y) = 0

Dividing by 2 on both sides and distributing the DT across the differ-

ence gives

DT DwT − DT y = 0

Adding DT y to both sides gives

DT DwT = DT y

Multiplying on the left by the inverse of the matrix DT D on both

sides of the above equation finally yields the famous linear regression
formula,

w T = (D T D ) −1 D T y

Now, assuming salaries are related to college GPAs according to the

relation

Y = w0 + w1 X + e,

we can plug in our GPA for X, and our optimal w0 and w1 to find the
corresponding predicted salary Y, give or take some error e. Note
that since we chose w0 and w1 to minimize the errors, it is likely that
the corresponding error for our GPA and predicted salary is small
(we assume that our (GPA, Salary) pair come from the same “true”
distribution as our samples).

Generalization
Our above example is a simplistic one, relying on the very naive as-
sumption that salary is determined solely by college GPA. In fact
60 seeing theory

there are many factors which influence someones salary. For ex-
ample, earnings could also be related to the salaries of the person’s
parents, as students with more wealthy parents are likely to have
more opportunities than those who come from a less wealthy back-
ground. In this case, there are more predictors than just GPA. We
could extend the relationship to

Y = w 0 + w 1 X1 + w 2 X2 + w 3 X3 + e

where X1 , X2 , and X3 are the GPA, Parent 1 salary, and Parent 2

salary respectively.
By now it is clear that we can extend this approach to accomo-
date an arbitrary number of predictors X1 , . . . , Xd by modifying the
relationship so that

Y = w 0 + w 1 X1 + w 2 X2 + · · · + w d X d + e

or more concisely,

Y = X·w+e

where the vectors X, w ∈d+1 are the extensions

.
X = (1, X1 , X2 , . . . , Xd )
.
w = ( w0 , w1 , w2 , . . . , w d )

the parameters wi can be thought of as “weights” since the larger

any particular weight is, the more influence its attached predictor
has in the above equation. Recall that in Exercise 6.1, we verified the
function f (w) = X · w was linear in the vector w ∈2 . In fact, when we
extend w to be a vector in d+1 , the function f (w) = X · w is still linear
in w.
The linear regression formula still holds, i.e. that the optimal
weights are given by

w T = (D T D ) −1 D T y

where the matrix D is still constructed by stacking the observed

samples,
   (1) (2) (d)

x1 1 x1 x1 . . . x1
   (1) (2) (d) 
.  x2   1 x2 x2 . . . x2 
D=  ..  = 
 
..
 
 .  .  

xn (1) (2) (d)
1 xn xn . . . xn

where the ith sample is written

. (1) (2) (d)
xi = (1, xi , xi , . . . , xi )
 regression analysis 61

Correlation

Throughout the past chapters, we often made the assumption that

two random variables are independent in various exercises and meth-
ods. In reality, most random variables are not actually independent.
In this section we give some measures to quantify how “related” a
collection of random variables are.
The example in the Linear Regression chapter began with the ob-
servation that GPAs are positively correlated with future salaries.
That is, we assumed that as college GPA increased, future salary also
increased. Qualitatively, this was enough to motivate the problem of
regression. However, there were other predictors that contributed to
the future salary, some of which were also positively correlated to
the projected salary. The fact that some variables contributed “more
positively” than others was manifested in the size of the weights that
were attached to the variables in the equation Y = X · w + e. If one
Xi were more predictive of Y than another, then its corresponding
weight was larger. In the following section we examine the covari-
ance of two random variables, which is another attempt to quantify
the relationship between random variables.

Covariance
Suppose we have two random variables X and Y, not necessarily
independent, and we want to quantify their relationship with a num-
ber. This number should satisfy two basic requirements.

(a) The number should be positive when X and Y increase/decrease

together.

(b) It should be negative when one of X or Y decreases while the

other increases.

Consider the following random variable.

( X − EX )(Y − EY )

Consider the possible realizations of the random variables X = x and

Y = y. The collection of these pairs is the sample space Ω. We can
think of the outcomes of sampling an X and a Y as pairs ( x, y) ∈ Ω.
Suppose the probability distribution governing X and Y on Ω assigns
most of the probability mass on the pairs ( x, y) such that x > EX
and y > EY. In this case, the random variable ( X − EX )(Y − EY )
is likely to be positive most of the time. Similarly, if more mass were
placed on pairs ( x, y) such that x < EX and y < EY, the product
( X − EX )(Y − EY ) would be a negative number times a negative
62 seeing theory

number, which means it would still be positive most of the time.

Hence the product ( X − EX )(Y − EY ) being positive is indicative of
X and Y being mutually more positive or mutually more negative.
By similar reasoning, the product ( X − EX )(Y − EY ) is more often
negative if the distribution assigns more mass to pairs ( x, y) that have
x < EX and y > EY, or that satisfy x > EX and y < EY. In either
case, the product ( X − EX )(Y − EY ) will be a product of a positive
and negative number, which is negative.
We are almost done. Remember at the beginning of this discus-
sion we were searching for a number to summarize a relationship
between X and Y that satisfied the requirements (a) and (b). But
( X − EX )(Y − EY ) is a random variable, (that is, a function map-
ping Ω to ) not a number. To get a number, we take the expectation.
Finally we arrive at the definition of covariance.
Definition 0.0.19. The covariance of two random variables X and Y,
written Cov( X, Y ), is defined

Cov( X, Y ) = E[( X − EX )(Y − EY )]

This definition may look similar to the definition for variance of a

random variable X, except we replace one of the terms in the product
with the difference Y − EY. Similar to Proposition 2.11 (c), there is
another useful form of the covariance.
Proposition 0.0.3. Let X and Y be two random variables with means EX
and EY respectively. Then

Cov( X, Y ) = E[ XY ] − E[ X ] E[Y ]

Proof. By the definition of covariance, we can foil the product inside

the expectation to get

Cov( X, Y ) = E[ XY − XEY − YEX + EXEY ]

= E[ XY ] − E[ XEY ] − E[YEX ] + E[ EXEY ] (linearity of E)
= E[ XY ] − EYEX − EXEY + EXEY (linearity of E)
= E[ XY ] − EXEY

The Correlation Coefficient

The covariance quantity we just defined satisfies conditions (a) and
(b), but can become arbitrarily large depending on the distribution
of X and Y. Thus comparing covariances between different pairs of
random variables can be tricky. To combat this, we normalize the
quantity to be between −1 and 1. The normalized quantity is called
the correlation, defined below.
 regression analysis 63

Definition 0.0.20. The correlation coefficient between two random

variables X and Y with standard deviations σx and σy , is denoted ρ and is
defined

Cov( X, Y )
ρ xy =
σx σy

Exercise 0.0.6. Verify that for given random variables X and Y, the correla-
tion ρ xy lies between −1 and 1.

Heuristic. The rigorous proof for this fact requires us to view X and
Y as elements in an infinite-dimensional normed vector space and ap-
ply the Cauchy Schwartz inequality to the quantity E[( X − EX )(Y −
EY )]. Since we haven’t mentioned any of these terms, we instead try
to understand the result using a less fancy heuristic argument.
Given a random variable X, the first question we ask is,

What is the random variable most positively correlated with X?

The random variable that correlates most positively with X should

increase exactly with X and decrease exactly with X. The only random
variable that accomplishes this feat is X itself. This implies that the
correlation coefficient between X and any random variable Y is less
than that between X and itself. That is,

Cov( X, X ) Var( X )
ρ xy ≤ ρ xx = = =1
σx σx Var( X )

By now you’ve probably guessed the second question we need to ask.

What is the random variable least positively correlated with X?

In other words, we are looking for a random variable with which

the correlation between X and this random variable is the most neg-
ative it can be. This random variable should increase exactly as X
decreases, and it should also decrease exactly as X increases. The
candidate that comes to mind is − X. This would imply that the cor-
relation coefficient between X and any random variable Y is greater
than that between X and − X.
This implies that

Cov( X, − X )
ρ xy ≥ ρ x,− x =
σx σ− x

By Proposition 6.3, the expression on the right becomes

E[ X (− X )] − E[ X ] E[− X ] −( E[ X 2 ] − ( EX )2 ) −Var( X )
= p p = = = −1
Var( X ) Var(− X ) Var ( X ) Var( X )

Hence, we conclude that −1 ≤ ρ xy ≤ 1.

64 seeing theory

Interpretation of Correlation
The correlation coefficient between two random variables X and
Y can be understood by plotting samples of X and Y in the plane.
Suppose we sample from the distribution on X and Y and get

Sample = {( X1 , Y1 ), . . . , ( Xn , Yn )}

There are three possibilities.

Case 1: ρ xy > 0. We said that this corresponds to X and Y increas-
ing mutually or decreasing mutually. If this is the case, then if we
took n to be huge (taking many samples) and plotted the observa-
tions, the best fit line would have a positive slope. In the extreme case
if ρ xy = 1, the samples ( Xi , Yi ) would all fall perfectly on a line with
slope 1.
Case 2: ρ xy = 0. This corresponds to X and Y having no observable
relationship. However, this does not necessarily mean that X and Y
have no relationship whatsoever. It just means that the measure we
are using the quantify their relative spread (the correlation) doesn’t
capture the underlying relationship. We’ll see an example of this
later. In terms of the plot, the samples ( Xi , Yi ) would look scattered
on the 2 plane with no apparent pattern.
Case 3: ρ xy < 0. We said that this case corresponds to one of X
or Y decreasing while the other increases. If this were the case, then
the best fit line is likely to have a negative slope. In the extreme case
when ρ xy = −1, all samples fall perfectly on a line with slope −1.

Independence vs Zero Correlation

There is a commonly misunderstood distinction between the follow-
ing two statements.

1. “X and Y are independent random variables.”

2. “The correlation coefficient between X and Y is 0.”

The following statement is always true.

Proposition 0.0.4. If X and Y are independent random variables, then

ρ xy = 0.

The converse is not. That is, ρ xy = 0 does not necessarily imply that
X and Y are independent.
In “Case 2” of the previous section, we hinted that even though
ρ xy = 0 corresponded to X and Y having no observable relationship,
there could still be some underlying relationship between the random
variables, i.e. X and Y are still not independent. First let’s prove
Proposition 6.6
 regression analysis 65

Proof. Suppose X and Y are independent. Then functions of X and Y

are independent. In particular, the functions
.
f ( X ) = X − EX
.
g(Y ) = Y − EY

are independent. By the definition of correlation,

Cov( X, Y )
ρ xy =
σx σy
E[( X − EX )(Y − EY )]
=
σx σy
E[ f ( X ) g(Y )]
=
σx σy
E[ f ( X )] E[ g(Y )]
= (independence of f ( X ) and g(Y ))
σx σy
0·0
= ( E[ f ( X )] = E( X − EX ) = 0)
σx σy
=0

Hence if X and Y are independent, ρ xy = 0.

Now let’s see an example where the converse does not hold. That
is, an example of two random variables X and Y such that ρ xy = 0,
but X and Y are not independent.

Example 0.0.11. Suppose X is a discrete random variable taking on values

in the set {−1, 0, 1}, each with probability 31 . Now consider the random
variable | X |. These two random variables are clearly not independent, since
once we know the value of X, we know the value of | X |. However, we can
show that X and | X | are uncorrelated. By the definition of correlation and
Proposition 6.3,

E( X · | X |) − EX · E| X |
ρ x,| x| = (6)
σx σ| x|

Let’s compute the numerator. By looking at the distribution of X, we can see

that the product X · | X | can only take on three possible values. If X = 0,
then | X | = 0 so X · | X | = 0. If X = −1, then | X | = 1 and X · | X | = −1.
Finally if X = 1, then | X | = 1 and X · | X | = 1. Each of these cases occur
with probability 13 . Hence,

X · | X | ∼ Uniform{−1, 0, 1}

It follows that the expectation of X · | X | is

1 1 1
E( X · | X |) = · (−1) + · (0) + · (1) = 0.
3 3 3
66 seeing theory

Also by the definition of expectation,

1 1 1
E[ X ] = · (−1) + · (0) + · (1) = 0.
3 3 3
Plugging these values into the numerator in expression (3), we get ρ x,| x| =
0. Thus, the two random variables X and | X | are certainly not always equal,
they are not independent, and yet they have correlation 0. It is important to
keep in mind that zero correlation does not necessarily imply independence.