Lecture 6

The Method of Maximum Likelihood for Simple Linear Regression

36-401, Fall 2015, Section B

1 Review
We introduced the method of maximum likelihood for simple linear regression
in the notes for two lectures ago. Let’s review.
We start with the statistical model, which is the Gaussian-noise simple linear
regression model, defined as follows:

1. The distribution of X is arbitrary (and perhaps X is even non-random).

2. If X = x, then Y = β0 + β1 x + , for some constants (“coefficients”,
“parameters”) β0 and β1 , and some random noise variable .
3.  ∼ N (0, σ 2 ), and is independent of X.

4.  is independent across observations.

A consequence of these assumptions is that the response variable Y is indepen-
dent across observations, conditional on the predictor X, i.e., Y1 and Y2 are
independent given X1 and X2 (Exercise 1).
As you’ll recall, this is a special case of the simple linear regression model: the
first two assumptions are the same, but we are now assuming much more about
the noise variable : it’s not just mean zero with constant variance, but it has
a particular distribution (Gaussian), and everything we said was uncorrelated
before we now strengthen to independence1 .
Because of these stronger assumptions, the model tells us the conditional pdf
of Y for each x, p(y|X = x; β0 , β1 , σ 2 ). (This notation separates the random
variables from the parameters.) Given any data set (x1 , y1 ), (x2 , y2 ), . . . (xn , yn ),
we can now write down the probability density, under the model, of seeing that
data:
n n
Y Y 1 (yi −(β0 +β1 xi ))2
p(yi |xi ; β0 , β1 , σ 2 ) = √ e− 2σ 2
2πσ 2
i=1 i=1

In multiplying together the probabilities like this, we are using the independence
of the Yi .
When we see the data, we do not known the true parameters, but any guess
at them, say (b0 , b1 , s2 ), gives us a probability density:
n n
Y Y 1 (yi −(b0 +b1 xi ))2
p(yi |xi ; b0 , b1 , s2 ) = √ e− 2s2

i=1 i=1 2πs2

1 See
the notes for lecture 1 for a reminder, with an explicit example, of how uncorrelated
random variables can nonetheless be strongly statistically dependent.

1
This is the likelihood, a function of the parameter values. It’s just as informa-
tive, and much more convenient, to work with the log-likelihood,
n
Y
2
L(b0 , b1 , s ) = log p(yi |xi ; b0 , b1 , s2 ) (1)
i=1
n
X
= log p(yi |xi ; b0 , b1 , s2 ) (2)
i=1
n
n 1 X
= − log 2π − n log s − 2 (yi − (b0 + b1 xi ))2 (3)
2 2s i=1

In the method of maximum likelihood, we p[ick the parameter values

which maximize the likelihood, or, equivalently, maximize the log-likelihood.
After some calculus (see notes for lecture 5), this gives us the following estima-
tors:
Pn
(x − x)(yi − y) cXY
βb1 = Pn i
i=1
2
= 2 (4)
i=1 (xi − x) sX
βb0 = y − βb1 x (5)
n
1X
b2
σ = (yi − (βb0 + βb1 xi ))2 (6)
n i=1

As you will recall, the estimators for the slope and the intercept exactly match
the least squares estimators. This is a special property of assuming independent
Gaussian noise. Similarly, σc2 is exactly the in-sample mean squared error.

2 Sampling Distributions
We may seem not to have gained much from the Gaussian-noise assumption,
because our point estimates are just the same as they were from least squares.
What makes the Gaussian noise assumption important is that it gives us an
exact conditional distribution for each Yi , and this in turn gives us a distribution
— the sampling distribution — for the estimators. Remember, from the notes
from last time, that we can write βb1 and βb0 in the form “constant plus sum of
noise variables”. For instance,
n
X xi − x
βb1 = β1 + i
i=1
ns2X

Now, in the Gaussian-noise model, the i are all independent Gaussians. There-
fore, βb1 is also Gaussian. Since we worked out its mean and variance last time,
we can just say
βb1 ∼ N (β1 , σ 2 /ns2X )

2
 Again, we saw that the fitted value at an arbitrary point x, m(x),
b is a
constant plus a weighted sum of the :
n  
1X xi − x
m(x)
b = β0 + β1 x + 1 + (x − x) 2 i
n i=1 sX

Once again, because the i are independent Gaussians, a weighted sum of them
is also Gaussian, and we can just say

σ2 (x − x)2
  
m(x) ∼ N β0 + β1 x, 1+
s2X
b
n

Slightly more complicated manipulation of the i makes it possible to show

that
σ2
nb
∼ χ2n−2
σ2
These are all important, because when we come to doing statistical infer-
ence on the parameters — forming confidence intervals, or testing hypotheses
— we need to know these sampling distributions. When we come to making
predictions of new Y ’s, these sampling distributions will let us give confidence
intervals for the expected values, m(x),
b as well as give prediction intervals (of
the form “when X = 5, Y will be between l and u with 95% probability”) or
full distributional forecasts. We will derive these inferential formulas in later
lectures.

2.1 Illustration
To make the idea of these sampling distributions more concrete, I present a small
simulation. Figure 1 provides code which simulates a particular Gaussian-noise
linear model: β0 = 5, β1 = −2, σ 2 = 3, with twenty X’s initially randomly
drawn from an exponential distribution, but thereafter held fixed through all
the simulations. The theory above lets us calculate just what the distribution
of βb1 should be, in repeated simulations, and the distribution of m(−1).
b (By
construction, we have no observations where x = −1; this is an example of using
the model to extrapolate beyond the data.) Figure 2 compares the theoretical
sampling distributions to what we actually get by repeated simulation, i.e., by
repeating the experiment.

3
# Fix x values for all runs of the simulation; draw from an exponential
n <- 20 # So we don't have magic #s floating around
beta.0 <- 5
beta.1 <- -2
sigma.sq <- 3
fixed.x <- rexp(n=n)

# Simulate from the model Y=\beta_0+\beta_1*x+N(0,\sigma^2)

# Inputs: intercept; slope; variance; vector of x; return sample or estimated
# linear model?
# Outputs: data frame with columns x and y OR linear model fit to simulated y
# regressed on x
sim.lin.gauss <- function(intercept=beta.0, slope=beta.1,
noise.variance=sigma.sq, x = fixed.x,
model=FALSE) {
# Make up y by adding Gaussian noise to the linear function
y <- rnorm(length(x), intercept + slope*x, sd=sqrt(noise.variance))
# Do we want to fit a model to this simulation and return that
# model? Or do we want to just return the simulated values?
if (model) { return(lm(y~x)) }
else { return(data.frame(x=x, y=y)) }
}

Figure 1: Function to simulate a Gaussian-noise simple linear regression model,

together with some default parameter values. Since, in this lecture, we’ll always be
estimating a linear model on the simulated values, it makes sense to build that into the
simulator, but I included a switch to

4
 0.8
Density

0.4
0.0

−3.5 −3.0 −2.5 −2.0 −1.5 −1.0 −0.5

^
β1
0.4
Density

0.2
0.0

4 6 8 10
^ (− 1)
m

par(mfrow=c(2,1))
slope.sample <- replicate(1e4, coefficients(sim.lin.gauss(model=TRUE))["x"])
hist(slope.sample,freq=FALSE,breaks=50,xlab=expression(hat(beta)[1]),main="")
curve(dnorm(x,-2,sd=sqrt(3/(n*var(fixed.x)))), add=TRUE, col="blue")
pred.sample <- replicate(1e4, predict(sim.lin.gauss(model=TRUE),
newdata=data.frame(x=-1)))
hist(pred.sample, freq=FALSE, breaks=50, xlab=expression(hat(m)(-1)),main="")
curve(dnorm(x, mean=beta.0+beta.1*(-1),
sd=sqrt((sigma.sq/n)*(1+(-1-mean(fixed.x))^2/var(fixed.x)))),
add=TRUE,col="blue")

Figure 2: Theoretical sampling distributions for βb1 and m(−1)

b (blue curves) versus
the distribution in 104 simulations (black histograms).

5
3 Virtues and Limitations of Maximum Likeli-
hood
The method of maximum likelihood does not always work; there are models
where it gives poor or even pathological estimates. For Gaussian-noise linear
models, however, it actually works very well. Indeed, in more advanced statistics
classes, one proves that for such models, as for many other “regular” statistical
models, maximum likelihood is asymptotically efficient, meaning that its
parameter estimates converge on the truth as quickly as possible2 . This is on
top of having exact sampling distributions for the estimators.
Of course, all these wonderful abilities come at a cost, which is the Gaussian
noise assumption. If that is wrong, then so are the sampling distributions I
gave above, and so are the inferential calculations which rely on those sampling
distributions. Before we begin to do those inferences on any particular data
set, and especially before we begin to make grand claims about the world on the
basis of those inferences, we should really check all those modeling assumptions.
That, however, brings us into the topics for next week.

Exercises
To think through or to practice on, not to hand in.

1. Let Y1 , Y2 , . . . Yn be generated from the Gaussian-noise simple linear re-

gression model, with the corresponding values of the predictor variable
being X1 , . . . Xn . Show that if i 6= j, then Yi and Yj are conditionally in-
dependent given (Xi , Xj ). Hint: If U and V are independent, then f (U )
and g(V ) are also independent, for any functions f and g.
2. In many practical fields (e.g., finance and geology) it is common to en-
counter noise whose distribution has much heavier tails than any Gaussian
could give us. One way to model this is with t distributions. Consider,
therefore, the statistical model where Y = β0 +β1 X +, and /σ ∼ tν , with
 independent of X and independent across observations. That is, rather
than having a Gaussian distribution, the noise follows a t distribution with
ν degrees of freedom (after scaling).
Note: Most students find most parts after (a) quite challenging.

(a) Write down the log-likelihood function. Use an explicit formula for
the density of the t distribution.
2 Veryroughly: writing θ for the true parameter, θb forh the MLE, iand θ̃ for any other consis-
h i
tent estimator, asymptotic efficiency means limn→∞ E nkθb − θk2 ≤ limn→∞ E nkθ̃ − θk .
(This way of formulating it takes it for granted that the MSE of estimation goes to zero like
1/n, but it typically does in parametric problems.) For more precise statements, see, for
instance, ?, ? or ?.

6
 (b) Find the derivatives of this log-likelihood with respect to the four
parameters β0 , β1 , σ (or σ 2 , if more convenient) and ν. Simplify as
much as possible. (It is legitimate to use derivatives of the gamma
function here, since that’s another special function.)
(c) Can you solve for the maximum likelihood estimators of β0 and β1
without knowing σ and ν? If not, why not? If you can, do they
match the least-squares estimators again? If they don’t match, how
do they differ?
(d) Can you solve for the MLE of all four parameters at once? (Again,
you may have to express your answer in terms of the gamma function
and its derivatives.)

3. Refer to the previous problem, and do part (a).

(a) In R, write a function to calculate the log-likelihood, taking as argu-
ments a data frame with columns names y and x, and the vector of
the four model parameters. Hint: use the dt function.
(b) In R, using optim, write a function to find the MLE of this model on
a given data set, from an arbitrary starting vector of guesses at the
parameters. This should call your function from part (a).
(c) In R, write a function which gives an unprincipled but straight-
forward initial estimate of the parameters by (i) calculating the slope
and intercept using least squares, and (ii) fitting a t distribution to
the residuals. Hint: call lm, and fitdistr from the package MASS.
(d) Combine your functions to write a function which takes as its only
argument a data frame containing columns called x and y, and returns
the MLE for the model parameters.
(e) Write another function which will simulte data from the model, tak-
ing as arguments the four parameters and a vector of x’s. It should
return a data frame with appropriate column names.
(f) Run the output of your simulation function through your MLE func-
tion. How well does the MLE recover the parameters? Does it get
better as n grows? As the variance of your x’s increases? How does
it compare to your unprincipled estimator?