Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
89 views

Chapter-3-Linear Models For Regression

1. This document discusses linear models for regression, including linear basis function models using polynomials and Gaussians as basis functions. 2. It also covers maximum likelihood and least squares estimation for linear regression, as well as regularized least squares and its use of Lasso regularization. 3. Bayesian linear regression is introduced, including derivation of the posterior distribution and examples of how the posterior changes as more data is observed.

Uploaded by

longfei zhang
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
89 views

Chapter-3-Linear Models For Regression

1. This document discusses linear models for regression, including linear basis function models using polynomials and Gaussians as basis functions. 2. It also covers maximum likelihood and least squares estimation for linear regression, as well as regularized least squares and its use of Lasso regularization. 3. Bayesian linear regression is introduced, including derivation of the posterior distribution and examples of how the posterior changes as more data is observed.

Uploaded by

longfei zhang
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 61

PATTERN RECOGNITION

AND MACHINE LEARNING


CHAPTER 3: LINEAR MODELS FOR REGRESSION
Linear Basis Function Models (1)

Example: Polynomial Curve Fitting

确定映射关系
Sum-of-Squares Error Function

确定目标函数

确定 w 使目标函数最小化
0th Order Polynomial
1st Order Polynomial
3rd Order Polynomial
9th Order Polynomial

避免过拟合： 1 、增加训练集 2 、增加惩罚项


Linear Basis Function Models (2)
Generally

where Áj(x) are known as basis functions.


Typically, Á0(x) = 1, so that w0 acts as a bias.
In the simplest case, we use linear basis
functions : Ád(x) = xd.
Linear Basis Function Models (3)
Polynomial basis functions:

These are global; a small


change in x affect all basis
functions.
Linear Basis Function Models (4)
Gaussian basis functions:

These are local; a small change


in x only affect nearby basis
functions. ¹j and s control
location and scale (width).
Linear Basis Function Models (5)
Sigmoidal basis functions:

where

Also these are local; a small


change in x only affect nearby
basis functions. ¹j and s control
location and scale (slope).
Maximum Likelihood and Least Squares (1)

Assume observations from a deterministic function


with added Gaussian noise:
方差
where

which is the same as saying,

Given observed inputs, , and targets,


, we obtain the likelihood function

最大化似然函数
Maximum Likelihood and Least Squares (2)
在高斯噪声下，才能写成该形
Taking the logarithm, we get 式，此时最小均方误差和最大
似然估计等价

where

is the sum-of-squares error.


Maximum Likelihood and Least Squares (3)

Computing the gradient and setting it to zero yields

Solving for w, we get The Moore-Penrose


pseudo-inverse, .

where

多个（ x,y ）对应


Geometry of Least Squares
Consider Φ 相当于空间中的基，称其为基函数

N-dimensional
M-dimensional

S is spanned by .
wML minimizes the distance
between t and its orthogonal
projection on S, i.e. y.

Y 的每一个值为 M 个向量的线型组合
Sequential Learning
Data items considered one at a time (a.k.a.
online learning); use stochastic (sequential)
gradient descent:

This is known as the least-mean-squares (LMS)


algorithm. Issue: how to choose ´?
1st Order Polynomial
3rd Order Polynomial
9th Order Polynomial
Over-fitting

Root-Mean-Square (RMS) Error:


Polynomial Coefficients
Regularized Least Squares (1)
Consider the error function:
添加惩罚项

Data term + Regularization term

With the sum-of-squares error function and a


quadratic regularizer, we get

¸ is called the
which is minimized by regularization
coefficient.
Regularized Least Squares (1)

Is it true that or
Regularized Least Squares (2)
With a more general regularizer, we have

Lasso Quadratic
Regularized Least Squares (3)
Lasso tends to generate sparser solutions than a
quadratic
regularizer.
Understanding Lasso Regularizer

• Lasso regularizer plays the role of


thresholding
Multiple Outputs (1)
Analogously to the single output case we have:

Given observed inputs, , and targets,


, we obtain the log likelihood function
Multiple Outputs (2)
Maximizing with respect to W, we obtain

If we consider a single target variable, tk, we see that

where , which is identical with the


single output case.
The Bias-Variance Decomposition (1)
Recall the expected squared loss,

where

The second term of E[L] corresponds to the noise


inherent in the random variable t.
What about the first term?
The Bias-Variance Decomposition (2)
Suppose we were given multiple data sets, each of
size N. Any particular data set, D, will give a
particular function y(x;D). We then have
The Bias-Variance Decomposition (3)
Taking the expectation over D yields

偏差描述与真实值之间的差距
方差描述与观测值均值之间的差距
The Bias-Variance Decomposition (4)
Thus we can write

where
Bias-Variance Tradeoff
The Bias-Variance Decomposition (5)
Example: 25 data sets from the sinusoidal, varying
the degree of regularization, ¸.
The Bias-Variance Decomposition (6)
Example: 25 data sets from the sinusoidal, varying
the degree of regularization, ¸.
The Bias-Variance Decomposition (7)
Example: 25 data sets from the sinusoidal, varying
the degree of regularization, ¸.
The Bias-Variance Trade-off
From these plots, we note
that an over-regularized
model (large ¸) will have a
high bias, while an under-
regularized model (small ¸)
will have a high variance.
Bayesian Linear Regression (1)
Define a conjugate prior over w

Combining this with the likelihood function and using


results for marginal and conditional Gaussian
distributions, gives the posterior

where
Bayesian Linear Regression (2)
A common choice for the prior is

for which

Next we consider an example …


Bayesian Linear Regression (3)
0 data points observed
Prior Data Space
Bayesian Linear Regression (4)
1 data point observed
Likelihood Posterior Data Space
Bayesian Linear Regression (5)
2 data points observed
Likelihood Posterior Data Space
Bayesian Linear Regression (6)
20 data points observed
Likelihood Posterior Data Space
Predictive Distribution (1)
Predict t for new values of x by integrating
over w:

where
Predictive Distribution (1)
How to compute the predictive distribution ?
Predictive Distribution (2)
Example: Sinusoidal data, 9 Gaussian basis functions,
1 data point
Predictive Distribution (3)
Example: Sinusoidal data, 9 Gaussian basis functions,
2 data points
Predictive Distribution (4)
Example: Sinusoidal data, 9 Gaussian basis functions,
4 data points
Predictive Distribution (5)
Example: Sinusoidal data, 9 Gaussian basis functions,
25 data points
Salesman’s Problem
Given
• a consumer a described by a vector x
• a product b to sell with base cost c
• estimated price distribution of b in the mind
of a is Pr(t|x, w) = N(xTw, 2)

What is price should we offer to a ?


Salesman’s Problem
Bayesian Model Comparison (1)
How do we choose the ‘right’ model?
Bayesian Model Comparison (1)
How do we choose the ‘right’ model?

Assume we want to compare models Mi, i=1, …,L,


using data D; this requires computing

Posterior Prior Model evidence or


marginal likelihood
Bayesian Model Comparison (1)
How do we choose the ‘right’ model?

Posterior Prior Model evidence or


marginal likelihood
Bayesian Model Comparison (3)
For a model with parameters w, we get the
model evidence by marginalizing over w
Bayesian Model Comparison (4)
For a given model with a
single parameter, w, con-
sider the approximation

where the posterior is


assumed to be sharply
peaked.
Bayesian Model Comparison (4)
For a given model with a
single parameter, w, con-
sider the approximation

where the posterior is


assumed to be sharply
peaked.
Bayesian Model Comparison (5)
Taking logarithms, we obtain

Negative

With M parameters, all assumed to have the same


ratio , we get

Negative and linear in M.


Bayesian Model Comparison (5)

Data matching Model Complexity


Bayesian Model Comparison (6)
Matching data and model complexity
What You Should Know
• Least square and its maximum likelihood
interpretation
• Sequential learning for regression
• Regularization and its effect
• Bayesian regression
• Predictive distribution
• Bias variance tradeoff
• Bayesian model selection

You might also like