Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
9 views

Linear Regression

Linear regression is a machine learning technique for modeling the relationship between inputs and continuous outputs. It can be used for polynomial curve fitting by treating the input features as polynomial basis functions. The order of the polynomial, M, determines the complexity of the model, with higher orders potentially overfitting the data. Maximum likelihood estimation and least squares methods are commonly used to fit the linear regression model by minimizing the error between predicted and actual target values.

Uploaded by

e1840006
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
9 views

Linear Regression

Linear regression is a machine learning technique for modeling the relationship between inputs and continuous outputs. It can be used for polynomial curve fitting by treating the input features as polynomial basis functions. The order of the polynomial, M, determines the complexity of the model, with higher orders potentially overfitting the data. Maximum likelihood estimation and least squares methods are commonly used to fit the linear regression model by minimizing the error between predicted and actual target values.

Uploaded by

e1840006
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Machine Learning & Pattern Recognition

Linear Regression –
Polynomial Curve fitting
Linear Regression - Polynomial Curve Fitting

• Given a training data set comprising N observations {xn}, where n = 1,...,N


together with corresponding target values {tn}. M is the order of the
polynomial and xj denotes the x raised to the power of j.

Although the function y(x,w) is a non-linear function of x, it is a linear function of coefficient of w.


Thushari Silva, PhD Essentials of Machine Learning
Linear Regression - Polynomial Curve Fitting

• Model Selection - How to choose M?

when M = 0 and M = 1, polynomials give rather poor fit to the data.

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression - Polynomial Curve Fitting

• Model Selection - How to choose M?

when M = 3 polynomial seems to give a better fit to the data. And when and M = 9, it gives
an excellent fit to the data. In fact polynomial passes exactly each data point and E(w∗) = 0
Thushari Silva, PhD Essentials of Machine Learning
Linear Regression - Polynomial Curve Fitting

• Over-fitting can be overcome by increasing the size of the data set.

Over fitting problem can also be avoided by introducing Bayesian approach.

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression - Polynomial Curve Fitting

• Model Selection - How to choose M?

when M is small,ERMS takes relatively larger values. When M is between 3 and 8,ERMS
takes relatively smaller values. And when M = 9, ERMS goes to Training error goes zero, but
the Testing Error is very large (i.e Polynomial with larger M has tuned to the random noise).
Thushari Silva, PhD Essentials of Machine Learning
Linear Basis Function Models

• The simplest linear model for regression is:


Y ( X ,W ) = w0 + w1 x1 + w2 x2 + ... + wD xD , where X = ( x1 , x2 ,..., xD )
• This simplest can be extended to cater many class of models by:

• In the simplest case, we use linear basis functions:


Thushari Silva, PhD Essentials of Machine Learning
Linear Basis Function Models

• Examples for basis functions:


Polynomial basis functions

• These are global. A small change in X


affects all basis functions.

Thushari Silva, PhD Essentials of Machine Learning


Linear Basis Function Models

• Examples for basis functions:


Gaussian basis functions

• These are local. A small change in X


only affects nearby basis functions.
µj and s control location and scale
(width) respectively.

Thushari Silva, PhD Essentials of Machine Learning


Linear Basis Function Models

• Examples for basis functions:


Sigmoidal basis functions

where

• These are local. A small change in X


only affects nearby basis functions.
µj and s control location and scale
(slope) respectively.

Thushari Silva, PhD Essentials of Machine Learning


Likelihood Function -Intuition

Likelihood Function
• The probability of observing the given data as a function parameters.
• The Likelihood function for{xi }in=1 which are independent and identically
distributed with P( xi ,q ) and{ti }in=1 are the corresponding targets.

• This approach essentially chooses the value for the parameter, θ ,that
maximizes the likelihood of seeing the sample data that is collected
• These parameters are known as Maximum Likelihood Estimators (MLE)

Thushari Silva, PhD Essentials of Machine Learning


Gaussian Distribution – Uni-variate vs. Multivariate

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression – Maximum Likelihood and least squares

• Assume observations from a deterministic function with added Gaussian


noise:

• which can be equally said as:

• The likelihood function for { xi }in=1 which are iid

Thushari Silva, PhD Essentials of Machine Learning


Gaussian Distribution – Uni-variate vs. Multivariate

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression – Maximum Likelihood and least squares

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression – Maximum Likelihood Estimators

• Computing the gradient and setting it to zero yields

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression – Maximum Likelihood Estimators
• Computing the gradient and setting it to zero yields

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression
Multiple Outputs

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression – Summary

Thushari Silva, PhD Essentials of Machine Learning


Linear Regression – Summary

Thushari Silva, PhD Essentials of Machine Learning

You might also like