9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

 Navigation

Want help with algorithms? Take the FREE Mini-Course.

Search... 

Parametric and Nonparametric Machine Learning

Algorithms
by Jason Brownlee on March 14, 2016 in Understand Machine Learning Algorithms

Tweet Share Share

What is a parametric machine learning algorithm and how is it different from a nonparametric machine
learning algorithm?

In this post you will discover the difference between parametric and nonparametric machine learning
algorithms.

Discover how machine learning algorithms work including kNN, decision trees, naive bayes, SVM,
ensembles and much more in my new book, with 22 tutorials and examples in excel.

Let’s get started.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 1/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Your Start in Machine ×

Learning
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

Email Address

START
Parametric and Nonparametric Machine MY EMAIL
Learning COURSE
Algorithms
Photo by John M., some rights reserved.

Learning a Function
Machine learning can be summarized as learning a function (f) that maps input variables (X) to output
variables (Y).

Y = f(x)

An algorithm learns this target mapping function from training data.

The form of the function is unknown, so our job as machine learning practitioners is to evaluate different
machine learning algorithms and see which is better at approximating the underlying function.
Your Start in Machine Learning

Different algorithms make different assumptions or biases about the form of the function and how it can be
learned.

Get your FREE Algorithms Mind Map

I've created a handy mind map of 60+ algorithms organized by type.

Download it, print it and use it.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 2/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Download For Free

Also get exclusive access to the machine

learning algorithms email mini-course.

Parametric Machine
Learning Algorithms
Sample of the handy machine learning algorithms mind map.
Your Start in Machine ×
Assumptions can greatly simplify the
Learning
learning process, but can also limit what can be learned. Algorithms that simplify the function to a known
form are called parametric machine learning algorithms.
You can master applied Machine Learning
without math or fancy degrees.
A learning model that summarizes data with a setFind
of parameters of free
out how in this fixed size
and (independent
practical course. of the
 number of training examples) is called a parametric model. No matter how much data you throw
at a parametric model, it won’t change its mind about how
Email many parameters it needs.
Address

— Artificial Intelligence: A Modern Approach, page 737 START MY EMAIL COURSE

The algorithms involve two steps:

1. Select a form for the function.

2. Learn the coefficients for the function from the training data.

An easy to understand functional form for the mapping function is a line, as is used in linear regression:

b0 + b1*x1 + b2*x2 = 0

Where b0, b1 and b2 are the coefficients of the line that control the intercept and slope, and x1 and x2 are
two input variables.

Assuming the functional form of a line greatly simplifies the learning process. Now, all we need to do is
Your Start in Machine Learning
estimate the coefficients of the line equation and we have a predictive model for the problem.

Often the assumed functional form is a linear combination of the input variables and as such parametric
machine learning algorithms are often also called “linear machine learning algorithms“.

The problem is, the actual unknown underlying function may not be a linear function like a line. It could be
almost a line and require some minor transformation of the input data to work right. Or it could be nothing
like a line in which case the assumption is wrong and the approach will produce poor results.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 3/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Some more examples of parametric machine learning algorithms include:

Logistic Regression
Linear Discriminant Analysis
Perceptron
Naive Bayes
Simple Neural Networks

Benefits of Parametric Machine Learning Algorithms:

Simpler: These methods are easier to understand and interpret results.

Speed: Parametric models are very fast to learn from data.
Less Data: They do not require as much training data and can work well even if the fit to the data is
not perfect. Your Start in Machine ×
Limitations of Parametric Machine Learning Algorithms: Learning
You can master applied Machine Learning
Constrained: By choosing a functional form these methods are highly constrained to the specified
without math or fancy degrees.
form.
Find out how in this free and practical course.
Limited Complexity: The methods are more suited to simpler problems.
Poor Fit: In practice the methods are unlikely to match the underlying mapping function.
Email Address

Nonparametric Machine Learning Algorithms

START MY EMAIL COURSE
Algorithms that do not make strong assumptions about the form of the mapping function are called
nonparametric machine learning algorithms. By not making assumptions, they are free to learn any
functional form from the training data.

Nonparametric methods are good when you have a lot of data and no prior knowledge, and
 when you don’t want to worry too much about choosing just the right features.

— Artificial Intelligence: A Modern Approach, page 757

Nonparametric methods seek to best fit the training data in constructing the mapping function, whilst
maintaining some ability to generalize to unseen data. As such, they are able to fit a large number of
functional forms.
Your Start in Machine Learning
An easy to understand nonparametric model is the k-nearest neighbors algorithm that makes predictions
based on the k most similar training patterns for a new data instance. The method does not assume
anything about the form of the mapping function other than patterns that are close are likely have a similar
output variable.

Some more examples of popular nonparametric machine learning algorithms are:

k-Nearest Neighbors
Decision Trees like CART and C4.5
https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 4/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Support Vector Machines

Benefits of Nonparametric Machine Learning Algorithms:

Flexibility: Capable of fitting a large number of functional forms.

Power: No assumptions (or weak assumptions) about the underlying function.
Performance: Can result in higher performance models for prediction.

Limitations of Nonparametric Machine Learning Algorithms:

More data: Require a lot more training data to estimate the mapping function.
Slower: A lot slower to train as they often have far more parameters to train.
Overfitting: More of a risk to overfit the training data and it is harder to explain why specific predictions
are made.
Your Start in Machine ×
Further Reading Learning
You can master applied Machine Learning
This section lists some resources if you are looking to learn more about the difference between parametric
without math or fancy degrees.
and non-parametric machine learning algorithms.
Find out how in this free and practical course.

Books
Email Address
An Introduction to Statistical Learning: with Applications in R, chapter 2
Artificial Intelligence: A Modern Approach, chapter 18
START MY EMAIL COURSE

Posts
What are the advantages of using non-parametric methods in machine learning? on Quora
What are the disadvantages of non-parametric methods in machine learning? on Quora
Nonparametric statistics on Wikipedia
Parametric statistics on Wikipedia
Parametric vs. Nonparametric on Stack Exchange

Summary
In this post you have discovered the difference between parametric and nonparametric machine learning
algorithms.
Your Start in Machine Learning
You learned that parametric methods make large assumptions about the mapping of the input variables to
the output variable and in turn are faster to train, require less data but may not be as powerful.

You also learned that nonparametric methods make few or no assumptions about the target function and in
turn require a lot more data, are slower to train and have a higher model complexity but can result in more
powerful models.

If you have any questions about parametric or nonparametric machine learning algorithms or this post,
leave a comment and I will do my best to answer them.
https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 5/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Update: I originally had some algorithms listed under the wrong sections like neural nets and naive bayes,
which made things confusing. All fixed now.

Frustrated With Machine Learning Math?

See How Algorithms Work in Minutes
…with just arithmetic and simple examples

Discover how in my new Ebook: Master Machine Learning Algorithms

Your Start in Machine ×
It covers explanations and examples of 10 top algorithms, like:
Learning
Linear Regression, k-Nearest Neighbors, Support Vector Machines and much
more…
You can master applied Machine Learning
Finally, Pullmath
without Back the Curtain
or fancy degrees. on
Find out Learning
Machine how in this free and practical course.
Algorithms
Skip the Academics. Just Results.
Email Address
Click to learn more.

START MY EMAIL COURSE

Tweet Share Share

About Jason Brownlee

Jason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with
modern machine learning methods via hands-on tutorials.
View all posts by Jason Brownlee →

 How Machine Learning Algorithms Work (they learn a mapping of input to output)
Your Startand
Supervised in Unsupervised
Machine Learning
Machine Learning Algorithms 

41 Responses to Parametric and Nonparametric Machine Learning Algorithms

REPLY 
confused beginner March 14, 2016 at 6:02 pm #

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 6/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

hi jason

thanks for taking your time to summarize these topics so that even a novice like me can understand. love
your posts

i have a problem with this article though, according to the small amount of knowledge i have on
parametric/non parametric models, non parametric models are models that need to keep the whole data set
around to make future predictions. and it looks like Artificial Intelligence: A Modern Approach, chapter 18
agrees with me on this fact stating neural nets are parametric and once the weights w are learnt we can get
rid of the training set. i would say its the same case with trees/naive bays as well.

so what was your thinking behind in categorizing these methods as non-parametric?

thanks,
a confused beginner
Your Start in Machine ×
Learning
REPLY 
Jason Brownlee July 17, 2016 at 6:57 am #
You can master applied Machine Learning
without math or fancy degrees.
Indeed simple multilayer perceptron neural nets are parametric models.
Find out how in this free and practical course.
Non-parametric models do not need to keep the whole dataset around, but one example of a non-
parametric algorithm is kNN that does keep the whole dataset. Instead, non-parametric models can vary
Email Address
the number of parameters, like the number of nodes in a decision tree or the number of support vectors,
etc.
START MY EMAIL COURSE

REPLY 
mlvi July 27, 2017 at 1:49 am #

Isn’t number of nodes in the decision tree a hyper parameter?

One more question is, How do you deploy non parametric machine learning models in production as
there parameters are not fixed?

REPLY 
Jason Brownlee July 27, 2017 at 8:10 am #

No, but the max depth of the tree is.

Your Start in Machine Learning
You can finalize your model, save it to file and load it later to make predictions on new data.

See this post:

http://machinelearningmastery.com/train-final-machine-learning-model/

Does that help?

REPLY 
Another confused beginner March 15, 2016 at 3:13 am #
https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 7/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

I am also interesting to know why Naive Bayes is categorized as non-parametric.

REPLY 
Jason Brownlee July 17, 2016 at 7:06 am #

Yes, Naive bayes is generally a parametric method as we choose a distribution (Gaussian) for
the input variables, although there are non-parametric formulations of the method that use a kernel
estimator. In fact, these may be more common in practice.

REPLY 
Ecolss March 15, 2016 at 5:41 pm #

Confused here too. Your Start in Machine ×

Learning
AFAIK, parametric models have fixed parameter set, i.e. the amount of parameters won’t change once you
have designed the model, whereas the amount of parameters of non-parametric models varies, for example,
You can master applied Machine Learning
Gaussian Process and matrix factorization for collaborative filtering etc.
without math or fancy degrees.
Correct me if I’m wrong Find out how in this free and practical course.

Email Address

REPLY 
Jason Brownlee July 17, 2016 at 7:06 am #
START MY EMAIL COURSE
This is correct.

REPLY 
Simon Tse July 16, 2016 at 10:21 pm #

I think the classification does not really depend on what ‘parameters’ are. It’s about the assumption
you have made when you try to construct a model or function. Parametric models usually has a probability
model (i.e. pdf) behind to support the function-finding process such as normal distribution or other
distribution model.

On the other hand, non-parametric model just depends on the error minimisation search process to identify
the set of ‘parameters’ which has nothing to do with a pdf.

Your Start in
So, parameters are still there for both parametric and non-parametric MLMachine Learning
algo. It just doesn’t have additional
layer of assumption to govern the nature of pdf of which the ML algo tries to determine.

REPLY 
Jason Brownlee July 17, 2016 at 7:10 am #

Hi Simon, the statistical definition of parametric and non-parametric does not agree with you.

The crux of the definition is whether the number of parameters is fixed or not.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 8/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

It might be more helpful for us to consider linear and non-linear methods instead…

REPLY 
Kevin August 11, 2016 at 1:11 pm #

Is there a relation between parametric/nonparametric models and lazy/eager learning?

REPLY 
ANUDEEP VANJAVAKAM September 24, 2016 at 11:29 am #

In machine learning literature, nonparametric methods are also

call instance-based or memory-based learning algorithms.
Your
-Store the training instances in a lookup table and interpolate Start in Machine ×
from these for prediction. Learning
-Lazy learning algorithm, as opposed to the eager parametric
methods, which have simple model and a small numberYou can master applied Machine Learning
without
of parameters, and once parameters are learned we no longermath or fancy degrees.
keep the training set. Find out how in this free and practical course.

Email Address

REPLY 
Jianye September 27, 2016 at 11:18 am #
START MY EMAIL COURSE
I have questions of distinguishing between parametric and non parametric algorithms: 1) for linear
regression, we can also introducing x^2, x^3 … to make the boundary we learned nonlinear, does it mean
that it becomes non parametric in this case?

2) The main difference between them is that SVM puts additional constraints on how do we select the
hyperplane . Why perception is considered as parametric while svm is not?

REPLY 
Jason Brownlee September 28, 2016 at 7:35 am #

Hi Jianye,

When it comes down to it, parametric means a fixed number of model parameters to define the modeled
decision. Your Start in Machine Learning

Adding more inputs makes the linear regression equation still parametric.

SVM can choose the number of support vectors based on the data and hyperparameter tuning, making
it non-parametric.

I hope that is clearer.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 9/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

REPLY 
Pramit Choudhary January 23, 2017 at 1:09 pm #

Hi Jason,
Nice content here. Had some suggestions,
1. Do you think, it would be a good idea to include histogram: as a simple non-parametric model for
estimation probability distribution ? Some beginners might be able to related to histograms.
2. Also, may be mentioning SVM(RBF kernel) as non-parametric to be precise.
What do you think ?

REPLY 
Jason Brownlee January 24, 2017 at 10:54 am #

Hi Pramit, Your Start in Machine ×

1. nice suggestion.
Learning
2. perhaps, there is debate about where SVM sits. I do think it is nonparametric as the number of
You can master applied Machine Learning
support vectors is chosen based on the data and the interaction with
without math orthe argument-defined
fancy degrees. margin.
Find out how in this free and practical course.

Email Address REPLY 

Manish Barnwal March 30, 2017 at 8:50 pm #

Jason, as always, an excellent post.

START MY EMAIL COURSE

REPLY 
Jason Brownlee March 31, 2017 at 5:54 am #

Thanks Manish.

REPLY 
amr gamal April 12, 2017 at 1:40 am #

jason ,it is a good post about parametric and non parametric model
but i still confused
did deep learning supposed to be parametric or non parametric and why
Your Start in Machine Learning
Best Regards

REPLY 
Jason Brownlee April 12, 2017 at 7:55 am #

There is not a hard line between parametric and non-parametric.

I think of neural nets as non-parametric myself.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 10/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

See this:
https://www.quora.com/Are-Neural-Networks-parametric-or-non-parametric-models

REPLY 
anubhav November 25, 2018 at 5:47 am #

Hello Jason, thanks for discussing this topic. your consideration of NN as non-parametric
doesn’t make sense to me as per your post & suggestions above.! Please [read here]
(https://stats.stackexchange.com/questions/322049/are-deep-learning-models-parametric-or-non-
parametric) & [hear](r/MachineLearning – Is artificial neural network a parametric or non-parametric
method?
(https://www.reddit.com/r/MachineLearning/comments/kvkud/is_artificial_neural_network_a_paramet
ric_or/c2noexo)) for more clarity and correct me where I am wrong.!
Your Start in Machine ×
Learning
You can master applied Machine Learning REPLY 
Jason Brownlee November 25, 2018 at 7:01 am #
without math or fancy degrees.
Perhaps you can summarize the linksFind
for out
me? how in this free and practical course.

Email Address

REPLY 
Aishwarya May 4, 2017 at 8:10 am #
START MY EMAIL COURSE
Hi
The answer is very convincing, i just have a small question, for pressure distribution plots which ML
algorithm should we consider?

REPLY 
Jason Brownlee May 5, 2017 at 7:26 am #

Sorry, I don’t know what pressure distribution plots are.

REPLY 
Sanket Maheshwari May 17, 2017 at 7:45 am #
Your Start in Machine Learning
Hi Jason,

Decision tree contains parameters like Splitting Criteria, Minimal Size, Minimal Leaf Size, Minimal Gain,
Maximal Depth then why it is called as non-parametric. Please throw some light on it.

REPLY 
Jason Brownlee May 17, 2017 at 8:45 am #

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 11/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

They are considered hyperparameters of the model.

The chosen split points are the parameters of the model and their number can vary based on specific
data. Thus, the decision tree is a nonparametric algorithm.

Does that make sense?

REPLY 
Sanket Maheshwari May 18, 2017 at 7:37 pm #

Could you please briefly tell me what are the parameters and hyperparameters in the following
models:

1.Naive Baye
2.KNN Your Start in Machine ×
3.Decision Tree
Learning
4.Multiple Regression
5.Logistic Regression You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
Jason Brownlee May 19, 2017 at 8:16 am #
Email Address

Yes, please search the blog for posts on each of these algorithms.
START MY EMAIL COURSE

REPLY 
Guiferviz November 3, 2017 at 10:45 pm #

Hi Jason! Nice blog.

I have a doubt about the “simple neural networks”, shouldn’t it be “neural networks” in general? The number
of parameters is determined a priori.

In addition, I think that linear SVM might be considered as a parametric model because, despite the number
of support vector varies with the data, the final decision boundary can be expressed as a fixed number of
parameters.

I know the distinction between parametric and non-parametric is a little bit ambiguous, but what I said makes
sense, right?
Your Start in Machine Learning

REPLY 
duribef May 12, 2018 at 4:06 am #

Up to this question! I have the same doubt about linear SVM.

Saludos!

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 12/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

REPLY 
Aniket Saxena November 7, 2017 at 3:25 am #

Hi Jason, I want to know that despite having not

required much data to train, does the parametric algorithms also cause overfitting? Or can they be lead to
underfitting, instead?

REPLY 
Jason Brownlee November 7, 2017 at 9:53 am #

Both types of algorithms can over and under fit data.

It is more common that parametric underfit and non-parametric overfit.

Your Start in Machine ×

Learning REPLY 
Aniket Saxena November 8, 2017 at 12:20 am #
You can master applied Machine Learning
Hi Jason, thanks for your help but there is a request by my
without sideortofancy
math also degrees.
look question posted above
my question because it is a nice question about distinctionFind
between parametric
out how in this freevs non-parametric
and and I am
practical course.
very curious to know your opinion about this question posted by Guiferviz on november 3, 2017. Please
answer to this question…….
Email Address

START MY EMAIL COURSE

REPLY 
Magnus January 31, 2018 at 9:10 pm #

Hi Jason, you mention that simple multilayer perceptron neural nets are parametric models. This I
understand, but which neural networks are then non-parametric? I assume e.g. that neural nets with
dropouts are non-parametric?

REPLY 
Jason Brownlee February 1, 2018 at 7:19 am #

Perhaps. Categorizing algorithms gets messy.

Your Start in Machine Learning

REPLY 
ali October 1, 2018 at 11:34 pm #

if we are doing regression for decision trees

do we need to check for correlation among the features?
when we talk about nonparamertic or parametric are we talking about the method like CART or we are
talking about the data.

and if my data are not normally distributed do I have to do data transformation to make them normally dis. if I
want to use parametric or nonparamertic

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 13/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

REPLY 
Jason Brownlee October 2, 2018 at 6:25 am #

It is a good idea to make the problem as simple as possible for the model.

Nonlinear methods do not require data with a Normal distribution.

REPLY 
sindhu October 7, 2018 at 12:37 am #

Hi Jason,
Good post.Could u pls explain parametric and non parametric methods by an example?
Bit confused about the parameters(what are the parameters,model parameters).For example,in the script
Your Start in Machine ×
the X and y values are the parameters?
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Jason Brownlee October 7, 2018 at 7:26 am #
Find out how in this free and practical course.

Great question, this post will help:

Email Address
https://machinelearningmastery.com/difference-between-a-parameter-and-a-hyperparameter/

START MY EMAIL COURSE

REPLY 
Yogesh Soni June 9, 2019 at 4:19 am #

Hi Jason

Can you post or let me know about parameter tuning.

REPLY 
Jason Brownlee June 9, 2019 at 6:22 am #

Yes, I have many posts, try a search for “grid search”

Your Start in Machine Learning

Leave a Reply

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 14/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Name (required)

Email (will not be published) (required)

Your Start in Machine ×

Website Learning
You can master applied Machine Learning
without math or fancy degrees.
SUBMIT COMMENT Find out how in this free and practical course.

Email Address
Welcome to Machine Learning Mastery!

Hi, I'm Jason Brownlee, PhD. START MY EMAIL COURSE

I write tutorials to help developers (like you) get results with machine learning.

Read More

More Tutorials?
Start Here
(over 800 tutorials!)

Your Start in Machine Learning

Understand How Algorithms Work
No theory, just clear descriptions.

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 15/16
9/2/2019 Parametric and Nonparametric Machine Learning Algorithms

Click to Get Started Now!

Your Start in Machine ×

Learning
You can master applied Machine Learning
without math or fancy degrees.
Join Newsletter
Find out how in this free and practical course.

Subscribe Now
(join over 150,000 Email
readers!)
Address

START MY EMAIL COURSE

© 2019 Machine Learning Mastery Pty. Ltd. All Rights Reserved.

Address: PO Box 206, Vermont Victoria 3133, Australia. | ACN: 626 223 336.
RSS | Twitter | Facebook | LinkedIn

Privacy | Disclaimer | Terms | Contact | Sitemap | Search

Your Start in Machine Learning

https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/ 16/16