20180723161729D4730 - Pert18 - K-Nearest Neighbor

Course : Artificial Intelligence
Effective Period : September 2018
K-Nearest Neighbor
Session 18
1
Learning Outcomes
At the end of this session, students will be able to:
 LO 5: Apply various learning algorithms to solve the problems
 LO 6: Apply AI algorithms on various applications such as

Game AI, Natural Language Processing, and Computer Vision
2
Outline
1. Non-Parametric Model
2. K-Nearest Neighbor
3. Distance Metric
3
Non-Parametric Model
• Parametric model
– A learning model that summarizes data with a set of

parameters of fixed size
– Linear regression & artificial neural network
• Non-parametric model
– One that cannot be characterized by a bounded set of

parameters
– K-nearest neighbors
4
Non-Parametric Model
• For example, suppose that each hypothesis we generate
simply retains within itself all of the training examples and uses
all of them to predict the next example
– It would be non-parametric solution (instance based

learning or memory based learning)
• The simplest instance based learning is
– Table lookup
5
Table Lookup
• Take all the training examples, put them in a lookup table,
and then when asked for h(x), see if x is in the table
– If it is, return the corresponding y
• The problem with this method is that it does not generalize

well: when x is not in the table all it can do is return some
default value
6
K-nearest Neighbors
• We can improve on table lookup with a slight variation: given a
query xq, find the k examples that are nearest to xq.
– This is called k-nearest neighbors lookup (NN(k,xq))
7
K-nearest Neighbors
• To do classification, first find NN(k, xq), then take the plurality
vote of the neighbors (which is the majority vote in the case of
binary classification)
– To avoid ties, k is always chosen to be an odd number
• To do regression, we can take the mean or median of the k

neighbors, or we can solve a linear regression problem on the
neighbors.
8
K-nearest Neighbors
• We can improve on table lookup with a slight variation: given a
query xq, find the k examples that are nearest to xq.
– This is called k-nearest neighbors lookup (NN(k,xq))
• To do classification, first find NN(k, xq), then take the plurality

vote of the neighbors (which is the majority vote in the case of
binary classification)
– To avoid ties, k is always chosen to be an odd number
9
Decision Boundary of k=1
Might be overfitting
10
Decision Boundary of k=5
11
Distance Metric
• The very word “nearest” implies a distance metric
• Typically, distances are measured with a Minkowski distance

or Lp norm
• P = 2  Euclidean distance
• P = 1  Manhattan distance
12
Distance Metric
• Euclidean distance is used if the dimensions are measuring
similar properties, such as the width, height and depth of
parts on a conveyor belt
• Manhattan distance is used if they are dissimilar, such as

age, weight, and gender of a patient.
13
Distance Metric
• Hamming distance?
• Mahalanobis distance?
14
Distance Metric
• If we use the raw numbers from each dimension then the total
distance will be affected by a change in scale in any dimension
– To avoid this, it is common to apply normalization to the

measurements in each dimension
– Simple approach is to compute mean and standard

deviation
15
Nearest Neighbor Classifier
• Assign label of nearest training data point to each test data
point
from Duda et al.
Voronoi partitioning of feature space

Source: D. Lowe for two-category 2D and 3D data 16
K-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2
x1
17
1-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2
x1
3-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2
x1
5-nearest neighbor
x
x
x o
x x
x
+ o
o x
x
o o+
o
o
x2
x1
Using K-NN
• Simple, a good one to try first
• With infinite examples, 1-NN provably has error that is at most

twice Bayes optimal error
References
• Stuart Russell, Peter Norvig. 2010. Artificial Intelligence : A
Modern Approach. Pearson Education. New Jersey.
ISBN:9780132071482
• https://www.cc.gatech.edu/~hays/compvision/lectures/17.pdf
22

20180723161729D4730 - Pert18 - K-Nearest Neighbor

Uploaded by

Copyright:

Available Formats

20180723161729D4730 - Pert18 - K-Nearest Neighbor

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

20180723161729D4730 - Pert18 - K-Nearest Neighbor

Uploaded by

Copyright:

Available Formats

Course : Artificial Intelligence

Effective Period : September 2018

 LO 5: Apply various learning algorithms to solve the problems

 LO 6: Apply AI algorithms on various applications such as

– A learning model that summarizes data with a set of

– Linear regression & artificial neural network

– One that cannot be characterized by a bounded set of

– It would be non-parametric solution (instance based

• The simplest instance based learning is

– If it is, return the corresponding y

• The problem with this method is that it does not generalize

– This is called k-nearest neighbors lookup (NN(k,xq))

– To avoid ties, k is always chosen to be an odd number

• To do regression, we can take the mean or median of the k

– This is called k-nearest neighbors lookup (NN(k,xq))

• To do classification, first find NN(k, xq), then take the plurality

– To avoid ties, k is always chosen to be an odd number

• Typically, distances are measured with a Minkowski distance

• Manhattan distance is used if they are dissimilar, such as

– To avoid this, it is common to apply normalization to the

– Simple approach is to compute mean and standard

from Duda et al.

Voronoi partitioning of feature space

• Simple, a good one to try first

• With infinite examples, 1-NN provably has error that is at most

You might also like