Performance Evaluation for

Classification
 Confusion Matrix – 2-class

• True Positive: Number of test samples correctly predicted as

positive class (Class1).
• True Negative: Number of test samples correctly predicted as
negative class (Class2).
• False Positive: Number of test samples predicted as positive
class (Class1) but actually belonging to negative class
(Class2).
• False Negative: Number of test samples predicted as negative
class (Class2) but actually belonging to positive class (Class1). 2
Confusion Matrix – 2-class

Total test
samples
in class1

3
Confusion Matrix – 2-class

Total test
samples
in class2

4
 Confusion Matrix – 2-class

Total test
samples
predicted as
class1

• Biometric authentication system to access account

– False Positive (wrongly detecting as genuine person) should be low
– Some False Negative (Not detecting a genuine person as genuine) is OK
– Precision should be high
5
 Confusion Matrix – 2-class

Total test
samples
predicted as
class2

• Medical image analysis of microscopic image to detect the presence

of cancer
– False Negative (Detecting cancerous image as not cancer) should be low
– Some False Positive (Detecting a non-cancerous images as cancer) is OK
– Recall should be high 6
Accuracy – 2-class

7
 Confusion Matrix - Multiclass
Illustration: Number of classes is 3. It can be extended to any
number of classes

• C11: Number of test examples predicted as class1 and actually

belonging to class1
• C12: Number of test examples predicted as class1, but actually
belonging to class2
• C13: Number of test examples predicted as class1, but actually
belonging to class3
• Similarly C21, C22, C23, C31, C32 and C33 are interpreted
8
 Confusion Matrix - Multiclass
With reference to Class1:

• True Positive: Number of test samples correctly predicted as

positive class (class1) (C11).
• True Negative: Number of test samples correctly predicted as
negative class (class2 and class3) (C22+C33).
• False Positive: Number of test samples predicted as positive class
(class1) but actually belonging to negative class (class2 and
class3) (C12+C13)
• False Negative: Number of test samples predicted as negative
class (class2 and class3) but actually belonging to positive class
(class1) (C21+C31) 9
 Confusion Matrix - Multiclass
With reference to Class2:

• True Positive: Number of test samples correctly predicted as

positive class (class2) (C22).
• True Negative: Number of test samples correctly predicted as
negative class (class1 and class3) (C11+C33).
• False Positive: Number of test samples predicted as positive class
(class2) but actually belonging to negative class (class1 and
class3) (C21+C23)
• False Negative: Number of test samples predicted as negative
class (class1 and class3) but actually belonging to positive class
(class2) (C12+C32) 10
 Confusion Matrix - Multiclass
With reference to Class3:

• True Positive: Number of test samples correctly predicted as

positive class (class3) (C33).
• True Negative: Number of test samples correctly predicted as
negative class (class1 and class2) (C11+C22).
• False Positive: Number of test samples predicted as positive class
(class3) but actually belonging to negative class (class1 and
class2) (C31+C32)
• False Negative: Number of test samples predicted as negative
class (class1 and class2) but actually belonging to positive class
(class3) (C13+C23) 11
 Confusion Matrix - Multiclass
Example: Number of classes = 3. Same concept can be extended to
number of classes more than 3

Total samples used for testing

12
 Accuracy of Multiclass Classification
Example: Number of classes = 3. Same concept can be extended to
number of classes more than 3

13
 Binary (2-class) Classification:
Precision, Recall and F-measure

• Precision:
– Number of samples correctly classified as positive class,
out of all the examples classified as positive class
– It is also called positive predictive value
 Binary (2-class) Classification:
Precision, Recall and F-measure

• Recall:
– Number of samples correctly classified as positive class,
out of all the examples belonging to positive class
– It is also called as sensitivity or true positive rate (TPR)

15
 Binary (2-class) Classification:
Precision, Recall and F-measure

• F-measure or F-score or F1-score:

– Combines precision and recall
– Recall and precision are evenly weighted.
– Harmonic mean of precision and recall

16
 Supervised Machine Learning:
Pattern Classification
K-Nearest Neighbor, Reference Template Method
 K-Nearest Neighbours (K-NN) Method
• Consider the class labels of the K training examples
nearest to the test example
• Step 1: Compute Euclidean distance for a test
example x with every training examples, x1, x2, …, xn,
…, xN • Step 2: Sort the examples in
the training set in the
ascending order of the
distance to x
x2 • Step 3: Choose the first K
examples in the sorted list
– K is the number of
neighbours for text
x1 example
• Step 4: Test example is assigned the most common
class among its K neighbours
18
 Reference Templates Method

• Each class is represented by its reference templates

– Mean of each data points of each class as reference
template
– Let the data of class i be
• Ni: Number of examples (data points) in class i

– Mean of data points of a class i, μi is given as:

19
 Reference Templates Method
• Each class is represented by its reference templates
– Mean of each data points of each class as reference template
• For a test example, compute an Euclidean distance to all the
reference template corresponding to each class, ED(x, μi)

μi: Mean vector of class i

• The class of the nearest
reference template
(mean) is assigned to
the test pattern

x2 Class label for x

i
i = 1,2,…, M
M = Number of classes
x1 20
 Reference Templates Method
• Each class is represented by its reference templates
– Mean of each data points of each class as reference template
• For a test example, compute an Euclidean distance to all the
reference template corresponding to each class, ED(x, μi)

μi: Mean vector of class i

• The class of the nearest
reference template
(mean) is assigned to
the test pattern

x2
• Learning: Estimating first
order statistics (mean)
from the data of each
class
x1 21
 Illustration of Reference Templates
Method: Adult(1)-Child(0) Classification

• Training Phase:
– Compute sample mean vector from training
data of class 0 (Child)
μ0= [103.60 30.66]

22
 Illustration of Reference Templates
Method: Adult(1)-Child(0) Classification
• Training Phase:
– Compute sample mean vector from training
data of class 0 (Child)
μ0= [103.60 30.66]

– Compute sample mean vector from training

data of class 1 (Adult)
μ1= [166.00 67.12]

23
 Illustration of Reference Templates Method:
Adult(1)-Child(0) Classification
• Test Phase - Classification:
Test Example, x:
Height Weight Class
μ0 103.60 30.66 0
μ1 166.60 67.12 1

Weight
in Kg

Height in cm

• Compute Euclidean distance of test sample, x with mean

vector of class 0 (Child), μ0: ED(x, μ0) = 50.50
• Compute Euclidean distance of test sample, x with mean
vector of class 1 (Adult), μ1: ED(x, μ1) = 23.00
Class label of x = Adult 24
Modified Reference Templates Method
• Each class is represented by its reference templates
– Mean and variance (covariance) of data points of each
class as reference template
– Let the data of class i be
• Ni: Number of examples (data points) in class i

– Mean of data points of a class i, μi is given as:

– Covariance matrix of data points of a class i, Σi is given

as:

σjk: Covariance of
jth and kth attribute
25
 Modified Reference Templates Method
• Each class is represented by one or more reference templates
– Mean and variance (covariance) of data points of each class as
reference template
• For a test example, compute a Mahalanobis distance to all
the reference template corresponding to each class, MD(x, μi, Σi)

μi & Σi : Mean vector and Covariance matrix of class i

x2

The Mahalanobis distance is a measure

of the distance between a point and
x1 a distribution 26
 Modified Reference Templates Method
• Each class is represented by one or more reference templates
– Mean and variance (covariance) of data points of each class as
reference template
• For a test example, compute a Mahalanobis distance to all
the reference template corresponding to each class, MD(x, μi, Σi)

μi & Σi : Mean vector and Covariance matrix of class i

• The class of the nearest
reference templates is
assigned to the test pattern

x2 Class label for x = argmin MD(x, μi, Σi)

i
i = 1,2,…, M
M = Number of classes
x1 27
 Modified Reference Templates Method
• Each class is represented by one or more reference templates
– Mean and variance (covariance) of data points of each class as
reference template
• For a test example, compute a Mahalanobis distance to all
the reference template corresponding to each class, MD(x, μi, Σi)

μi & Σi : Mean vector and Covariance matrix of class i

• The class of the nearest
reference templates is
assigned to the test pattern
• Learning: Estimating
x2 – first order statistics (mean) and
– Second order statistics
(variance and covariance) from
the data of each class

x1 28
Illustration of Reference Templates Method:
Adult(1)-Child(0) Classification

• Training Phase:
– Compute sample mean vector from training
data of class 0 (Child)
μ0= [103.60 30.66]
– Compute sample covariance matrix from
training data of class 0 (Child)

Σ0= 109.38 61.35

61.35 43.54

29
Illustration of Reference Templates Method:
Adult(1)-Child(0) Classification
• Training Phase:
– Compute sample mean vector from training
data of class 0 (Child)
μ0= [103.60 30.66]
– Compute sample covariance matrix from
training data of class 0 (Child)

Σ0= 109.38 61.35

61.35 43.54

– Compute sample mean vector from training

data of class 1 (Adult)
μ1= [166.00 67.12]
– Compute sample covariance matrix from
training data of class 1 (Adult)

Σ1= 110.67 160.53

160.53 255.49
30
 Illustration of Reference Templates Method:
Adult(1)-Child(0) Classification
• Test Phase - Classification:
Test Example, x:
μ0= [103.60 30.66]
Class
0
Σ0= 109.38 61.35
61.35 43.54

Weight
μ1= [166.00 67.12] in Kg
Class
1
Σ1= 110.67 160.53
160.53 255.49

Height in cm
• Compute Mahalanobis distance of test sample,
x with mean vector and covariance matrix of
class 0 (Child): MD(x, μ0, Σ0) = 4.87 Class label of x =
• Compute Mahalanobis distance of test sample, Adult
x with mean vector and covariance matrix of
class 1 (Adult): MD(x, μ2, Σ2) = 2.07 31
 Classification using Reference
Template Methods
• For a test example, a distance measure is computed with
the reference template of each class
• The class of the reference template with least distance is
assigned to the test pattern
• When Mahalanobis distance is used, it gives the notion
that distance measure is computed between a test
example and the distribution (density) of a class
– Distribution (density) of class: All the training examples are
drawn from that distribution
– Density here is normal (Gaussian) density
• In other way, we are interested to estimate probability of
class, P(Ci | x)
– Given the test example x, what is the probability that it
belongs to ith class (Ci)
• Solution: Bayes classifier
32
 Text Books

1. J. Han and M. Kamber, Data Mining: Concepts and

Techniques, Third Edition, Morgan Kaufmann Publishers,
2011.

2. S. Theodoridis and K. Koutroumbas, Pattern Recognition,

Academic Press, 2009.

3. C. M. Bishop, Pattern Recognition and Machine Learning,

Springer, 2006.

33