4.0 Classification Methodologies R

4.
0 Classification
Methodologies
Eugene Rex L. Jalao, Ph.D.

Associate Professor
Department Industrial Engineering and Operations Research
University of the Philippines Diliman
@thephdataminer
This Session’s Outline
• What is Classification? • Ensembles

• Frequency Table – Adaboost
– Zero R – Random Forests
– One R • Model Evaluation
– Naïve Bayes • Case Study
– Decision Tree
– Rule Based Classifiers
• Similarity
– K-Nearest Neighbors
• Perceptron Based
– ANN
– SVM
E.R. L. Jalao, Copyright UP NEC,
2 2
eljalao@up.edu.ph
Objectives for this Session
• Define supervised learning and classification

• Identify appropriate analytics problems that can be solved
using classification
• Implement various classifier algorithms and interpret their
results
• Measure the performance of a given classifier as
compared to other classifiers generated

3 3
eljalao@up.edu.ph
Required Input Dataset Structure
Attributes/Columns/Variables/Features (𝑝 + 1)
Tid Refund Marital Taxable

Status Income Cheat
1 Yes Single 125K No

2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
Rows/ Instances
6 No Married 60K No
/Tuples /Objects
(𝑛) 7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
10
Response Variable/
Dependent Variable/
Predictor Variables/Independent Class Variable/ Label
Variables/Control Variables Variable/ Target Variable
4 4
eljalao@up.edu.ph
What is Classification?
• Given a collection of records

– Multiple predictor variables usually 𝑥1 , 𝑥2 , … 𝑥𝑝
– One categorical response variable usually 𝑦
• Find a model for predicting the class variable from the
predictor variables.
– Use historical “training data” to build the model
• Goal: previously unseen records should be predicted a class
as accurately as possible.
– Use “testing data” to test the accuracy of the model

5 5
eljalao@up.edu.ph
Illustrating the Classification Process
Train
Model
Classification
Evaluation
Models
Model Model 1 Model

Model 1 Application
Generation Performance
Dataset
Best
Model 2 New Dataset
Training Model 2 Classification
Performance with
Dataset Model
Predictions
Model n
Validation Model n
Performance
Dataset
Historical
Dataset
Testing
New Dataset
Dataset

6 6
eljalao@up.edu.ph
Golf Dataset
Temperature Humidity
Outlook Windy Play (Class)
Nominal Nominal
overcast hot high false yes
overcast cool normal true yes
overcast mild high true yes
overcast hot normal false yes
rainy mild high false yes
rainy cool normal false yes
rainy cool normal true no
rainy mild normal false yes
rainy mild high true no
sunny hot high false no
sunny hot high true no
sunny mild high false no
sunny cool normal false yes
sunny mild normal true yes

7 7
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
8 8
eljalao@up.edu.ph
ZeroR
• ZeroR is the simplest classification methodology which

relies on the target and ignores all predictors.
• Simply predicts the majority class.
• Is useful for determining a baseline performance as a
benchmark for other classification methods.
• For the Golf Data Set: Majority is Yes (9 Yes, 5 No)

– Predict Everything to be Yes: Accuracy = 64%

9 9
eljalao@up.edu.ph
Golf Dataset
Outlook Windy Play (Class) Predictions
Nominal Nominal
overcast hot high false yes yes
overcast cool normal true yes yes
overcast mild high true yes yes
overcast hot normal false yes yes
rainy mild high false yes yes
rainy cool normal false yes yes
rainy cool normal true no yes
rainy mild normal false yes yes
rainy mild high true no yes
sunny hot high false no yes
sunny hot high true no yes
sunny mild high false no yes
sunny cool normal false yes yes
sunny mild normal true yes yes
For the Golf Data Set: Majority is Yes (9 Yes, 5 No)

Predict Everything to be Yes: Accuracy = 64%
10 10
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
11 11
eljalao@up.edu.ph
OneR
• OneR, short for "One Rule"

• Simple, yet accurate, classification algorithm that
generates one rule for each predictor in the data
• The rule with the smallest total error as its "one rule“
• Algorithm:
– For each predictor:
• For each value of that predictor, make a rule as follows:
– Count how often each value of target (class) appears
– Find the most frequent class
– Make the rule assign that class to this value of the predictor
• Calculate the total error of the rules of each predictor
– Choose the predictor with the smallest total error.
12 12
eljalao@up.edu.ph
Golf Dataset
Outlook Windy Play (Class)
Nominal Nominal
overcast hot high false yes
overcast cool normal true yes
overcast mild high true yes
overcast hot normal false yes
rainy mild high false yes
rainy cool normal false yes
rainy cool normal true no
rainy mild normal false yes
rainy mild high true no
sunny hot high false no
sunny hot high true no
sunny mild high false no
sunny cool normal false yes
sunny mild normal true yes

13 13
eljalao@up.edu.ph
Generate Frequency Tables
Actual Play Golf Actual Play Golf
Outlook Table Temperature Table
Yes No Yes No
Sunny 2 3 Hot 2 2
Outlook Overcast 4 0 Temp. Mild 4 2
Rainy 3 2 Cool 3 1
Actual Play Golf Actual Play Golf

Humidity Table Windy Table
Yes No Yes No
High 3 4 False 6 2
Humidity Windy
Normal 6 1 True 3 3

14 14
eljalao@up.edu.ph
OneR Model
Actual Play Golf

Humidity Table
Yes No
High 3 4
Humidity
Normal 6 1
• IF Humidity = High THEN Play Golf = No

• IF Humidity = Normal THEN Play Golf = Yes
Actual Play Golf

Confusion Matrix
Yes No
Yes 6 1
Predicted Accuracy = 71%
No 3 4
15 15
eljalao@up.edu.ph
Golf Dataset
Outlook Windy Play (Class) Predictions
Nominal Nominal
overcast hot high false yes no
overcast cool normal true yes yes
overcast mild high true yes no
overcast hot normal false yes yes
rainy mild high false yes no
rainy cool normal false yes yes
rainy cool normal true no yes
rainy mild normal false yes yes
rainy mild high true no no
sunny hot high false no no
sunny hot high true no no
sunny mild high false no no
sunny cool normal false yes yes
sunny mild normal true yes yes
IF Humidity = High THEN Play Golf = No

IF Humidity = Normal THEN Play Golf = Yes
16 16
eljalao@up.edu.ph
Summary
• Performance of the Classifier
Actual Play Golf

Confusion Matrix
Yes No
Yes 6 1
Predicted Accuracy = 71%
No 3 4

17 17
eljalao@up.edu.ph
Prediction Confidence
• Level of Confidence we get for each prediction rule

• Example 1:
– IF Humidity = High THEN Play Golf = No (4/7)
– The rule is correct 7/11 times
• Example 2:
– IF Humidity = Normal THEN Play Golf = No (1/7)
– The rule is correct 7/8 times
• Hence we are more comfortable using Rule 2 in predicting
play golf as compared to Rule 1.

18 18
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
19 19
eljalao@up.edu.ph
Naïve Bayes Classifier
• Uses a probabilistic framework for classification

• Review: Conditional Probability, Bayes theorem:
𝑃 𝐶
𝑃 𝐶 𝐴 = 𝑃 𝐴|𝐶
𝑃 𝐴
• Interpreted as:
– The probability of C happening given A is true is equal to the
Probability of A happening given C is true times the ratio of the
probability of C happening and probability of A happening

20 20
eljalao@up.edu.ph
Example of Bayes Theorem
• Given Historical Data:

– A doctor knows that if a patient has meningitis, the probability
that he has a stiff neck is 50%.
– Prior probability of any patient having meningitis is 1/50,000
– Probability of any patient having stiff neck is 1/20
• If a patient has stiff neck, what’s the probability he/she
has meningitis?
𝑃 𝑆=𝑌𝑀=𝑌 𝑃 𝑀=𝑌
𝑃 𝑀=𝑌𝑆=𝑌 =
𝑃 𝑆=𝑌
1
0.5 ∗
= 50000 = 0.0002
1
20
21 21
eljalao@up.edu.ph
Naïve Bayes Example
• Prediction Scenario:
– Given 𝐴𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦, and 𝐴 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡
– Will I play Golf? Outlook Temp
Play
(Class)
overcast hot yes
• Historical Data was gathered as follows: overcast cool yes
overcast mild yes
overcast hot yes
rainy mild yes
rainy cool yes
rainy cool no
rainy mild yes
rainy mild no
sunny hot no
sunny hot no
sunny mild no
22
E.R. L. Jalao, Copyright UP NEC, sunny cool yes 22
eljalao@up.edu.ph
sunny mild yes
• Given a record with 𝑝 attributes (𝐴1 , 𝐴2 , … , 𝐴𝑝 )

– Goal is to predict class 𝐶, specifically, we want to find the value of C that
maximizes 𝑃 𝐶 𝐴1 , 𝐴2 , … , 𝐴𝑝 )
• Example:
– We choose the bigger probability:
• 𝑃 Play = Yes 𝐴𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦, 𝐴 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡)
or
• 𝑃 Play = No 𝐴𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦, 𝐴 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡 )
• How can we compute these directly from the data?

23 23
eljalao@up.edu.ph
• Using Bayes Formula:
𝐏 𝐏𝐥𝐚𝐲 = 𝒀𝒆𝒔 𝑨𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑨𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)=

𝑃 𝐴𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦 Play = Yes ∗ 𝑃 𝐴 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡 Play = Yes
∗ 𝑃(Play = Yes)/constant
𝐏 𝐏𝐥𝐚𝐲 = 𝑵𝒐 𝑨𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑨𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)=

𝑃 𝐴𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦 Play = No ∗ 𝑃 𝐴 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡 Play = No
∗ 𝑃(Play = No)/constant

24 24
eljalao@up.edu.ph
How to Estimate Probabilities from Discrete
Data?
𝑁𝑐
Outlook Temp
Play
• Class: 𝑃 𝐶 =
(Class) 𝑁
overcast hot yes – 𝑃(𝑁𝑜) = 5/14,
overcast cool yes – 𝑃(𝑌𝑒𝑠) = 9/14
overcast mild yes
• For discrete attributes:
overcast hot yes
|𝐴𝑖𝑘 |
rainy mild yes 𝑃(𝐴𝑖 | 𝐶𝑘) =
rainy cool yes 𝑁𝑐
rainy cool no – where |𝐴𝑖𝑘 | is number of
rainy mild yes instances having attribute 𝐴𝑖 and
rainy mild no belongs to class 𝐶𝑘
sunny hot no
– Examples:
sunny hot no
sunny mild no 𝑃(𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦|𝑃𝑙𝑎𝑦 =
3
sunny cool yes 𝑌𝑒𝑠) =
9
sunny mild yes

25 25
eljalao@up.edu.ph
Example of a Naïve Bayes Classifier
• Given today’s Outlook = Rainy, Temperature = Hot, Humidity =

High and Windy = False, will I play golf?
– 𝑷 𝑷𝒍𝒂𝒚 = 𝒀𝒆𝒔 𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)

= 𝑃 𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦 𝑃𝑙𝑎𝑦 = 𝑌𝑒𝑠 ∗ 𝑃 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡 𝑃𝑙𝑎𝑦 = 𝑌𝑒𝑠
∗ 𝑃 𝑃𝑙𝑎𝑦 = 𝑌𝑒𝑠
– 𝑷 𝑷𝒍𝒂𝒚 = 𝑵𝒐 𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)
= 𝑃 𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦 𝑃𝑙𝑎𝑦 = 𝑁𝑜 ∗ 𝑃 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡 𝑃𝑙𝑎𝑦 = 𝑁𝑜

∗ 𝑃 𝑃𝑙𝑎𝑦 = 𝑁𝑜

26 26
eljalao@up.edu.ph
How to Estimate Probabilities from
Continuous Data?
• For continuous attributes:
– Discretize the range into bins
• one ordinal attribute per bin
• violates independence assumption
– Two-way split: (𝐴 < 𝑣) or (𝐴 > 𝑣)
• choose only one of the two splits as new attribute
• Example: (𝐴 < 15) or (𝐴 ≥ 15)
Original Data Discretized Data
10 Low
20 High
15 High
11 Low
27 27
eljalao@up.edu.ph
• If one of the conditional probability is zero, then the entire

expression becomes zero
𝑁𝑖𝑐
𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙: 𝑃 𝐴𝑖 𝐶 =
𝑁𝑐
𝑐: number of classes
• Laplace Probability Estimation: 𝑝: prior probability
𝑁𝑖𝑐 + 1
𝐿𝑎𝑝𝑙𝑎𝑐𝑒: 𝑃 𝐴𝑖 𝐶 =
𝑁𝑐 + 𝑐

28 28
eljalao@up.edu.ph
Business Scenario: Nursery Data
• The Nursery Data set originally developed to rank

applications into nursery schools
– parents: usual, pretentious, great_pret
– has_nurs: proper, less_proper, improper, critical, very_crit
– form: complete, completed, incomplete, foster
– children: 1, 2, 3, more
– housing: convenient, less_conv, critical
– finance: convenient, inconv
– social: non-prob, slightly_prob, problematic
– health: recommended, priority, not_recom
– Rank: not_recom, priority, reocommend, spec_prior, very_recom
• 12960 Profiles
29 29
eljalao@up.edu.ph
• New Data:
parents has_nur form children housing finance social health rank

usual proper complete 1convenient inconv problematic priority ?
pretentious critical foster 2critical convenient slightly_prob not_recom ?

30 30
eljalao@up.edu.ph
• Type the following lines of code in RStudio and run.

• library(“RWeka“)

31 31
eljalao@up.edu.ph
• Results

32 32
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
33 33
eljalao@up.edu.ph
Decision Trees
• Decision tree builds classification models in the form of a

tree structure.
• It breaks down a dataset into smaller and smaller subsets
while at the same time an associated decision tree is
incrementally developed.
• The final result is a tree with decision nodes and leaf
nodes.
– A decision node (e.g., Outlook) has two or more branches (e.g.,
Sunny, Overcast and Rainy). Leaf node (e.g., Play) represents a
classification or decision.

34 34
eljalao@up.edu.ph
Decision Trees
Outlook
• Example
=Overcast !=Overcast
Wind
Play=Yes
=True !=True
Play=No
=Rain !=Rain
Play=Yes
E.R. L. Jalao, Copyright UP NEC, Play=No
35 35
eljalao@up.edu.ph
Decision Tree Generation: The ID3 Algorithm
• The core algorithm for building decision trees called ID3

• Employs a top-down, greedy search through the space of
possible branches with no backtracking.
• ID3 uses Entropy and Information Gain to construct a
decision tree.
• A decision tree is built top-down from a parent node and
involves partitioning the data into subsets that contain
instances with similar values (homogenous).

36 36
eljalao@up.edu.ph
Definition: Entropy
• Entropy: Measures homogeneity of a node.
𝐺𝑖𝑛𝑖 𝑡 = 1 − ෍ 𝑝 𝑗|𝑡 2
𝑗
– Where 𝑝( 𝑗 | 𝑡) is the relative frequency of class j at node t

37 37
eljalao@up.edu.ph
The Tax Dataset

Status Income Cheat

3 No Single 70K No
4 Yes Married 120K No
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10

38 38
eljalao@up.edu.ph
Entropy Calculation Example
• Entropy of Class Tax Cheat

Actual Tax Cheat
2 Yes No
𝐺𝑖𝑛𝑖 𝑡 = 1 − ෍ 𝑝 𝑗|𝑡
3 7
𝑗
2 2
7 3
𝐺𝑖𝑛𝑖 𝑡 = 1 − −
10 10
𝐺𝑖𝑛𝑖 𝑡 = 0.42
• If the sample is completely homogeneous (e.g. 10/0) the
gini index is zero.
• If the sample is an equally divided (e.g. 5/5) it has gini of
0.5.
39 39
eljalao@up.edu.ph
Definition: Information Gain
• Gini Split: Measures Reduction in Gini achieved because of

the split.
𝑘
𝑛𝑖
𝐺𝑖𝑛𝑖 𝑖 𝐺𝑎𝑖𝑛 = 𝐺𝑖𝑛𝑖 𝑝 − ෍ 𝐺𝑖𝑛𝑖(𝑐𝑖 )
𝑛
𝑖=1
– Where parent node, 𝑝 is split into 𝑘 partitions; 𝑛𝑖 is number of
records in partition 𝑖

40 40
eljalao@up.edu.ph
Example: Gini Split
Actual Tax Cheat
• Split on Refund:
Yes No 𝐺𝑖𝑛𝑖 𝑡 = 0.42
3 7
Yes Refund?
No
Actual Tax Actual Tax

Cheat Cheat
Yes No Yes No
0 3 3 4

41 41
eljalao@up.edu.ph
Example: Gini Split
• Calculate Gini of Children Nodes

Cheat Cheat
Yes No Yes No
0 3 3 4
2 2 2 2
3 0 3 4
𝐺𝑖𝑛𝑖 𝑡1 =1− − 𝐺𝑖𝑛𝑖 𝑡2 =1− −
3 3 7 7
𝑮𝒊𝒏𝒊 𝒕𝟏 = 𝟎 𝑮𝒊𝒏𝒊 𝒕𝟐 = 𝟎. 𝟒𝟖
3 7
• Gini Child: 𝐺𝑖𝑛𝑖 𝑆𝑝𝑙𝑖𝑡 𝑐 = 0 ∗
10
+ 0.48 ∗
10
= 0.34
• Gini Gain: 𝐺𝑖𝑛𝑖 𝐺𝑎𝑖𝑛 𝑐 = 0.42 − 0.34 = 0.08

42 42
eljalao@up.edu.ph
Computing Gini Index for Continuous
Attributes
• For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and computing gini index
– Choose the split position that has the least gini index
Cheat No No No Yes Yes Yes No No No No
Taxable Income
Sorted Values 60 70 75 85 90 95 100 120 125 220
Split Positions 55 65 72 80 87 92 97 110 122 172 230
<= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= >
Yes 0 3 0 3 0 3 0 3 1 2 2 1 3 0 3 0 3 0 3 0 3 0
No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0
Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420

43 43
eljalao@up.edu.ph
ID3 Algorithm
• Constructing a decision tree is all about finding attribute

that returns the highest information gain (i.e., the most
homogeneous branches).
• Step 1: Calculate entropy of the parent class.
Actual Tax Cheat

Yes No
3 7
𝐺𝑖𝑛𝑖 𝑡 = 0.42

44 44
eljalao@up.edu.ph
ID3 Algorithm
• Step 2: Calculate the Information Gain for all Attributes.
Actual Tax Cheat Actual Tax Cheat

Outlook Table Taxable Income Table
Yes No Yes No
Yes 0 3 Taxable > 97 K 0 4
Refund
No 3 4 Income <= 97K 4 3
Gain =0.08 Gain = 0.12
Actual Tax Cheat

Marital Status Table
Yes No
Yes 0 4
Married
No 3 3
Gain =0.12
45 45
eljalao@up.edu.ph
ID3 Algorithm
• Step 3: Choose attribute with the largest gini gain as the

decision node.
Marital
Married Status Single, Divorced

Cheat Cheat
Yes No Yes No
0 4 3 3

46 46
eljalao@up.edu.ph
ID3 Algorithm
• Step 4a: Label a branch if entropy = 0 as a leaf node.
Marital
Cheat = No Actual Tax

Cheat
Yes No
3 3

47 47
eljalao@up.edu.ph
Information Gain
• Step 4b: The ID3 algorithm is run recursively on the non-

leaf branches, until all data is classified.
Tid Refund Marital Taxable Single, Divorced
Actual Tax
Status Income Cheat
Cheat
Yes No
𝐺𝑖𝑛𝑖 𝑡 = 0.5
2 No Married 100K No Refund
3 3
3 No Single 70K No
4 Yes Married 120K No Yes No
6 No Married 60K No
7 Yes Divorced 220K No Cheat Cheat
8 No Single 85K Yes Yes No
Yes No
9 No Married 75K No
0 2 3 1
48 48
10
eljalao@up.edu.ph
Information Gain
• Step 5: The ID3 algorithm is run recursively on the non-

leaf branches, until all data is classified.
Marital
Cheat = No Refund
Yes No
Cheat = No

49 49
eljalao@up.edu.ph
Final Decision Tree
Marital
Cheat = No Refund
Yes No
Cheat = No Income
>=77.5K <77.5K
Cheat = No Cheat = Yes

50 50
eljalao@up.edu.ph
From Decision Trees To Rules
Classification Rules
Refund
Yes No (Refund=Yes) ==> No
NO (Refund=No, Marital Status={Single,Divorced},

Marita l
Taxable Income<80K) ==> No
{Single, Status
{Married} (Refund=No, Marital Status={Single,Divorced},
Divorced}
Taxable Income>80K) ==> Yes
Taxable NO
Income (Refund=No, Marital Status={Married}) ==> No
< 80K > 80K
NO YES
Rules are mutually exclusive and exhaustive
Rule set contains as much information as the
tree

51 51
eljalao@up.edu.ph
Rule Based Classifiers
• These are classifiers that utilizes rules to generate

predictions
• Created using the RIPPER Algorithm
• Same principle as the decision trees
• Based on frequency tables

52 52
eljalao@up.edu.ph
Characteristics of Rule-Based Classifier
• Mutually exclusive rules

– Classifier contains mutually exclusive rules if the rules are
independent of each other
– Every record is covered by at most one rule
• Exhaustive rules
– Classifier has exhaustive coverage if it accounts for every possible
combination of attribute values
– Each record is covered by at least one rule

53 53
eljalao@up.edu.ph
How does Rule-based Classifiers Work?
R1: (Give Birth = no)  (Can Fly = yes)  Birds
R2: (Give Birth = no)  (Live in Water = yes)  Fishes
R3: (Give Birth = yes)  (Blood Type = warm)  Mammals
R4: (Give Birth = no)  (Can Fly = no)  Reptiles
R5: (Live in Water = sometimes)  Amphibians
Name Blood Type Give Birth Can Fly Live in Water Class
lemur warm yes no no ?
turtle cold no no sometimes ?
dogfish shark cold yes no yes ?
A lemur triggers rule R3, so it is classified as a mammal

A turtle triggers both R4 and R5
A dogfish shark triggers none of the rules. Usually a default rule is
recommended

54 54
eljalao@up.edu.ph
Rule Coverage and Accuracy
• Coverage of a rule: Status Income Class
– Fraction of records that satisfy 1 Yes Single 125K No
the antecedent of a rule
• Accuracy of a rule: 3 No Single 70K No
– Fraction of records that satisfy 4 Yes Married 120K No
both the antecedent and 5 No Divorced 95K Yes
consequent of a rule 6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
If (Status=Single)  Cheat= No 10
Coverage = 40%, Accuracy = 50%

55 55
eljalao@up.edu.ph
Business Scenario: Bank Data
• A leading bank’s marketing department would like to

profile its clients to know which factors lead to the
purchase of one of its flagship products: PEP (Personal
Equity Plan)
• 600 Clients where gathered from the company’s various
databases each having variables such as: age, region,
income, sex, married, children, car, save_act, current_act,
and mortgage

56 56
eljalao@up.edu.ph

57 57
eljalao@up.edu.ph
58 58
eljalao@up.edu.ph
• To export the plot, click on Export –>Save as Image
• Set the Width to 2000 and Height to 1000.

• Click on Save.
• An image of the plot is saved in the working directory.

59 59
eljalao@up.edu.ph

60 60
eljalao@up.edu.ph

61 61
eljalao@up.edu.ph

62 62
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
63 63
eljalao@up.edu.ph
Nearest Neighbor Classifiers
• Basic idea:
– If it walks like a duck, quacks like a duck, then it’s probably a duck
Compute
Distance Test
Record
Training Choose k of the

Records “nearest” records

64 64
eljalao@up.edu.ph
Nearest Neighbor Classifiers
Unknown record  Requires three things
– The set of stored records
– Distance Metric to compute
distance between records
– The value of k, the number of
nearest neighbors to retrieve
 To classify an unknown record:

– Compute distance to other
training records
– Identify k nearest neighbors
– Use class labels of nearest
neighbors to determine the
class label of unknown record
(e.g., by taking majority vote)
65
eljalao@up.edu.ph
Definition of Nearest Neighbor
X X X
(a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor
K-nearest neighbors of a record x are data points

that have the k smallest distance to x
66
eljalao@up.edu.ph
Nearest Neighbor Classification
• Compute distance between two points:

– Euclidean distance
𝑑 𝑝, 𝑞 = ෍ 𝑝𝑖 − 𝑞𝑖 2
𝑖
• Determine the class from nearest neighbor list
– take the majority vote of class labels among the k-nearest
neighbors
– Weigh the vote according to distance
1
• weight factor, 𝑤 =
𝑑2

67 67
eljalao@up.edu.ph
Nearest Neighbor Classification
• Choosing the value of 𝑘:

– If k is too small, sensitive to noise points
– If k is too large, neighborhood may include points from other
classes
– Good Value for 𝑘 = 𝑛

68 68
eljalao@up.edu.ph
Business Scenario: Delivery Time Data
deltime ncases deldistance
Medium 7 560
Fast 3 220
• Management would like to Fast 3 340
Fast 4 80
determine whether a new order Fast 6 150
for soft drinks would be either a Medium
Fast
7
2
330
110
“Fast,” “Medium” or “Slow” Medium 7 210
Slow 30 1460
delivery from the Number of Slow 5 605
Cases and Distance. Slow 16 688
Slow 10 215
• 25 Historical Deliveries where Fast 4 255
Medium 6 462
gathered and profiled whether it Slow 9 448
Slow 10 776
was “Fast,” “Medium” or “Slow” Fast 6 200
Medium 7 132
Fast 3 36
Slow 17 770
Medium 10 140
Slow 26 810
Medium 9 450
69 Medium 8 63569
eljalao@up.edu.ph
Fast 4 150
Delivery Time Data: 1 Nearest Neighbor
deltime ncases deldistance sncases sdeldistance eucdistance
Medium 7 560 -0.26 0.46 0.616
Fast 3 220 -0.84 -0.58 1.447
• New Order: Fast 3 340 -0.84 -0.21 1.262
– 11 Cases Fast 4 80 -0.69 -1.01 1.644
Fast 6 150 -0.40 -0.80 1.299
– 500 ft Distance
Medium 7 330 -0.26 -0.24 0.782
• Standardized New Fast 2 110 -0.98 -0.92 1.774
Order Medium 7 210 -0.26 -0.61 1.064
Slow 30 1460 3.09 3.23 4.042
– 0.33 SCases
Slow 5 605 -0.55 0.60 0.930
– 0.28 SDistance Slow 16 688 1.05 0.86 0.929
• Distance = Slow 10 215 0.18 -0.60 0.888
Fast 4 255 -0.69 -0.47 1.266
0.33 − (−0.26 ) 2 + Medium 6 462 -0.40 0.16 0.736
0.28 − 0.46 2 Slow 9 448 0.03 0.12 0.332
– Distance = 0.616 Slow 10 776 0.18 1.13 0.861
Fast 6 200 -0.40 -0.64 1.174
• Closest Neighbor: Medium 7 132 -0.26 -0.85 1.272
– 9 Cases Fast 3 36 -0.84 -1.15 1.840
– 450 Ft Distance Slow 17 770 1.20 1.11 1.204
Medium 10 140 0.18 -0.83 1.117
• Prediction Slow 26 810 2.51 1.23 2.379
– Medium Delivery Medium 9 450 0.03 0.13 0.329
70 Time Medium 8 635 -0.11 0.69 700.602
eljalao@up.edu.ph
Fast 4 150 -0.69 -0.80 1.481
Delivery Time Data: 5 Nearest Neighbors
deltime ncases deldistance sncases sdeldistance eucdistance
Medium 7 560 -0.26 0.46 0.610
Fast 3 220 -0.84 -0.58 1.447
• New Order: Fast 3 340 -0.84 -0.21 1.262
Fast 4 80 -0.69 -1.01 1.644
– 11 Cases
Fast 6 150 -0.40 -0.80 1.299
– 500 ft Distance Medium 7 330 -0.26 -0.24 0.782
Fast 2 110 -0.98 -0.92 1.774
• Standardized Medium 7 210 -0.26 -0.61 1.064
– 0.33 SCases Slow 30 1460 3.09 3.23 4.042
Slow 5 605 -0.55 0.60 0.930
– 0.28 SDistance
Slow 16 688 1.05 0.86 0.929
• Closest Neighbors: Slow 10 215 0.18 -0.60 0.888
Fast 4 255 -0.69 -0.47 1.266
– 4 Medium Medium 6 462 -0.40 0.16 0.736
– 1 Slow Slow 9 448 0.03 0.12 0.332
Slow 10 776 0.18 1.13 0.861
• Prediction Fast 6 200 -0.40 -0.64 1.174
– Medium Delivery Medium 7 132 -0.26 -0.85 1.272
Fast 3 36 -0.84 -1.15 1.840
Time
Slow 17 770 1.20 1.11 1.204
Medium 10 140 0.18 -0.83 1.117
Slow 26 810 2.51 1.23 2.379
Medium 9 450 0.03 0.13 0.329
71 Medium 8 635 -0.11 0.69 710.602
eljalao@up.edu.ph
Fast 4 150 -0.69 -0.80 1.481

– Decision Tree
• Similarity
– ANN
– SVM
72 72
eljalao@up.edu.ph
The Perceptron
• Let 𝑥1 , 𝑥2 , … 𝑥𝑝 be numerical variables

B1
• Let a binary 𝑦 = 𝐺𝑟𝑒𝑒𝑛 𝑜𝑟 𝑅𝑒𝑑
𝑥2
• Given data
– Find a line that separates the data
𝛽𝑜 + 𝛽1 𝑥1 +. . +𝛽𝑝 𝑥𝑝 = 0
𝑜𝑟
𝛽𝑇 𝑥 = 0
𝑥1
𝐺𝑟𝑒𝑒𝑛 𝑖𝑓 𝛽𝑇 𝑥 < 0
• Predict 𝑦ො = ൝
𝑅𝑒𝑑 𝑖𝑓 𝛽𝑇 𝑥 > 0

73 73
eljalao@up.edu.ph
Perceptron in a Nutshell
Input
nodes Black box
Output
X1 b1 node
X2 b2  y
X3 b3 X0
𝑦ො = 𝑠𝑖𝑔𝑛(𝛽𝑇 𝑥)

74 74
eljalao@up.edu.ph
Perceptron Learning Algorithm
• Find weights 𝛽 using the Stochastic Gradient Descent

methodology
– Start with random 𝛽 values
– Update the 𝛽 values as follows
𝛽𝑘+1 = 𝛽𝑘 + 𝜆 𝑦𝑖 − 𝑦ො𝑖 𝑥𝑖
– 𝜆 is called the Step Size
• 𝜆 may decrease with the number of iterations
• Iterate through each row and adjust 𝛽 for each point
misclassified

75 75
eljalao@up.edu.ph
Simple Perceptron Induction Example
• Given a loan applicant’s business age as well as the

number of days delinquent in payments, will that
applicant default in a specific loan?
• Historical Data is shown as follows:
Business Age (Ave = Number of Days Delinquent (Ave Default
10, SD=3) = 50, SD=10) ?
13 years 70 days Yes
– Standardized
7 years Values: 70 days No
10 years 40 days No
sBUSAGE sDAYSDELQ DEFAULT

1 2 Yes
-1 2 No
76 76
0 eljalao@up.edu.ph
-1 No
• Graphically: • Algebraically:
1
𝑥= ,𝑦 = 1
2
−1
𝑥= , 𝑦 = −1
2
0
𝑥1 (𝐵𝑈𝑆𝐴𝐺𝐸) 𝑥= , 𝑦 = −1
−1
Default
Did Not
Default
𝑥2 (𝐷𝐴𝑌𝑆𝐷𝐸𝐿𝑄)
77 77
eljalao@up.edu.ph
• Graphically:
𝑥1 (𝐵𝑈𝑆𝐴𝐺𝐸) 𝑥1 − 0.8𝑥2 = 0
Default
Did Not
Default
78 78
eljalao@up.edu.ph
• Graphically:
𝑥1 (𝐵𝑈𝑆𝐴𝐺𝐸) 2𝑥1 + 1.2 𝑥2 = 0
Default
Did Not
Default
79 79
eljalao@up.edu.ph
• Graphically:
𝑥1 (𝐵𝑈𝑆𝐴𝐺𝐸) 3𝑥1 − 0.8 𝑥2 = 0
Default
Did Not
Default
80 80
eljalao@up.edu.ph
• Graphically:
𝑥1 (𝐵𝑈𝑆𝐴𝐺𝐸) 3𝑥1 + 0.2 𝑥2 = 0
Default
Did Not
Default
81 81
eljalao@up.edu.ph
Perceptron Induction Usage
• Final Equation of the line: 3𝑥1 + 0.2 𝑥2 = 0
BUSAGE DAYSDELQ DEFAULT

1 2 Yes 3 1 + 0.2 2 = 3.4 > 0
-1 2 No 3 −1 + 0.2 2 = −2.6 < 0
0 -1 No 3 0 + 0.2 −1 = −0.2 < 0

82 82
eljalao@up.edu.ph
Perceptron Induction Issues
• Separable data implies there is a plane that classifies

training data perfectly
• Algorithm converges for separable data, if 𝜆 is sufficiently
small
• If separable, many solutions, depends on the initial 𝑤
• If not separable, does not converge: Needs ANN

83 83
eljalao@up.edu.ph
Artificial Neural Networks
• Is a network of Perceptrons or Nodes that mimics a

biological network of Neurons in a brain
• An ANN attempts to recreate the computational mirror of
the biological neural network
• Each neuron takes many input signals, then based on an
internal weighting system, produces a single output signal
that's typically sent as input to another neuron.
• Finding the weights in an ANN constitute the bulk of the
time done in learning the ANN.

84 84
eljalao@up.edu.ph
General Structure of ANN
x1
x2
x3
y
Training ANN means learning
x4 the weights of the neurons
x5
Input Neuron i Outpu
I1 wi1
Input Hidden Output Activation
Layer Layer wi2
Layer I2
wi3
Si function Oi Oi
g(Si )
I3
threshold, t
85 85
eljalao@up.edu.ph
Choice of Hidden Layers
• Typically the number of hidden units is somewhere in the

range of 5 to 100, with the number increasing with the
number of inputs and number of training cases.
• Choice of the number of hidden layers is guided by
background knowledge and experimentation. (Friedman
et al.)
• Foundations of Deep Learning and Artificial Intelligence

86 86
eljalao@up.edu.ph
ANN ZIP code example
• A 16x16 8 bit greyscale image of numbers

87 87
eljalao@up.edu.ph
• Net-1: No hidden layer,

equivalent to multinomial
logistic regression.
• Net-2: One hidden layer, 12
hidden units fully
connected.
• Net-3: Two hidden layers
locally connected.
• Net-4: Two hidden layers,
locally connected with
weight sharing.
• Net-5: Two hidden layers,
locally connected, two
levels of weight sharing.

88 88
eljalao@up.edu.ph

89 89
eljalao@up.edu.ph
Business Scenario: Credit Scoring
• Credit scoring is the practice of analyzing a person’s

background and credit application in order to assess the
creditworthiness of the person
• The variables income (yearly), age, loan (size in euros)
and LTI(the loan to yearly income ratio) are available.
• The goal is to devise a model which predicts, whether or
not a default will occur within 10 years.
http://www.r-bloggers.com/using-neural-
networks-for-credit-scoring-a-simple-
example/
90 90
eljalao@up.edu.ph
• Predict whether the following applicants will default in

their loan payments within the next 10 years:
income age loan LTI default10yr

42710 46 6104 0.143 ?
66953 19 8770 0.131 ?
24904 57 15 0.001 ?

91 91
eljalao@up.edu.ph

92 92
eljalao@up.edu.ph

93 93
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
94 94
eljalao@up.edu.ph
Support Vector Machines
• A Support Vector Machine (SVM) performs classification

by finding a plane that maximizes the margin between
the two classes. The vectors (cases) that define the plane
are the support vectors.

95 95
eljalao@up.edu.ph
Recall: Limitations of Perceptrons
• Solution not unique if separable

• Data needs to be separable to converge
• Linear Model
• These are solved using Support Vector Machines (SVM)

96 96
eljalao@up.edu.ph
• Find a linear hyperplane (decision boundary) that will separate the data
97
eljalao@up.edu.ph
B1
• One Possible Solution

98
eljalao@up.edu.ph
B2
• Another possible solution

99
eljalao@up.edu.ph
B2
• Other possible solutions

100
eljalao@up.edu.ph
B1
B2
• Which one is better? B1 or B2?

• How do you define better?
101
eljalao@up.edu.ph
B1
B2
b21
b22
margin
b11
b12
• Find hyperplane maximizes the margin => B1 is better than B2

102
eljalao@up.edu.ph
B1
𝛽𝑇 𝑥 = 0
𝑇
𝛽 𝑥 + 𝑏 = −1 𝛽𝑇 𝑥 + 𝑏 = 1
2
𝑀𝑎𝑟𝑔𝑖𝑛 = 2
𝛽
b11
𝑇 b12
1 𝑖𝑓 𝛽 𝑥 ≥ 1
𝑓 𝑥 =൝
−1 𝑖𝑓 𝛽 𝑇 𝑥 ≤ −1
103 103
eljalao@up.edu.ph
2
• We want to maximize: 𝑀𝑎𝑟𝑔𝑖𝑛 = 𝛽 2
𝛽 2
– Which is equivalent to minimizing: 𝑍 𝛽 =
2
– But subjected to the following constraint:
𝑦𝑖 𝛽 𝑇 𝑥𝑖 + 𝑏 ≥ 1, ∀ 𝑦𝑖 , 𝑥𝑖 , 𝑖 = 1,2. . . 𝑛
• If 𝑦𝑖 = 1, prediction 𝛽 𝑇 𝑥𝑖 + 𝑏 = 1 and if 𝑦𝑖 = −1 then 𝛽 𝑇 𝑥𝑖 + 𝑏 = −1
– This is a constrained optimization problem

• Numerical approaches to solve it (e.g., quadratic programming)

104 104
eljalao@up.edu.ph
Limitations of Perceptrons

• Linear Model

105 105
eljalao@up.edu.ph
• What if the problem is not linearly separable?
ξi
ξi

106 106
eljalao@up.edu.ph
• What if the problem is not linearly separable?

– Introduce slack variables
• Need to minimize:
𝛽 2 𝑘
𝑀𝑖𝑛 𝑍 𝛽 = + 𝜆 σ𝑁 𝜉
𝑖=1 𝑖
2
Subject to:
𝑦𝑖 𝛽𝑇 𝑥𝑖 + 𝑏 ≥ 1 − 𝜉𝑖 , ∀ 𝑦𝑖 , 𝑥𝑖 , 𝑖 = 1,2. . . 𝑛

107 107
eljalao@up.edu.ph

108 108
eljalao@up.edu.ph
The Tuning/Complexity Parameter 𝜆
• If 𝜆 is Large: Focus on Minimizing Errors

109 109
eljalao@up.edu.ph
The Tuning/Complexity Parameter 𝜆
• If 𝜆 is Small: Focus on Maximizing Margin

110 110
eljalao@up.edu.ph
Limitations of Perceptrons

• Linear Model

111 111
eljalao@up.edu.ph
Nonlinear Support Vector Machines
• What if decision boundary is not linear?

112 112
eljalao@up.edu.ph
Nonlinear Support Vector Machines
• Transform data into higher dimensional space

113 113
eljalao@up.edu.ph
Non-linear SVMs: Feature spaces
• General idea: the original feature space can always be

mapped to some higher-dimensional feature space where
the training set is separable:
Φ: 𝑥 → 𝜑(𝑥)

114 114
eljalao@up.edu.ph
Kernels
• Kernel methods transform the training data into higher

dimensional spaces such that the data can be more easily
separated
• Kernel Trick: A complex function can be approximated by a
Kernel:
– E.g. 𝑓 𝑥1 , 𝑥2 = 1 + 2𝑥1 + 2𝑥2 + 𝑥12 + 𝑥22 + 2𝑥1 𝑥2
– Kernel: 𝑘 𝑥1 , 𝑥2 = 1 + 𝑥1 𝑥2𝑡 2

115 115
eljalao@up.edu.ph
• We will be using the credit scoring dataset again.
income age loan LTI default10yr

42710 46 6104 0.143 ?
66953 19 8770 0.131 ?
24904 57 15 0.001 ?
http://www.r-bloggers.com/using-neural-
networks-for-credit-scoring-a-simple-
example/
116 116
eljalao@up.edu.ph

117 117
eljalao@up.edu.ph

118 118
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
119 119
eljalao@up.edu.ph
Ensemble Methods
• Construct a set of classifiers from the training data

• Predict class label of previously unseen records by
aggregating predictions made by multiple classifiers
• Wisdom of the Crowd
• Types of Ensembles
– Parallel Ensemble
– Serial Ensemble

120 120
eljalao@up.edu.ph
Parallel Ensembles
• Parallel: Combines approx. independent, diverse base

learners
• Different learners should make different errors
• Ensemble can outperform any one of its components
• Bagging, Random Forest (RF) examples

121 121
eljalao@up.edu.ph
General Idea
Original
D Training data
Step 1:
Create Multiple D1 D2 .... Dt-1 Dt
Data Sets
Step 2:
Build Multiple C1 C2 Ct -1 Ct
Classifiers
Step 3:
Combine C*
Classifiers
122 122
eljalao@up.edu.ph
Generation of Datasets: Bagging
• Sampling with replacement

Original Data 1 2 3 4 5 6 7 8 9 10
Bagging (Round 1) 7 8 10 8 2 5 10 10 5 9
Bagging (Round 2) 1 4 9 1 2 3 2 7 3 2
Bagging (Round 3) 1 8 5 10 5 5 9 6 3 7
• Build classifier on each bootstrap sample

1 𝑛
• Each sample has probability 1 − of NOT being
𝑛
selected
1
• ≈ is not inside the bag: OOB
3

123 123
eljalao@up.edu.ph
Bagging
• Example Initial Data

𝑥 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
𝑦 1 1 1 -1 -1 -1 -1 1 1 1
• Round 1: Random Data Selected

𝑥 0.1 0.2 0.2 0.3 0.4 0.4 0.5 0.6 0.9 0.9
𝑦 1 1 1 1 -1 -1 -1 -1 1 1
• Round 1 Optimal Split

– if 𝑥 ≤ 0.35 → 𝑦 = 1, 𝑥 > 0.35 → 𝑦 = −1
𝑥 0.1 0.2 0.2 0.3 0.4 0.4 0.5 0.6 0.9 0.9
𝑦ො 1 1 1 1 -1 -1 -1 -1 -1 -1

124 124
eljalao@up.edu.ph
Bagging

𝑥 0.2 0.4 0.5 0.6 0.7 0.7 0.7 0.8 0.9 0.9
𝑦 1 -1 -1 -1 -1 -1 -1 1 1 1
• Round 2 Optimal Split:

– if 𝑥 ≤ 0.75 → 𝑦 = −1, 𝑥 > 0.75 → 𝑦 = 1
𝑥 0.2 0.4 0.5 0.6 0.7 0.7 0.7 0.8 0.9 0.9
𝑦ො -1 -1 -1 -1 -1 -1 -1 1 1 1

125 125
eljalao@up.edu.ph
Bagging

𝑥 0.1 0.1 0.1 0.3 0.3 0.4 0.8 0.9 0.9 1
𝑦 1 1 1 1 1 -1 1 1 1 1
• Round 3 Optimal Split:

𝑦= 1
𝑥 0.1 0.1 0.1 0.3 0.3 0.4 0.8 0.9 0.9 1

𝑦ො 1 1 1 1 1 1 1 1 1 1

126 126
eljalao@up.edu.ph
Bagging Ensemble
• Ensemble Model
• If 𝑥 ≤ 0.35 → 𝑦 = 1, 𝑥 > 0.35 → 𝑦 = −1
• If 𝑥 ≤ 0.75 → 𝑦 = −1, 𝑥 > 0.75 → 𝑦 = 1
• 𝑦= 1
Round 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
1 1 1 1 -1 -1 -1 -1 -1 -1 -1
2 -1 -1 -1 -1 -1 -1 -1 1 1 1
3 1 1 1 1 1 1 1 1 1 1
Majority 1 1 1 -1 -1 -1 -1 1 1 1
True 1 1 1 -1 -1 -1 -1 1 1 1

127 127
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
128 128
eljalao@up.edu.ph
Serial Ensemble
• Serial: New learner uses previously built learners

– Iteratively reweight the training data to focus on
previous errors—known as boosting
– Combine as linear combination –Adaboost Algorithm
(1996)
– Dramatic increase in accuracy with even weak base
learners—can reduce bias and variance
– Initially, all 𝑛 records are assigned equal weights
– Unlike bagging, weights may change at the end of
boosting round

129 129
eljalao@up.edu.ph
Boosting
• Records that are wrongly classified will have their weights

increased
• Records that are classified correctly will have their weights
decreased
Original Data 1 2 3 4 5 6 7 8 9 10
Boosting (Round 1) 7 3 2 8 7 9 4 10 6 3
Boosting (Round 2) 5 4 9 4 2 5 1 7 4 2
Boosting (Round 3) 4 4 8 10 4 5 4 6 3 4
• Example 4 is hard to classify

• Its weight is increased, therefore it is more
likely to be chosen again in subsequent rounds
130 E.R. L. Jalao, Copyright UP NEC, 130
eljalao@up.edu.ph
Adaboost - Adaptive Boosting
• Instead of resampling, uses training set re-weighting

– Each training sample uses a weight to determine the probability
of being selected for a training set.
• AdaBoost is an algorithm for constructing a “strong”
classifier as linear combination of “simple” “weak”
classifier
• Final classification based on weighted vote of weak

classifiers
131

132 132
eljalao@up.edu.ph

133 133
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
134 134
eljalao@up.edu.ph
Random Forests
• The random forest (Breiman, 2001) is an ensemble of

decision trees
• Trees are combined by average (regression) or voting
(classification)
• Tree provides a class probability estimates so that
weighted votes can be used

135 135
eljalao@up.edu.ph
Random Forests

136 136
eljalao@up.edu.ph
Random Forests Algorithm
• For some number of trees 𝑇

• Sample 𝑁 cases at random with replacement to create a
subset of the data at each node:
– For some number 𝑝, 𝑝 predictor variables are selected at random
from all the predictor variables.
– The predictor variable that provides the best split, according to
some objective function, is used to do a binary split on that node.
– At the next node, choose another 𝑝 variables at random from all
predictor variables and do the same.

137 137
eljalao@up.edu.ph
Random Forests
• Selecting 𝑝 variables randomly reduces correlation

between trees (variance)
• Brieman suggests three possible values for m: ½ 𝑝, 𝑝,
and 2 𝑝

138 138
eljalao@up.edu.ph
Business Scenario: Census Income
• Predict whether a person’s income exceeds $50K/yr based

on various demographic data like: age, workclass,
education, marital status, occupation, relationship, race,
sex, capital gains, hours per week and country.
• Sample Data:
workcla finalwei educati educati marital- occupat relation capitalg capitallos hoursp
age ss ght on on-num status ion ship race sex ain s erweek country Income
Asian-
Pac-
Some- Never- Craft- Own- Island
27 Private 116358 college 10 married repair child er Male 0 1980 40 Philippines ?
Asian-
Pac-
State- Some- Never- Adm- Unmarri Island
29 gov 71592 college 10 married clerical ed er Female 0 0 40 Philippines ?
Married
Bachelo -civ- Adm- Other-
32 Private 270335 rs 13 spouse clerical relative White Male 0 0 40 Philippines ?
139 139
eljalao@up.edu.ph
Business Scenario: Income Data

140 140
eljalao@up.edu.ph
Business Scenario: Income Data

141 141
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
142 142
eljalao@up.edu.ph
Comparison of Alternatives
Parameter ZeroR OneR Decision Trees Rule-Based Naïve Bayes
Classifiers
Complexity Simple Simple Simple Simple Simple
Processing N/A Strong Strong Strong Strong
Categorical
Data
Processing N/A Weak Weak Weak Weak
Numerical
Data
Speed of Fast Fast Fast Fast Fast
Algorithm
Handling N/A Good Good Good Good
Outliers/Noise
Handling N/A Can run with Can run with Can run with Can run with
Missing Data incomplete incomplete incomplete incomplete
data data data data
Interpretability High High High High High
Relative Low Low Medium Medium Low

Predictive
143 Power 143
eljalao@up.edu.ph
Comparison of Alternatives
Parameter kNN Neural Support Random Adaboost
Networks Vector Forests
Machines
Complexity Simple Complex Complex Complex Complex
Processing Categorical Weak Weak Weak Strong Depends on
Data the algorithm
Processing Numerical Strong Strong Strong Weak Depends on

Data the algorithm
Speed of Algorithm N/A Slow Slow Fast Fast
Handling Outliers/Noise Weak Good Good Good Depends on

the algorithm
Handling Missing Data Needs Needs Needs Can run with Depends on
Complete Complete Complete incomplete the algorithm
Data Data Data data
Interpretability High Blackbox Blackbox Blackbox Blackbox
Relative Predictive Power Medium High High High High

144 144
eljalao@up.edu.ph
Model Evaluation
• How to evaluate the performance of a model?

– Metrics for Performance Evaluation
• How to obtain reliable estimates?
– Methods for Performance Evaluation
– Overfitting and Underfitting
• How to compare the relative performance among
competing models?
– Methods for Model Comparison

145 145
eljalao@up.edu.ph
Model Evaluation
• Model Evaluation is a methodology that helps to find the

best model that represents our data and how well the
chosen model will work in the future.
Generate
Training Classification
Data Model
Use Model to How to

New Data Predict New Estimate
Rows Performance?

146 146
eljalao@up.edu.ph
Definition of Terms
• Error:
𝑒𝑟𝑟𝑜𝑟 = 1 𝑖𝑓 𝑦ො ≠ 𝑦
– If predicted value of the classifier model is not equal to the actual
value, then we define that as an error.
• We would like to minimize error
• Given a binary classification model, how do we count the
error predictions?

147 147
eljalao@up.edu.ph
Metrics for Performance Evaluation
• Confusion Matrix: A way to tabulate errors
PREDICTED CLASS
Class=Yes Class=No
ACTUAL Class=Yes a b
CLASS Class=No c d
a: TP (true positive)
b: FN (false negative)
c: FP (false positive)
d: TN (true negative)

148 148
eljalao@up.edu.ph
Metrics for Performance Evaluation
PREDICTED CLASS
Class=Yes Class=No
Class=Yes a b
ACTUAL (TP) (FN)
CLASS
Class=No c d
(FP) (TN)
• Most widely-used metric:

𝑎+𝑑 𝑇𝑃 + 𝑇𝑁
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = =
𝑎 + 𝑏 + 𝑐 + 𝑑 𝑇𝑃 + 𝐹𝑁 + 𝐹𝑃 + 𝑇𝑁
149 149
eljalao@up.edu.ph
Re-Substitution Errors
• Re-substitution errors: error on training data
Generate
All Data Classification
(600 rows) Model
All Data
(600 rows)
Use Model to
All Data
Measure
Predict New
(600 rows) Performance
Rows

150 150
eljalao@up.edu.ph

151 151
eljalao@up.edu.ph

152 152
eljalao@up.edu.ph
Generalization Errors
• Generalization errors: error on unseen data
Generate
Train Data Classification
(400 Rows) Model
All Data
(600 rows)
Use Model to
Test Data
Measure
Predict New
(200 Rows) Performance
Rows

153 153
eljalao@up.edu.ph

154 154
eljalao@up.edu.ph

155 155
eljalao@up.edu.ph
Model Evaluation

competing models?

156 156
eljalao@up.edu.ph
Methods for Performance Evaluation
• Disclaimer: Performance of a model may depend on other

factors besides the learning algorithm:
– Class distribution
– Cost of misclassification
– Size of training and test sets

157 157
eljalao@up.edu.ph
Methods of Estimation
• Holdout
– Reserve x% for training and 100-x% for testing
• Cross validation
– Estimates the performance of the Model generated using all data.
– Partition data into k disjoint subsets
– k-fold: train on 𝑘 − 1 partitions, test on the remaining one
– If 𝑘 = 10, 11 Decision Tree Models will be created. One for each
fold, and 1 using all data.

158 158
eljalao@up.edu.ph
Cross Validation
Round 1 Round 2 Round 3 Round 10
...
92% 95% 85% 89%

Overall Accuracy = Ave(Round 1, Round 2...)
Where:
Training Data Testing Data

159 159
eljalao@up.edu.ph
Cross Validation

160 160
eljalao@up.edu.ph
Cross Validation

161 161
eljalao@up.edu.ph
Underfitting and Overfitting
• Underfit
– Simple Model
– Lots of Errors
– Stable Prediction

162 162
eljalao@up.edu.ph
• Overfit
– Complex Model
– No Errors
– Unstable Prediction
– Predicts Even Noise

163 163
eljalao@up.edu.ph
• Just Right
– Small Errors
– Stable Predictions

164 164
eljalao@up.edu.ph
Overfitting due to Noise
Decision boundary is distorted by noise point

165
eljalao@up.edu.ph
Overfitting
Underfitting: when model is too simple, both training and test errors are large
166
eljalao@up.edu.ph
Notes on Overfitting
• Overfitting results in classification models that are more

complex than necessary
• Training error no longer provides a good estimate of how
well the tree will perform on previously unseen records
• Need new ways for estimating errors

167 167
eljalao@up.edu.ph
How to Address Overfitting
• Post-pruning
– Grow model to its entirety
– Trim the complexity of the model
– If generalization error improves after trimming, replace the
complex model with the less complex model

168 168
eljalao@up.edu.ph
Addressing Overfitting and Underfitting
• Fitting a complex model:

169 169
eljalao@up.edu.ph
Overfitting Model
• Complexity:
• Training Error:
• Testing Error

170 170
eljalao@up.edu.ph
• Fitting a underfit model:

171 171
eljalao@up.edu.ph
Underfit Model
• Complexity:
• Training Error:
• Testing Error

172 172
eljalao@up.edu.ph
• Fitting a just right model:

173 173
eljalao@up.edu.ph
Just Right Model
• Complexity:
• Training Error:
• Testing Error:

174 174
eljalao@up.edu.ph
Model Summaries
Type of Model Training Error Testing Error

Underfit Model 16.5% 19.67%
Just Right Model 7.67% 10.16%
Overfit Model 4.67% 13.50%

175 175
eljalao@up.edu.ph
Model Evaluation

competing models?

176 176
eljalao@up.edu.ph
Limitation of Accuracy
• Consider a 2-class problem

– Number of Class “–” examples = 9990
– Number of Class “+” examples = 10
• If model predicts everything to be class “–”, accuracy is

9990/10000 = 99.9 %
– Accuracy is misleading because model does not detect any class
“+” example

177 177
eljalao@up.edu.ph
Cost Matrix
PREDICTED CLASS
C(i|j) Class=Yes Class=No
Class=Yes C(Yes|Yes) C(No|Yes)

ACTUAL
CLASS Class=No C(Yes|No) C(No|No)
C(i|j): Cost of misclassifying class j example as class i

178
eljalao@up.edu.ph
Computing Cost of Classification
Cost PREDICTED CLASS
Matrix
C(i|j) + -
ACTUAL
+ -1 100
CLASS
- 1 0
Model PREDICTED CLASS Model PREDICTED CLASS
M1 M2
+ - + -
ACTUAL ACTUAL
+ 255 40 + 250 45
CLASS CLASS
- 60 145 - 5 200
Accuracy = 80% Accuracy = 90%
Cost = 3805 Cost = 4255
179
eljalao@up.edu.ph
Computing Cost of Classification
• With Cost
• Result:

180 180
eljalao@up.edu.ph
Cost-Sensitive Measures
𝑇𝑃
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃 + 𝐹𝑃
𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃 + 𝐹𝑁
2𝑇𝑃
𝐹 𝑀𝑒𝑎𝑠𝑢𝑟𝑒 =
2𝑇𝑃 + 𝐹𝑁 + 𝐹𝑃
• Precision is biased towards C(Yes|Yes) & C(Yes|No)

• Recall is biased towards C(Yes|Yes) & C(No|Yes)
• F-measure is biased towards all except C(No|No)

181 181
eljalao@up.edu.ph
Breast Cancer Recurrence Example
• The Breast Cancer dataset contains 9 variables describing

286 women that have suffered and survived breast cancer
and whether or not the disease recurred within 5 years.
• 286 women: 201 did not suffer a recurrence + 85 suffered
recurrence.
• We’ll test three models:
– A model that predicts that all women will not have breast cancer
again
– A model that predicts that all women will have breast cancer
again
– Decision Tree Model

182 182
eljalao@up.edu.ph
Model 1
• Predicts that all women will not have breast cancer again
Predict Recurrence Predict No Recurrence

Recurrence 0 85
No Recurrence 0 201
201
• 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = = 70.3%
286
0 85 women will incorrectly
• 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = =0 think their breast cancer
0+0
was not going to reoccur but
0
• 𝑅𝑒𝑐𝑎𝑙𝑙 = =0 will reoccur
0+85
2∗0
• 𝐹 𝑀𝑒𝑎𝑠𝑢𝑟𝑒 = = 0
2∗0+0+85
183 183
eljalao@up.edu.ph
Model 2
• Predicts that all women will have breast cancer again

Recurrence 85 0
No Recurrence 201 0
85
286
85 201 women will incorrectly
• 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = = 29.7% think their breast cancer is
85+201
going to reoccur but will not
85
• 𝑅𝑒𝑐𝑎𝑙𝑙 = = 100% reoccur
0+85
2∗85
• 𝐹 𝑀𝑒𝑎𝑠𝑢𝑟𝑒 = = 45.8
2∗85+201+0
184 184
eljalao@up.edu.ph
Model 3
• A J48 Model with the following confusion matrix

Recurrence 10 75
No Recurrence 13 188
198
286
10
• 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = = 43%
10+13
10
• 𝑅𝑒𝑐𝑎𝑙𝑙 = = 12%
10+75
2∗10
• 𝐹 𝑀𝑒𝑎𝑠𝑢𝑟𝑒 = = 18.15%
2∗10+13+75
185 185
eljalao@up.edu.ph
Model Summaries
Model 1 Model 2 Model 3

Accuracy 70.3% 29.7% 69.2%
Precision 0% 29.7% 43%
Recall 0% 100% 12%
F-Measure 0% 45.8% 18.15%

186 186
eljalao@up.edu.ph
ROC (Receiver Operating Characteristic)
• Developed in 1950s for signal detection theory to analyze

noisy signals
– Characterize the trade-off between positive hits and false alarms
• ROC curve plots TP (on the y-axis) against FP (on the x-
axis)
• Performance of each classifier represented as a curve with
varying TP and FP performance

187 187
eljalao@up.edu.ph
ROC Curve
• (TP,FP):
• (0,0): declare everything
to be negative class
• (1,1): declare everything
to be positive class
• (1,0): ideal
• Diagonal line:
– Random guessing
– Below diagonal line:
• prediction is opposite of the true class

188 188
eljalao@up.edu.ph
http://www.datasciencecentral.com/profiles/
blogs/how-to-assess-quality-and-
correctness-of-classification-models

189 189
eljalao@up.edu.ph
Using ROC for Model Comparison
 No model consistently
outperform the other
 M1 is better for
small FPR
 M2 is better for
large FPR
 Area Under the ROC

curve
 Ideal:
 Area =1
 Random guess:
 Area = 0.5
190
eljalao@up.edu.ph
Practical Application of ROCs
• Predicting Churn
– Must have high true positive rate = very low false negative rate
• Must be able to capture all Churn Customers
– Ok to have high false positive rate
• Predicting that you will Churn but actually will Not Churn
– Choose Model 2
• Predicting Flu
– Must have low false positive rate
• Predicting You Have Flu but actually have no Flu
– Must have an acceptable true positive rate
• Must be able to detect Flu at least 60% of the time
– Choose Model 1

191
eljalao@up.edu.ph
Using ROC Curves

192 192
eljalao@up.edu.ph
Using ROC Curves

193 193
eljalao@up.edu.ph
Using ROC Curves

194 194
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
195 195
eljalao@up.edu.ph
Case 2: Classification Using R
• Selecting the Best Classification Model for the Churn Data

Using R

196 196
eljalao@up.edu.ph
Algorithm Cheat Sheet
http://scikit-
learn.org/stable/tutorial/machine_learning_map/index.html
197 197
eljalao@up.edu.ph
Algorithm Cheat Sheet
http://www.datasciencecentral.com/profiles/blogs/key-tools-of-big-
data-for-transformation-review-case-study
198 198
eljalao@up.edu.ph

– Decision Tree
• Similarity
– ANN
– SVM
199 199
eljalao@up.edu.ph
References
• Data Mining Overview: http://www.saedsayad.com/

• Tan et al. Intro to Data Mining Notes
• Runger, G. IEE 520 notes
• http://axon.cs.byu.edu/Dan/678/miscellaneous/SVM.exa
mple.pdf
• G. Runger, ASU IEE 578

200 200
eljalao@up.edu.ph

4.0 Classification Methodologies R

Uploaded by

Copyright:

Available Formats

4.0 Classification Methodologies R

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

4.0 Classification Methodologies R

Uploaded by

Copyright:

Available Formats

4.

Eugene Rex L. Jalao, Ph.D.

• What is Classification? • Ensembles

• Define supervised learning and classification

E.R. L. Jalao, Copyright UP NEC,

Tid Refund Marital Taxable

1 Yes Single 125K No

• Given a collection of records

E.R. L. Jalao, Copyright UP NEC,

Model Model 1 Model

E.R. L. Jalao, Copyright UP NEC,

E.R. L. Jalao, Copyright UP NEC,

• What is Classification? • Ensembles

• ZeroR is the simplest classification methodology which

• For the Golf Data Set: Majority is Yes (9 Yes, 5 No)

E.R. L. Jalao, Copyright UP NEC,

For the Golf Data Set: Majority is Yes (9 Yes, 5 No)

• What is Classification? • Ensembles

• OneR, short for "One Rule"

E.R. L. Jalao, Copyright UP NEC,

Actual Play Golf Actual Play Golf

E.R. L. Jalao, Copyright UP NEC,

Actual Play Golf

• IF Humidity = High THEN Play Golf = No

Actual Play Golf

IF Humidity = High THEN Play Golf = No

• Performance of the Classifier

Actual Play Golf

E.R. L. Jalao, Copyright UP NEC,

• Level of Confidence we get for each prediction rule

E.R. L. Jalao, Copyright UP NEC,

• What is Classification? • Ensembles

• Uses a probabilistic framework for classification

E.R. L. Jalao, Copyright UP NEC,

• Given Historical Data:

• Given a record with 𝑝 attributes (𝐴1 , 𝐴2 , … , 𝐴𝑝 )

E.R. L. Jalao, Copyright UP NEC,

• Using Bayes Formula:

𝐏 𝐏𝐥𝐚𝐲 = 𝒀𝒆𝒔 𝑨𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑨𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)=

𝐏 𝐏𝐥𝐚𝐲 = 𝑵𝒐 𝑨𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑨𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)=

E.R. L. Jalao, Copyright UP NEC,

E.R. L. Jalao, Copyright UP NEC,

• Given today’s Outlook = Rainy, Temperature = Hot, Humidity =

– 𝑷 𝑷𝒍𝒂𝒚 = 𝒀𝒆𝒔 𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)

– 𝑷 𝑷𝒍𝒂𝒚 = 𝑵𝒐 𝑶𝒖𝒕𝒍𝒐𝒐𝒌 = 𝑹𝒂𝒊𝒏𝒚, 𝑻𝒆𝒎𝒑 = 𝑯𝒐𝒕)

= 𝑃 𝑂𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑅𝑎𝑖𝑛𝑦 𝑃𝑙𝑎𝑦 = 𝑁𝑜 ∗ 𝑃 𝑇𝑒𝑚𝑝 = 𝐻𝑜𝑡 𝑃𝑙𝑎𝑦 = 𝑁𝑜

E.R. L. Jalao, Copyright UP NEC,

• If one of the conditional probability is zero, then the entire

E.R. L. Jalao, Copyright UP NEC,

• The Nursery Data set originally developed to rank

parents has_nur form children housing finance social health rank

E.R. L. Jalao, Copyright UP NEC,

• Type the following lines of code in RStudio and run.

E.R. L. Jalao, Copyright UP NEC,

E.R. L. Jalao, Copyright UP NEC,

• What is Classification? • Ensembles

• Decision tree builds classification models in the form of a

E.R. L. Jalao, Copyright UP NEC,

• The core algorithm for building decision trees called ID3

E.R. L. Jalao, Copyright UP NEC,