Classification using

Decision Tree

CSE-454: Data Warehousing and

Data Mining Sessional

MD. JAKARIA
LECTURER
DEPT. OF CSE, MIST
 Classification
Definition: In machine learning, classification refers to a
predictive modeling problem where a class label is predicted for a
given example of input data.
Examples of classification problems include:
• Given an example, classify if it is spam or not.
• Given a handwritten character, classify it as one of the known characters.
• Given recent user behavior, classify as churn or not.

There are perhaps four main types of classification tasks that you may
encounter; they are:
• Binary Classification
• Multi-Class Classification
• Multi-Label Classification
• Imbalanced Classification

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 2
MINING SESSIONAL
 Decision Tree Classification
Definition: A decision tree represents a function that takes as input
a vector of attribute values and returns a “decision” — a single
output value.
As an example, we will build a decision tree to decide whether to
wait for a table at a restaurant. The list of attributes that we will
consider:
1. Alternate 6. Price
2. Bar 7. Raining
3. Fri/Sat 8. Reservation
4. Hungry 9. Type
5. Patrons 10.Wait Estimate

Note that every variable has a small set of possible values.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 3
MINING SESSIONAL
 Decision Tree Classification

Fig: A decision tree for deciding whether to wait for a table.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 4
MINING SESSIONAL
 Decision Tree Classification
Decision tree -
• A flow-chart-like tree structure
• Internal node denotes a test on an attribute
• Branch represents an outcome of the test
• Leaf nodes represent class labels or class distribution

Use of decision tree:

• Classifying an unknown sample
• Test the attribute values of the sample against the decision tree

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 5
MINING SESSIONAL
 Expressiveness of decision trees
A Boolean decision tree is logically equivalent to the assertion that
the goal attribute is true if and only if the input attributes satisfy
one of the paths leading to a leaf with value true.
Goal ⇔ (Path1 ∨ Path2 ∨ ···)
where each Path is a conjunction of attribute-value tests required
to follow that path.
Path = (Patrons = Full ∧ WaitEstimate = 0–10)

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 6
MINING SESSIONAL
 Constructing decision trees

Table: Examples for the restaurant domain.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 7
MINING SESSIONAL
 Constructing decision trees
  • We want a tree that is consistent with the examples and is as small as
possible
• It is an intractable problem to find the smallest consistent tree; there
is no way to efficiently search through the trees
• With some simple heuristics, we can find a good approximate
solution: a small (but not smallest) consistent tree
• The DECISION-TREE-LEARNING algorithm adopts a greedy
divide-and-conquer strategy: always test the most important attribute
first.
• By “most important attribute,” we mean the one that makes the most
difference to the classification of an example.
o Type is a poor attribute
o Patrons is a fairly important attribute

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 8
MINING SESSIONAL
 Constructing decision trees

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 9
MINING SESSIONAL
 Constructing decision trees
Basic steps (a greedy approach)
• Tree is constructed in a top-down recursive divide-and-conquer
manner
• At start, all the training examples are at the root
• Attributes are categorical (if continuous-valued, they can be
discretized in advance)
• Examples are partitioned recursively based on selected attributes.
• Test attributes are selected on the basis of a heuristic or statistical
measure (e.g., information gain)

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 10
MINING SESSIONAL
 Constructing decision trees
Conditions for stopping partitioning
• All samples for a given node belong to the same class
• There are no remaining attributes for further partitioning – majority
voting is employed for classifying the leaf
• There are no samples left

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 11
MINING SESSIONAL
 The DECISION-TREE-LEARNING algorithm
There are four cases to consider
i. If the remaining examples are all positive (or all negative), we can
answer Yes or No.
ii. If there are some positive and some negative examples, then
choose the best attribute to split them.
iii. If there are no examples left, we return a default value calculated
from the plurality classification of parent node.
iv. If there are no attributes left, but both positive and negative
examples, return the plurality classification of the remaining
examples.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 12
MINING SESSIONAL
 The DECISION-TREE-LEARNING algorithm
function DECISION-TREE-LEARNING (examples, attributes, parent_examples) returns
a tree

 if examples is empty then return PLURALITY-VALUE(parent_examples)

 else if all examples have the same classification then return the classification
 else if attributes is empty then return PLURALITY-VALUE(examples)
 else
   A ← argmaxa ∈ attributes IMPORTANCE(a, examples)
   tree ← a new decision tree with root test A
   for each value vk of A do
     exs ← { e : e ∈ examples and e.A = vk }
     subtree ← DECISION-TREE-LEARNING(exs, attributes − A, examples)
     add a branch to tree with label (A = vk) and subtree subtree
   return tree

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 13
MINING SESSIONAL
 Choosing attribute tests
• The goal is to have the resulting decision tree as small as possible
(Occam’s Razor)
- But, finding the minimal decision tree consistent with the data is NP-hard
• The recursive algorithm is a greedy heuristic search for a simple tree,
but cannot guarantee optimality.
• The main decision in the algorithm is the selection of the next
attribute to condition on.

• Which is the best attribute?

— The one which will result in the smallest tree
— Heuristic: choose the attribute that produces the “purest” nodes
• Need a good measure of purity!
— Maximal when?
— Minimal when?

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 14
MINING SESSIONAL
 Impurity/Entropy (informal)
Measures the level of impurity in a group of examples

Very Impure Group Less Impure Group Minimum Impure Group

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 15
MINING SESSIONAL
 Impurity/Entropy (informal)
 Measures the level of impurity in a group of examples

• Here, pi is the probability of class i.

• Compute it as the proportion of class i in the set.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 16
MINING SESSIONAL
 Impurity/Entropy (informal)
 Measures the level of impurity in a group of examples

• Here, pi is the probability of class i.

• Compute it as the proportion of class i in the set.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 17
MINING SESSIONAL
 Impurity/Entropy (informal)
 Measures the level of impurity in a group of examples

• Here, pi is the probability of class i.

• Compute it as the proportion of class i in the set.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 18
MINING SESSIONAL
 Information Gain
• We want to determine which attribute in a given set of training
feature vectors is most useful for discriminating between the classes
to be learned.

• Information gain tells us how important a given attribute of the

feature vectors is.

• We will use it to decide the ordering of attributes in the nodes of a

decision tree.

𝐼𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛
  𝐺𝑎𝑖𝑛=𝐸𝑛𝑡𝑟𝑜 𝑝𝑦 ( 𝑝𝑎𝑟𝑒𝑛𝑡 ) – [ 𝐴 𝑣𝑒𝑟𝑎𝑔𝑒 𝐸 𝑛𝑡𝑟𝑜𝑝𝑦 (𝑐h𝑖𝑙𝑑𝑟𝑒𝑛)]

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 19
MINING SESSIONAL
 Information Gain

Entropy =
Entropy =

Entropy =

(Weighted) Average Entropy of Children =

 
= 0.996 - 0.615 = 0.38

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 20
MINING SESSIONAL
 Information Gain
Which test is more informative?

Less or equal 50K Over 50k Unemployed Employed

Split over whether Balance exceeds 50K Split over whether applicant is employed

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 21
MINING SESSIONAL
 The DECISION-TREE-LEARNING algorithm

Fig: The decision tree induced from the 12-example training set.

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 22
MINING SESSIONAL
 References:
i. Artificial Intelligence: A Modern Approach by Peter Norvig and Stuart J. Russell
Chapter 18 (18.3 Learning Decision Tree)

CSE 454: DATA WAREHOUSING AND DATA

LEC MD. JAKARIA 23
MINING SESSIONAL