Decision Tree Learning – CART &

CHAID

U DINESH KUMAR
 Classification/Decision Trees
• Introduction to Decision Trees.

• Classification/Decision Tree approach for classification.

• Classification and Regression Tree (CART)

• Chi-Square Automatic Interaction Detection (CHAID)

 Classification/Decision Trees
• Decision trees (aka decision tree learning) are collection of
predictive analytics techniques that uses tree like graph for
predicting the value of response variable (or target variable)
based values of explanatory variables (or predictors).

• In a decision tree when the response variable takes discrete

values then they are called classification trees.
 Classification Trees – Generic Steps
• Start with root node in which all the data is present.

• The root node is split into two or more branches (called edges)
using split criteria resulting in internal nodes. Each internal
node will have exactly one incoming edge.

• Each internal node is further divided until no further splitting is

possible. The terminal nodes (aka leaf nodes) will not have any
outgoing edges.

• Terminal nodes are used for generating business rules.

• Tree pruning (aka stopping criteria a process of restricting the size

of the tree) is used to avoid large trees and overfitting the data.
Classification and Regression Trees
(CART)
 Classification and Regression Trees
(CART)
• Splits are chosen based either on Gini Index or
Entropy criteria.

• CART is a binary tree, whereas CHAID can split

the initial node into more than 2 branches.
Gini Index (Classification Impurity)
• Gini Index is used to measure the impurity at a
node (in classification problem) and is given
by:
J
Gini(k)   P( j k )(1  P( j k ))
j 1
where P(j k) is the proportion of category j in node k

Smaller Gini Index implies less impurity.

 Entropy (Impurity Measure)
• Entropy is another impurity measure that is
frequently used. Entropy at node “k” is given
by:
c
Entropy(k)    P(j | k)log(P(j | k))
j1
 Gini Index Calculation
Number of classes 2 (say 0 and 1)

Consider node label k with 10 1s and 90 0s.

2
Gini(k)   P( j k )(1  P( j k ))  2 P( j k )(1  P( j k ))
j 1

Gini(k)  2  0.1 0.9  0.18

Smaller number
implies less
impurity
 Entropy Calculation
Number of classes 2 (say 0 and 1)

Consider node label k with 10 1s and 90 0s.

2
Entropy(k)    P( j k ) log( P( j k ))
j 1

Entropy(k)  0.1 log(0.1)  0.9  log(0.9)  0.4689

Higher than Gini

coefficient
 Classification Tree Logic
Node t

Node tL Node tR Reduction in

impurity

Maxi(t)  PL  i(t L )  PR  i(t L )

i(.)  Impurity at node (.)
PL  Proportion of observations in the left node
PR  Proportion of observations in the right node
CART Example: German Credit Data
 First Split – 0DM
Impurity at Node 0 = 2 x 0.299 x 0.701 = 0.4191

Impurity at Node 1 = 2 x 0.237 x 0.763 x 0.73875

= 0.2671

Impurity at Node 2 = 2 x 0.474 x 0.526 x 0.26125

= 0.130272

Change in impurity = 0.4191 – 0.2671 –

CART Tree 0.130271 = 0.0217
 First Split – Credit Amount
Impurity at Node 0 = 2 x 0.299 x 0.701 = 0.4191

Impurity at Node 1 = 2 x 0.271 x 0.729 x 0.8775 =

0.3467

Impurity at Node 2 = 2 x 0.5 x 0.5 x 0.1225 =

0.06125

CART Tree Change in impurity = 0.4191 – 0.3467 – 0.06125 =

0.0112
Node 5: 0DM = 1 and duration is
<= 11.5, classify the observation
Y as 0 (88% accuracy)
 CHAID
Chi-square Automatic Interaction Detection
 Introduction to CHAID
• CHAID is one of the popular decision tree techniques used in
solving classification problems.

• Initial models of CHAID used chi-square test of independence for

splitting. CHAID was first presented in an article titled, “An
exploratory technique for Investigating large quantities of
categorical variables”, by G V Kass in Applied Statistics (1980)
 CHAID
• CHAID partitions the data into mutually exclusive, exhaustive,
subsets that best describe the dependent categorical variable.

• CHAID is an iterative procedure that examines the predictors (or

classification variables) and uses them in the order of their
statistical significance.
 CHAID Splitting Rule
• The splitting in CHAID is based on the type of
the dependent variable (or target variable).
– For continuous dependent variable F test is used.
– For categorical dependent variable chi-square test
of independence is used.
– For ordinal Y, likelihood ratio test is used.
 CHAID Example
• German Credit Rating Data Set.

• Target Variable – Checking Account Balance

 Chi-Square test of independence
• Chi-square test of independence starts with an assumption that
there is no relationship between two variables.

• For example, we assume that there is no relationship between

checking account balance and default.
 Chi-Square test of Independence
German Credit Case

• H0: There is no relationship between checking account

balance and default.

• HA: There is a relationship between checking account balance

and default.
 Contingency Table
Checking account Default
balance 1 0 Total
0 DM 135 139 274
Other than
0 DM 165 561 726
Total 300 700 1000
 Chi-Square test of Independence
in German Credit Data

• H0: Checking account balance and default are independent.

• HA: Checking account balance and default are dependent.

Chi-Square test of Independence –
Test Statistic

2
χ  statistic   

 O  E 
2

 E 
 
row sum  column sum
Expected frequency E 
total sum
 Chi-Square test of Independence
Observed Expected
frequency Frequency (O-E)^2/E
0DM-1 135 82.2 33.91
0DM-0 139 191.8 14.53
N0DM-1 165 217.8 12.8
N0DM-0 561 508.2 5.48
1000 1000 66.73
P-value 3.10E-16

p-value is less than 0.05, we reject the null

hypothesis.
 CHAID Procedure
• Step 1: Examine each predictor variable for its statistical
significance with the dependent variable using F test (for
continuous dependent) or chi-square test for categorical
dependent).

• Step 2: Determine the most significant among the predictors

(predictor with smallest p value after Bonferonni correction).

• Step 3: Divide the data by levels of the most significant predictor

Each of these groups will be examined individually further.

• Step 4: For each sub-group, determine the most significant

variable from the remaining predictor and divide the data again.

• Step 5: Repeat step 4 till stopping criteria is reached.

 Merging - CHAID
• CHAID uses both splitting and merging steps.

• In merging, least significantly different groups are merged to form

one class.
 CHAID Stopping Criteria
• Maximum tree depth is reached (which is pre-
defined).

• Minimum number of a cases to be a parent node is

reached (again pre-defined)

• Minimum number of to be a child node is reached.

 CHAID Input
• Significance level for partitioning a variable.

• Significance level for merging.

• Minimum number of records for the cells.

CHAID Example: German Credit Data
 First Split – 0DM
OBSERVED
Response variable Total
Predictor Y=1 Y=0
0DM = 1 99 110 209
0DM = 0 140 451 591
Total 239 561 800

Expected
Response Variable
Predictor Total
Y=1 Y=0
0DM = 1 62.43875 146.56 209
CHAID Tree 0DM = 0 176.5613 414.44 591
Total 239 561 800

P-value = 1.28 x 10-10

 First Split – 0DM
Business Rules

Business rules are always derived for leaf nodes

(node without any branch)

Business rules:

Node 1: If 0 DM = 0 then classify the observation

as 0 (good credit) which will be accurate 76% of
the times

CHAID Tree
Node 7, 90% accuracy

Node 5, 87.8 % accuracy

 Business Rule
• Node 7: If the Checking account balance is not less than 200
DM and there are no other installments, then classify the
observation as Y = 0.
 Tree Pruning
• Tree pruning is a process of reducing the size
of the tree using different strategies.

• Pruning is important since large tree can result

in over fitting.

• Pruning can be either based on the size of the

or purity.
Classification and Regression Tree
(CART)
 Classification and Regression Trees
(CART)
• Splits are chosen based either on Gini Index or
Entropy criteria.

• CART is a binary tree, whereas CHAID can split

the initial node into more than 2 branches.
Gini Index (Classification Impurity)
• Gini Index is used to measure the impurity at a
node (in classification problem) and is given
by:
J
Gini(k)   P( j k )(1  P( j k ))
j 1
where P(j k) is the proportion of category j in node k

Smaller Gini Index implies less impurity.

 Entropy (Impurity Measure)
• Entropy is another impurity measure that is
frequently used. Entropy at node “k” is given
by:
c
Entropy(k)    P(j | k)log(P(j | k))
j1
 Gini Index Calculation
Number of classes 2 (say 0 and 1)

Consider node label k with 10 1s and 90 0s.

2
Gini(k)   P( j k )(1  P( j k ))  2 P( j k )(1  P( j k ))
j 1

Gini(k)  2  0.1 0.9  0.18

Smaller number
implies less
impurity
 Entropy Calculation
Number of classes 2 (say 0 and 1)

Consider node label k with 10 1s and 90 0s.

2
Entropy(k)    P( j k ) log( P( j k ))
j 1

Entropy(k)  0.1 log(0.1)  0.9  log(0.9)  0.4689

Higher than Gini

coefficient
 Classification Tree Logic
Node t

Node tL Node tR Reduction in

impurity

Maxi(t)  N L  i(t L )  N R  i(t L )

i(.)  Impurity at node (.)
PL  Proportion of observations in the left node
PR  Proportion of observations in the right node
CART Example: German Credit Data
 First Split – 0DM
Impurity at Node 0 = 2 x 0.299 x 0.701 = 0.4191

Impurity at Node 1 = 2 x 0.237 x 0.763 x 0.73875

= 0.2671

Impurity at Node 2 = 2 x 0.474 x 0.526 x 0.26125

= 0.130272

Change in impurity = 0.4191 – 0.2671 –

CART Tree 0.130271 = 0.0217
 First Split – Credit Amount
Impurity at Node 0 = 2 x 0.299 x 0.701 = 0.4191

Impurity at Node 1 = 2 x 0.271 x 0.729 x 0.8775 =

0.3467

Impurity at Node 2 = 2 x 0.5 x 0.5 x 0.1225 =

0.06125

CART Tree Change in impurity = 0.4191 – 0.3467 – 0.06125 =

0.0112
Node 5: 0DM = 1 and duration is
<= 11.5, classify the observation
Y as 0 (88% accuracy)
 Ensemble Methods
• Ensemble methods is a majority voting method in which
models are developed using various techniques such as
logistic regression, discriminant analysis, CHAID, CART etc.

• For each observation, the classification is done based on

majority voting.

• Ensemble methods in general improve the accuracy of

prediction.
 Random Forest
• Random forest is an ensemble method in
which several trees are generated (thus the
name forest) by sampling the predictor
variable.

• The sampling can also be on the training data

set.