Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
3 views

ID3 Decision Tree Algorithm

The document provides an overview of the ID3 decision tree algorithm, which uses Information Gain to classify data by recursively splitting datasets based on features. It highlights the advantages of ID3, such as its simplicity and efficiency, while also noting its limitations, including overfitting and bias towards features with many categories. The document also mentions C4.5 as an improvement over ID3, addressing some of its limitations by handling continuous data and reducing bias.

Uploaded by

yuvrajas4074
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
3 views

ID3 Decision Tree Algorithm

The document provides an overview of the ID3 decision tree algorithm, which uses Information Gain to classify data by recursively splitting datasets based on features. It highlights the advantages of ID3, such as its simplicity and efficiency, while also noting its limitations, including overfitting and bias towards features with many categories. The document also mentions C4.5 as an improvement over ID3, addressing some of its limitations by handling continuous data and reducing bias.

Uploaded by

yuvrajas4074
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 18

Basic Decision Tree Learning

Algorithms
A Comprehensive Overview
Er. Zubair
Overview of ID3 Algorithm
• Some of the common Decision Tree Learning Algorithm are:
 ID3 (Iterative Dichotomiser 3)

 C4.5

 CART (Classification and Regression Trees)

1. ID3 (Iterative Dichotomiser 3)


• ID3 is an early and well-known decision tree algorithm for classification
problems.
• The algorithm splits the dataset recursively based on features, ultimately
creating a model for classification.
• Key feature: The algorithm uses Information Gain to decide which feature
to split on at each node.
Understanding Information Gain
• Information Gain (IG) measures how well a feature separates the
dataset into distinct classes.
• Information Gain is based on Entropy, which measures the
uncertainty or impurity of a dataset.
• The formula for Entropy: Entropy(S) = -∑p(i) log2 p(i)
 Where p(i) is the proportion of class i in the dataset.

• Information Gain is the difference in entropy before and after a


split based on a feature.
How ID3 Uses Information Gain
• At each node, ID3 calculates the Information Gain for each
feature.
• The feature with the highest Information Gain is chosen to split
the dataset.
• By doing this, ID3 aims to reduce uncertainty and create purer
nodes at each step.
• Example: Splitting the data based Outlook when classifying
whether to play tennis.
Advantages of ID3
Simple to understand: Based on a clear metric, Information Gain.
Efficient: Computationally efficient for small to medium datasets.
Transparent: The tree structure is easy to visualize and interpret.
Limitations of ID3
• Overfitting: Tends to create deep trees that may overfit the
training data.
• Bias towards features with many categories: May favor
features with many values.
• No support for continuous data: Needs data to be discretized
or transformed for continuous attributes.
Improvement: C4.5
C4.5 is an extension of ID3 that addresses some limitations:

• Handles both continuous and categorical features.

• Uses Gain Ratio instead of Information Gain to reduce bias


towards features with many categories.

• Includes pruning to remove unnecessary branches and reduce


overfitting.
Example of ID3
• Given a dataset of weather conditions, ID3 calculates the
Information Gain for each feature.
• Entropy of the whole dataset (using the formula):

• where p+​ and p−are the proportions of the positive and negative
classes, respectively.
• Step 1: Entropy of the Entire Dataset
Example of ID3
Example of ID3
Example of ID3
Example of ID3
Example of ID3
• With these values, you can build a decision tree, Weather
provides the highest information gain. So, it should be used as
the root node, followed by other attributes based on their
information gain values.
• Since Weather has the highest information gain, we choose it
as the root node. The possible values for Weather are: Sunny,
Cloudy, Rain. We'll split the dataset based on these values
and compute the information gain for the remaining
attributes (Temperature, Humidity, and Wind) for each
subset.
Example of ID3
• With these values, you can build a decision tree, Weather
provides the highest information gain. So, it should be used as
the root node, followed by other attributes based on their
information gain values.
• Since Weather has the highest information gain, we choose it
as the root node. The possible values for Weather are: Sunny,
Cloudy, Rain. We'll split the dataset based on these values
and compute the information gain for the remaining
attributes (Temperature, Humidity, and Wind) for each
subset.
Example of ID3

Day Weather Temperature Humidity Wind Play Football?

Day 1 Sunny Hot High Weak No

Day 2 Sunny Hot High Strong No

Day 8 Sunny Mild High Weak Yes

Day 9 Rain Cool Normal Weak Yes

Day 11 Sunny Mild Normal Strong Yes


Example of ID3
Play
Day Weather Temperature Humidity Wind Football?

Day 4 Rain Mild High Weak Yes


Day 5 Rain Cool Normal Weak Yes
Day 6 Rain Cool Normal Strong Yes
Day 9 Rain Cool Normal Weak Yes
Day 10 Rain Mild Normal Weak Yes
Day 14 Rain Hot High Strong No
Conclusion
• ID3 is a simple and powerful decision tree algorithm that uses
Information Gain for decision-making.

• While effective for many problems, improvements like C4.5


address its limitations such as handling continuous data and
reducing bias.

• ID3 and C4.5 laid the foundation for modern decision tree
algorithms.

You might also like