The document discusses decision tree classification. It explains that decision trees break down a dataset into smaller subsets while developing a tree structure with decision and leaf nodes. The ID3 algorithm is used to build decision trees by calculating entropy to measure the homogeneity of samples and selecting the attribute with the highest information gain at each node. Information gain is the decrease in entropy after a dataset is split on an attribute, with the goal being to create the most homogeneous branches. The algorithm recursively splits the data until all instances are classified at the leaf nodes.
The document discusses decision tree classification. It explains that decision trees break down a dataset into smaller subsets while developing a tree structure with decision and leaf nodes. The ID3 algorithm is used to build decision trees by calculating entropy to measure the homogeneity of samples and selecting the attribute with the highest information gain at each node. Information gain is the decrease in entropy after a dataset is split on an attribute, with the goal being to create the most homogeneous branches. The algorithm recursively splits the data until all instances are classified at the leaf nodes.
The document discusses decision tree classification. It explains that decision trees break down a dataset into smaller subsets while developing a tree structure with decision and leaf nodes. The ID3 algorithm is used to build decision trees by calculating entropy to measure the homogeneity of samples and selecting the attribute with the highest information gain at each node. Information gain is the decrease in entropy after a dataset is split on an attribute, with the goal being to create the most homogeneous branches. The algorithm recursively splits the data until all instances are classified at the leaf nodes.
The document discusses decision tree classification. It explains that decision trees break down a dataset into smaller subsets while developing a tree structure with decision and leaf nodes. The ID3 algorithm is used to build decision trees by calculating entropy to measure the homogeneity of samples and selecting the attribute with the highest information gain at each node. Information gain is the decrease in entropy after a dataset is split on an attribute, with the goal being to create the most homogeneous branches. The algorithm recursively splits the data until all instances are classified at the leaf nodes.
Download as DOCX, PDF, TXT or read online from Scribd
Download as docx, pdf, or txt
You are on page 1/ 11
Decision Tree Classification
Decision tree builds classification or regression models in the form of a tree structure. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes.
A decision node (e.g., Outlook) has two or more branches (e.g., Sunny, Overcast and Rainy).
Leaf node (e.g., Play) represents a classification or decision.
The topmost decision node in a tree which corresponds to the best predictor called root node. Decision trees can handle both categorical and numerical data.
Information theory
Introduced by Claude Shannon. It has quantified entropy. This is key measure of information which is usually expressed by the average number of bits needed to store or communicate one symbol in a message.
Information theory measure information in bits
ID3 Algorithm:
Based on greedy search through the space of all possible branches with no backtracking
Includes all predictors
Entropy
A decision tree is built top-down from a root node and involves partitioning the data into subsets that contain instances with similar values (homogenous). ID3 algorithm uses entropy to calculate the homogeneity of a sample. If the sample is completely homogeneous the entropy is zero and if the sample is an equally divided it has entropy of one.
entropy(p1,p2,…,pn)=−p1log(p1)−p2log(p2)−⋯−pnlog(pn) To build a decision tree, we need to calculate two types of entropy using frequency tables as follows:
a) Entropy using the frequency table of one attribute:
b) Entropy using the frequency table of two attributes:
Information Gain The information gain is based on the decrease in entropy after a dataset is split on an attribute. Constructing a decision tree is all about finding attribute that returns the highest information gain (i.e., the most homogeneous branches).
Step 1: Calculate entropy of the target.
Step 2: The dataset is then split on the different attributes. The entropy for each branch is calculated. Then it is added proportionally, to get total entropy for the split. The resulting entropy is subtracted from the entropy before the split. The result is the Information Gain, or decrease in entropy.
Step 3: Choose attribute with the largest information gain as the decision node, divide the dataset by its branches and repeat the same process on every branch. Step 4a: A branch with entropy of 0 is a leaf node.
Step 4b: A branch with entropy more than 0 needs further splitting.
Step 5: The ID3 algorithm is run recursively on the non-leaf branches, until all data is classified. Decision Tree to Decision Rules A decision tree can easily be transformed to a set of rules by mapping from the root node to the leaf nodes one by one.
Detailed Explanation about Calculations
Overall Entropy
Expected new entropy for each attribute
Outlook
Outlook attribute contains 3 distinct values: Overcast, Rainy, Sunny
