This document provides an overview of classification techniques in data mining. It discusses classification vs prediction, the steps in classification (learning and prediction), and issues regarding data preparation and algorithm selection. It also provides examples of calculating entropy, information gain, and building a decision tree for a weather dataset. The document demonstrates how attributes are selected and added to the decision tree based on maximum information gain.
This document provides an overview of classification techniques in data mining. It discusses classification vs prediction, the steps in classification (learning and prediction), and issues regarding data preparation and algorithm selection. It also provides examples of calculating entropy, information gain, and building a decision tree for a weather dataset. The document demonstrates how attributes are selected and added to the decision tree based on maximum information gain.
Original Description:
Issues regarding classification and prediction
Original Title
Classification - issues regarding classification and prediction
This document provides an overview of classification techniques in data mining. It discusses classification vs prediction, the steps in classification (learning and prediction), and issues regarding data preparation and algorithm selection. It also provides examples of calculating entropy, information gain, and building a decision tree for a weather dataset. The document demonstrates how attributes are selected and added to the decision tree based on maximum information gain.
This document provides an overview of classification techniques in data mining. It discusses classification vs prediction, the steps in classification (learning and prediction), and issues regarding data preparation and algorithm selection. It also provides examples of calculating entropy, information gain, and building a decision tree for a weather dataset. The document demonstrates how attributes are selected and added to the decision tree based on maximum information gain.
• Bank loan • Buys computer • Best treatment • The data analysis task is classification • A model or classifier is constructed to predict categorical labels, such as • “safe” or “risky” for the loan application data • “yes” or “no” for the marketing data • “treatment A,” “treatment B,” or “treatment C” for the medical data 2/39 Classification Vs Prediction
• Predict how much a given customer will spend
during a sale • The model constructed predicts a continuous-valued function, or ordered value, as opposed to a categorical label • Process is known as prediction/regression • The model is known as predictor
3/39 Classification Step1
4/39 Classification Step2
5/39 Classification: Step1-Learning
• Learning step using the training set
• Training set: Database tuples and their associated class labels • A tuple X = (x1 , x2 , . . . , xn ) is an attribute vector belongs to a predefined class label (categorical, discrete and unordered) • Learning y = f (X ), where y is the class label for tuple X • Because the class label of each training tuple is provided, this step is also known as supervised 6/39 learning Classification: Step2- Prediction
• A test set is used, made up of test tuples and
their associated class labels which are not used to construct the classifier • Accuracy of a classifier is the percentage of test set tuples that are correctly classified by the classifier
7/39 Points to note
• Supervised vs Unsupervised learning
• In prediction, no class label attribute, just the predicted attribute • Can be viewed as a mapping or function, y = f (X ), where X is the input (e.g., a tuple describing a loan applicant), and the output y is a continuous or ordered value • Eg :- Predicted loan amount
8/39 Issues Regarding Classification and Prediction Data Preparation
• Prepare data for classification and prediction to
improve the accuracy, efficiency, and scalability: • Data cleaning to avoid noise and missing values • Relevance analysis to identify whether any two given attributes are statistically related (use correlation analysis) • Attribute subset selection to remove irrelevant/redundant attributes • Data Transformation. Eg :- Normalization 9/39 Comparing Classification and Prediction Methods
• Accuracy • Speed • Robustness (performance given noisy data or data with missing values) • Scalability • Iterpretability (level of understanding and insight that is being provided by the classifier/predictor)
10/39 Decision Tree Induction Decision Tree Algorithm Basics
• Three algorithms • All were proposed in 1980s • ID3 (Iterative Dichotomiser) • C4.5 (a successor of ID3) • Classification and Regression Trees (CART) • Has three parameters: D, attribute_list, and Attribute_selection_method • Refer to Figure 6.3, Page 293, Han and Kamber 2nd Ed Textbook 11/39 Decision Tree Algorithms
• D is a data partition, Initiallly the entire training
data set • attribute_list is a list of attributes describing the tuples • Attribute_selection_method specifies a heuristic procedure for selecting the attribute that “best” discriminates the given tuples according to class • information gain or the gini index (binary only) • We use information gain as our trees many not 12/39 always be binary Entropy(H)
• H(V ) = P(vk ) log2 P(v1 k ) =
P k P − k P(vk ) log2 P(vk ) • Let X be a collection of training examples • p+ is the collection of positive examples • p− is the collection of negative examples • H(X ) = −p+ log2 p+ − p− log2 p−
in the given data, caused by partitioning the example based on an attribute • The information gain Gain(X , A) of an attribute A, relative to a collection of examples X is defined as X |X (v )| Gain(X , A) = H(X ) − H(X (v )) |X | v ∈Values(A)
15/39 Decision Tree Demo
16/39 Entropy of Outlook
• Entropy of the dataset
• H(X ) = H(9+, 5−) = −9/14 log2 (9/14) − 5/14 log2 (5/14) = 0.94 • To determine the entropy of Outlook • Values of Outlook −→ Sunny , Overcast, Rainy
