CSDS 440: Machine Learning

https://cwru.zoom.us/rec/share/s-8J2q_vYWRh7RGNOg9HDAoweQgXgpRSbwc5Akph-
Y6sBl1fsi-guHYSRPZiOphx.F6GOaUOhJkuUzEuH
Soumya Ray (sray@case.edu)
Office: Zoomlandia 920 0026 0871/ 598943
Office hours: F 9-10am or by appointment
• Please ensure your full name is visible on zoom.
• Check that your mic is connected by looking for the mic symbol next to your name in the
Participants list. If not you can configure your mic and speakers using arrow next to the mic
button in the lower left of the zoom interface.
• You will be muted on entry (mic symbol has a line through it). Unmute to ask/answer
questions.
• Leave your mic on mute and video off until you are speaking.
• To ask a question, use “Raise Hand” and wait to be called or send your question via chat.
• To answer a question, use “Raise Hand” and wait to be called.
• After asking or answering your question, click “Raise Hand” again to lower your hand.
• This meeting is being recorded. The recording will be made available, likely via canvas.
• If I drop out/can’t be heard/screen freezes/slides disappear etc, please send a note in the chat
window.
• If I get disconnected completely, I will rejoin asap. Please be patient and wait. If there is a
serious issue and I cannot rejoin, I will send email and the class will be postponed.

9/15/2020 Soumya Ray, Case Western Reserve U. 1

Recap
• What is overfitting?
• We control overfitting in trees through (ES) and (PP).
• How does ES work? Why might it not work well in practice?
• PP uses a ____ set. It iteratively _____ a node and evaluates the
result. It selects a node that ________. It stops when _____.
• What is the geometry of the tree’s decision boundary?
• What are some pros of using decision trees? Cons?
• What is goal of learning algorithm performance evaluation?
• Given a finite dataset, we want the training set to an algorithm to
be as ____ as possible. We also want the test sets to be _____.
• These goals are achieved by __-___ ___ ___.
• How does this procedure work?
• What is leave one out?
• What is stratified CV?
9/15/2020 Soumya Ray, Case Western Reserve U. 2
Today
• Evaluation Metrics for Classification
• Review of Optimization

9/15/2020 Soumya Ray, Case Western Reserve U. 3

 Contingency Table
Class according to Target Concept
(Correct Answer)
Positive Negative
Class according to Learned Classifier

True Positives False Positives

(Predicted Answer)

Positive (TP) (FP)

(Type I error)

False Negatives True Negatives

Negative
(FN) (TN)
(Type II error)

9/15/2020 Soumya Ray, Case Western Reserve U. 4

Accuracy
• Most commonly used measure for comparing
classification algorithms

TP  TN
Accuracy 
TP  TN  FP  FN

9/15/2020 Soumya Ray, Case Western Reserve U. 5

Weaknesses of Accuracy
• Does not account for:
– Skewed class distributions
– Differential misclassification costs
– Confidence estimates from learning algorithms

9/15/2020 Soumya Ray, Case Western Reserve U. 6

Weighted/Balanced Accuracy
• Corrects for skewed class distributions

1  TP TN 
WAcc    
2  Allpos Allneg 
1  TP TN 
   
2  TP  FN TN  FP 
True Positive Rate True Negative Rate

9/15/2020 Soumya Ray, Case Western Reserve U. 7

Measuring one class
• Often, just a single class is “interesting”
– Call this the “positive” class
Positive Negative

True Positives False Positives

Positive (TP) (FP)
(Type I error)

False Negatives True Negatives

Negative
(FN) (TN)
(Type II error)

9/15/2020 Soumya Ray, Case Western Reserve U. 8

Precision
• Of the examples the learner predicted
positive, how many were actually positive?

TP
Precision 
TP  FP

9/15/2020 Soumya Ray, Case Western Reserve U. 9

Recall/TP rate/Sensitivity
• Of the examples that were actually positive,
how many did the learner predict correctly?

TP TP
Recall  
TP  FN Allpos

9/15/2020 Soumya Ray, Case Western Reserve U. 10

Specificity/TN rate
• Counterpart of recall for the negative class

TN TN
Specificity  
TN  FP Allneg
• So:
1
WAcc   Sensitivity  Specificity 
2
9/15/2020 Soumya Ray, Case Western Reserve U. 11
F1 score
• Combines precision and recall into a single
measure, giving each equal weight
1 1 1 1 
   
F1 2  Precision Recall 
2
F1 
1 1

Precision Recall
9/15/2020 Soumya Ray, Case Western Reserve U. 12
Beyond point estimates
• Everything above is a “point estimate”

• Because they will be computed on the basis of

a sample, we can also compute variance
estimates for each quantity

• Important to show “stability” of solutions, and

when comparing across algorithms (later)
9/15/2020 Soumya Ray, Case Western Reserve U. 13
Learning Curves
• Often useful to plot each metric as a function
of training sample size
• Provides insight into how many examples the
algorithm needs to become effective
1

Metric 0.5
(e.g. Accuracy)

100 200 1000

Training Sample Size
9/15/2020 Soumya Ray, Case Western Reserve U. 14
Metrics with Confidence Measures
• Many learning algorithms produce classifiers
or models that can provide estimates of how
confident they are about a prediction

• In this case, can plot Precision-Recall (PR) and

Receiver Operating Characteristic (ROC)
graphs

9/15/2020 Soumya Ray, Case Western Reserve U. 15

Precision-Recall graphs
Confidence Recall Precision
True Class
On + (x axis) (y axis)

Example 1 + 0.9

Example 2 − 0.8

Example 3 + 0.4

Example 4 − 0.3

9/15/2020 Soumya Ray, Case Western Reserve U. 16

Precision-Recall graphs
True Class
Confidence Recall Precision Precision
On + (x axis) (y axis)

1
Example 1 + 0.9 0.5 1
0.5
Example 2 − 0.8 0.5 0.5

Example 3 + 0.4 1 0.67

0.5 1
Example 4 − 0.3 1 0.5 Recall

9/15/2020 Soumya Ray, Case Western Reserve U. 17

ROC graphs
FP Rate
Confidence Sens./Recall
True Class (1-Spec.)
On + (y axis)
(x axis)

Example 1 + 0.9

Example 2 − 0.8

Example 3 + 0.4

Example 4 − 0.3

9/15/2020 Soumya Ray, Case Western Reserve U. 18

ROC graphs
True Class
Confidence FP Rate Sens./Recall TP rate
On + (x axis) (y axis)

1
Example 1 + 0.9 0 0.5
0.5
Example 2 − 0.8 0.5 0.5

Example 3 + 0.4 0.5 1

0.5 1
Example 4 − 0.3 1 1 FP rate

9/15/2020 Soumya Ray, Case Western Reserve U. 19

Properties of ROC graphs
• Random guessing is a diagonal line
– Also majority class classifier
– If your classifier is any good its ROC must lie above
the diagonal
• Monotonically increasing
• Often use “AUC”/ “AROC” as comparison
statistic (later)
• Can be misleading if class distribution is too
skewed (use PR graphs instead)
9/15/2020 Soumya Ray, Case Western Reserve U. 20