The following study notes are from the textbook "Introduction to Machine Learning with

Python". Refer to the author's GitHub repo at

https://github.com/amueller/introduction_to_ml_with_python.

Supervised learning
Supervised learning is used whenever we want to predict a certain outcome from a given
input, and we have examples of input/output (predictor/response) pairs. We build a
machine learning model model from these input/output pairs, which comprise our training
set. Our goal is to make accurate predictions for new, never-before-seen data. Supervised
learning often requires human effort to build the training set, but afterward automates and
often speeds up an otherwise laborious or infeasible task.

Classification and regression

There are two major types of supervised machine learning problems:
• Classification
• Regression
In classification, the goal is to predict a class label, which is a choice from a predefined list
of possibilities. Classification is sometimes separated into binary classification, which is
the special case of distinguishing between exactly two classes, and multiclass
classification, which is classification between more than two classes.
You can think of binary classification as trying to answer a yes/no question. For example,
classifying emails as either spam or not spam. In this binary classification task, the yes/no
question being asked would be "Is this email spam?" On the other hand, predicting what
language a website is in from the text on the website, is an example of a multiclass
classification problem. The classes here would be a pre-defined list of possible languages.
In regression, the goal is to predict a continuous number, or a floating-point number in
programming terms (or real number in mathematical terms). Predicting a person's annual
income from their education, their age, and where they live is an example of a regression
task. When predicting income, the predicted value is an amount, and can be any number in
a given range.
An easy way to distinguish between classification and regression tasks is to ask whether
there is some kind of continuity in the output. If there is continuity between possible
outcomes, then the problem is a regression problem.
Generalization, overfitting, and underfitting
In supervised learning, we want to build a model on the training data and then be able to
make accurate predictions on new, unseen data that has the same characteristics as the
training set that we used. If a model is able to make accurate predictions on unseen data,
we say it is able to generalize from the training set to the test set. We want to build a
model that is able to generalize as accurately as possible.
Building a model that is too complex for the amount of information we have is called
overfitting. Overfitting occurs when you fit a model too closely to the particularities of the
training set and obtain a model that works well on the training set but is not able to
generalize to new data.
On the other hand, if your model is too simple then you might not be able to capture all the
aspects of and variability in the data, and your model will do badly even on the training set.
Choosing too simple a model is called underfitting.
The more complex we allow our model to be, the better we will be able to predict on the
training data. However, if our model becomes too complex, we start focusing too much on
each individual data point in our training set, and the model will not generalize well to new
data.
There is a sweet spot in between that will yield the best generalization performance. This is
the model we want to find. The trade-off between overfitting and underfitting is illustrated
in Figure 2-1.

Source: Müller, A. C., & Guido, S. (2016). Introduction to Machine Learning with Python: A
GUIDE FOR DATA SCIENTISTS (IMLP). "O'Reilly Media, Inc.".
On the other hand, we should be aware that larget dataset size itself will lead to bigger
model complexity, turning the sweet spot further right on the complexity spectrum. In the
real world, you often have the ability to decide how much data to collect, which might be
more beneficial than tweaking and tuning your model. Never underestimate the power of
more data.
Creating a sample dataset
Scikit-learn has a lot of tools for creating synthetic (meaning made-up) datasets, which
are great for testing machine learning algorithms. I'm going to utilize the make_blobs
method.
I create a synthetic two-class classification dataset which has two features. The following
code creates a dataset and a scatter plot visualizing all of the data points in this dataset. The
plot has the first feature on the x-axis and the second feature on the y-axis. As is always the
case in scatter plots, each data point is represented as one dot. The color of the dot
indicates its class.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.datasets import make_blobs

%matplotlib inline

X, y = make_blobs(n_samples=30, centers=2, random_state=4)

y[np.array([7, 27])] = 0
mask = np.ones(len(X), dtype=bool)
mask[np.array([0, 1, 5, 26])] = 0
X, y = X[mask], y[mask]

colormap = np.array(['b', 'orange'])

plt.scatter(X[:, 0], X[:, 1], c=colormap[y], marker="o", s=50,
label=colormap[y])
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.show()

print("X.shape: {}".format(X.shape)) # This dataset consists of 26

data points, with 2 features.
X.shape: (26, 2)

Figure 2-2. Scatter plot of the same dataset from the textbook IMLP
k-Nearest Neighbors
The k-NN algorithm is arguably the simplest machine learning algorithm. Building the
model consists only of storing the training dataset. To make a prediction for a new data
point, the algorithm finds the closest data points in the training dataset--its "nearest
neighbors". This is a non-parametric, lazy learning algorithm.
• Non-parametric: No parameters are learned from the model
• Lazy algorithm: There is no explicit training phase involved
k-Neighbors classification
In its simplest version, the k-NN algorithm only considers exactly one nearest neighbor,
which is the closest training data point to the point we want to make a prediction for and
assigns its label to the test data. This is 1-NN. Figure 2-4 illustrates this for the case of
classification on the above dataset.
Here, we added three new data points, shown as stars. For each of them, we marked the
closest point in the training set. The prediction of the one-nearest-neighbor algorithm is
the label of that point (shown by the color of the cross).
Python code:
• from sklearn.neighbors import KNeighborsClassifier
• KNeighborsClassifier(n_neighbors=1)

Figure 2-4. Predictions made by the one-nearest-neighbor model

Instead of considering only the closest neighbor, we can also consider an arbitrary number,
k , of neighbors. This is where the name of the k-nearest neighbors (k-NN) algorithm comes
from.
When considering more than one neighbor, we use voting to assign a label. This means that
for each test point, we count how many neighbors belong to class 0 and how many
neighbors belong to class 1. We then assign the class that is more frequent: in other words,
the majority class among the k-nearest neighbors. The following example (Figure 2-5) uses
the three closest neighbors (3-NN).

Figure 2-5. Predictions made by the three-nearest-neighbors model

Python code:
• KNeighborsClassifier(n_neighbors=3)

The prediction is shown as the color of the cross. You can see that the prediction for the
new data point at the top left is not the same as the prediction when we used only one
neighbor.
While this illustration is for a binary classification problem, this method can be applied to
datasets with any number of classes. For more classes, we count how many neighbors
belong to each class and again predict the most common class (majority voting).
Now let's look at how we can apply the k-nearest neighbors algorithm using scikit-
learn. First, we split our data into a training and a test set so we can evaluate
generalization performance, as discussed above.
Steps to apply a k-NN model:
from sklearn.model_selection import train_test_split

# Create the training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y,
random_state=0)

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=3) # Instantiate a k-NN model
with 3 nearest neighbors

# Fit the model, knn, to the training set. For k-NN, it's simply
storing the data.
knn.fit(X_train, y_train)

print("Test set predictions: {}".format(knn.predict(X_test)))

# We calculate the predictions for y_test with the knn model. For each
data point in the test set,
# this computes its nearest neighbors in the training set and finds
the most common class among these.

print("Test set accuracy: {:.2f}".format(knn.score(X_test, y_test)))

# To evaluate how well our model generalizes, we call the score method
with the test data together
# with the test labels.

Test set predictions: [1 0 1 0 1 0 0]

Test set accuracy: 0.86

We see that our model is about 86% accurate, meaning the model predicted the class
correctly for 86% of the samples in the test dataset.
Analyzing KNeighborsClassifier
For two-dimensional datasets, we can also illustrate the prediction for all possible test
points in the xy-plane. We color the plane according to the class that would be assigned to a
point in this region. This lets us view the decision boundary, which is the divide between
where the algorithm assigns class 0 versus where it assigns class 1 to the new test data.
Let's build k-NN model for k =1 ,3 , 9 on the above dataset and plot the decision boundary.
from matplotlib.colors import ListedColormap
from sklearn import neighbors, datasets

h = 0.02 # Step size in the mesh

# Create color maps

cmap_light = ListedColormap(["lightblue", "lightyellow"])
cmap_bold = ["darkorange", "darkblue"]

for n_neighbors in [1, 3, 9]:

 knn = neighbors.KNeighborsClassifier(n_neighbors)
knn.fit(X, y)

# Plot the decision boundary. For that, we will assign a color to

each
# point in the mesh [x_min, x_max]x[y_min, y_max].
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min,
y_max, h))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot

Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

# Plot the training points

plt.scatter(X[:, 0], X[:, 1], c=colormap[y], marker="o", s=50,
label=colormap[y])
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.title("2-Class classification (k = %i)" % n_neighbors)

plt.show()
As you can see in the first figure, using a single neighbor results in a decision boundary that
follows the training data closely. Considering more and more neighbors leads to a
smoother decision boundary. A smoother boundary corresponds to a simpler model. In
other words, using few neighbors corresponds to high model complexity (as shown on the
right side of Figure 2-1), and using many neighbors corresponds to low model complexity
(as shown on the left side of Figure 2-1).
If you consider the extreme case where the number of neighbors is the number of all data
points in the training set, each test point would have exactly the same neighbors (all
training points) and all predictions would be the same: the class that is most frequent in the
training set.
k-NN accuracy on breast cancer data
Let's see the connection between model complexity and generalization (ability of the model
to make accurate predictions on new, unseen dataset). We will do this on the real-world
breast cancer dataset.
Breast cancer dataset records clinical measurements of breast cancer tumors. Each tumor
is labeled as "benign" (for harmless tumors) or "malignant" (for cancerous tumors), and the
task is to learn to predict whether a tumor is malignant based on the measurements of the
tissue. The data can be loaded using the load_breast_cancer function from scikit-
learn.
from sklearn.datasets import load_breast_cancer

cancer = load_breast_cancer()

print("Shape of cancer data: {}".format(cancer.data.shape))

print("Sample counts per class:\n{}".format({n: v for n, v in
zip(cancer.target_names, np.bincount(cancer.target))}))

Shape of cancer data: (569, 30)

Sample counts per class:
{'malignant': 212, 'benign': 357}

The dataset consists of 569 data points, with 30 features each. Of these 569 data points,
212 are labeled as malignant and 357 as benign.
We begin by splitting the dataset into a training and a test set.
# Create training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(cancer.data,
cancer.target, stratify=cancer.target, random_state=66)

print ("Training data shape:{}".format(X_train.shape))

print ("Test data shape :{}".format(X_test.shape))

Training data shape:(426, 30)

Test data shape :(143, 30)

Then we evaluate training and test set performance with different numbers of neighbors.
training_accuracy = []
test_accuracy = []
# try n_neighbors from 1 to 20
neighbors_settings = range(1, 21)

for n_neighbors in neighbors_settings:

# build the model
knn = KNeighborsClassifier(n_neighbors=n_neighbors)
knn.fit(X_train, y_train)
# record training accuracy
training_accuracy.append(knn.score(X_train, y_train))
# record test accuracy
test_accuracy.append(knn.score(X_test, y_test))

plt.plot(neighbors_settings, training_accuracy, label="training

accuracy")
plt.plot(neighbors_settings, test_accuracy, label="test accuracy")
plt.ylabel("Accuracy")
plt.xlabel("n_neighbors")
plt.title("Comparison of training and test accuracy as a function of
n_neighbors\n")
plt.legend()
plt.show()

While real-world plots are rarely very smooth, we can still recognize some of the
characteristics of overfitting and underfitting (note that considering fewer neighbors
corresponds to a more complex model).
Considering a single nearest neighbor, the prediction on the training set is perfect (close to
1). But when more neighbors are considered, the model becomes simpler and the training
accuracy drops. The test set accuracy for using a single neighbor is lower than when using
more neighbors, indicating that using the single nearest neighbor leads to a model that is
too complex. On the other hand, when considering 10 neighbors, the model is too simple
and performance is even worse. The best performance is somewhere in the middle, using
around 6 neighbors (6-NN) with a test accuracy of 93.7%. The worst performance is
around 88% accuracy (for n_neighbour=2), which might still be acceptable.
Keep in mind that best n_neighbours vary from dataset to dataset and even within same
dataset of different training/test data size.
print('--- Based on test set ---\n')

index = test_accuracy.index(max(test_accuracy))
print('Best n_neighour = {} | Training accuracy = {:0.3f} | Test
accuracy = {:0.3f}'.format(neighbors_settings[index],
training_accuracy[index], test_accuracy[index]))

index = test_accuracy.index(min(test_accuracy))
print ('Worst n_neighour = {} | Training accuracy = {:0.3f} | Test
accuracy = {:0.3f}'.format(neighbors_settings[index],
training_accuracy[index], test_accuracy[index]))

--- Based on test set ---

Best n_neighour = 6 | Training accuracy = 0.946 | Test accuracy =

0.937
Worst n_neighour = 2 | Training accuracy = 0.977 | Test accuracy =
0.888

Summary
Parameters
The two most important parameters of k-NN are:
• n_neighbors: the number of neighbours
• metric: how we measure the distance between data points
n_neighbors

• Lower n_neighbors: More complex model ⇒ Overfitting

• Higher n_neighbors: More simplistic model ⇒ Underfitting
• We should find the sweet spot between a simple and complex model.
We have already seen how to select the best n_neighbours based on the test set score
(accuracy).
metric

Default distance metric is Minkowski with power parameter of 2 which generally works
well.
Minkowsi distance of order p between two points =

( ∑|x i − y i ) p )
1/p
for i=1 , . .. , n

• Minkowski with p=1 ⇒ Manhattan distance ( ℓ1 ) = ∑|x i − y i )

Minkowski with p=2 ⇒ Euclidean distance ( ℓ2 ) = ( ∑|x i − y i ) )
2 1 /2
•

In practice, using a small number of neighbors like three or five often works well, but you
should certainly adjust this parameter. Choosing the right distance measure requires
understanding the context. By default, Euclidean distance is used, which works well in
many settings.
For further details on parameters, refer the scikit-learn docmentation:
KNeighborsClassifier
Strength and weakaness
Strength:
• Simple, easy to understand and implement
• Less number of parameters to tune
• Nonparametric (i.e., no weights are learnt)
• Fast training phase as it just have to store all the training sampels
• Works well for multiclass data sets
• A good baseline method to try before considering more advanced techniques
Weakness:
• Computationally expensive testing phase (hence, impractical in industrial
standards)
• Prediction can be slow, when the training set is large (in terms of number of
features or number of samples)
• k-NN can suffer from skewed class distributions (certain class being frequent in
training set will dominate majority voting).
• Often does not perform well on datasets with many features (hundreds or more),
and it does particularly badly with datasets where most features are 0 most of the
time (so-called sparse datasets)
Data have to be preprocessed (to avoid skewing the distance metric in favor of a particular
feature).
So, while the nearest k-neighbors algorithm is easy to understand, it is not often used in
practice, due to prediction being slow and its inability to handle many features. The method
we discuss next has neither of the k-NNs drawbacks.