Statistical Machine Learning

Unit – I

Dr.S.Veena, Associate Professor, SRMIST 1

 Introduction
• Statistical modeling is the use of mathematical models and statistical
assumptions to generate sample data and make predictions about the
real world. A statistical model is a collection of probability
distributions on a set of all possible outcomes of an experiment.
• Machine learning is a branch of study in which a model can learn
automatically from the experiences based on data without exclusively
being modeled like in statistical models.

Dr.S.Veena, Associate Professor, SRMIST 2

 Statistical terminology for
model building and validation
• Statistics is the branch of mathematics dealing with the collection, analysis, interpretation, presentation,
and organization of numerical data.
• Statistics are mainly classified into two subbranches:

• Statistical modeling is applying statistics on data to find underlying hidden relationships by analyzing the
significance of the variables.

Dr.S.Veena, Associate Professor, SRMIST 3

 Machine learning
• Machine learning is the branch of computer science that utilizes past
experience to learn from and use its knowledge to make future
decisions.
• Machine learning is at the intersection of computer science,
engineering, and statistics.
• The goal of machine learning is to generalize a detectable pattern or
to create an unknown rule from given examples.

Dr.S.Veena, Associate Professor, SRMIST 4

 Machine learning
• An overview of machine learning landscape is as follows:

Dr.S.Veena, Associate Professor, SRMIST 5

 Machine learning
• Machine learning is broadly classified into three
categories but nonetheless, based on the situation,
these categories can be combined to achieve the
desired results for particular applications:
• Supervised Learning
• Un Supervised Learning
• Reinforcement Learning

Dr.S.Veena, Associate Professor, SRMIST 6

 Supervised learning:
• This is teaching machines to learn the relationship between other
variables and a target variable, similar to the way in which a teacher
provides feedback to students on their performance.
• The major segments within supervised learning are as follows:
• Classification problem
• Regression problem

Dr.S.Veena, Associate Professor, SRMIST 7

 Unsupervised Learning
• In unsupervised learning, algorithms learn by themselves without any
supervision or without any target variable provided.
• It is a question of finding hidden patterns and relations in the given
data.
• The categories in unsupervised learning are as follows:
• Dimensionality reduction
• Clustering

Dr.S.Veena, Associate Professor, SRMIST 8

 Reinforcement Learning
• This allows the machine or agent to learn its behavior based on
feedback from the environment.
• In reinforcement learning, the agent takes a series of decisive actions
without supervision and, in the end, a reward will be given, either +1
or -1.
• Based on the final payoff/reward, the agent reevaluates its paths.
• Reinforcement learning problems are closer to the artificial
intelligence methodology rather than frequently used machine
learning algorithms.

Dr.S.Veena, Associate Professor, SRMIST 9

Major differences between statistical
modeling and machine learning

Dr.S.Veena, Associate Professor, SRMIST 10

Major differences between statistical
modeling and machine learning

Dr.S.Veena, Associate Professor, SRMIST 11

Steps in machine learning model
development and deployment

Dr.S.Veena, Associate Professor, SRMIST 12

Steps in machine learning model
development and deployment

Dr.S.Veena, Associate Professor, SRMIST 13

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 14

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 15

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 16

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 17

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 18

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 19

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 20

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 21

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 22

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 23

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 24

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 25

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 26

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 27

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 28

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 29

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 30

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 31

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 32

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 33

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 34

Statistical fundamentals and terminology for
model building and validation
• In statistics, the coefficient of determination, denoted R2 or r2 and
pronounced "R squared", is the proportion of the variation in the
dependent variable that is predictable from the independent variable(s).

Dr.S.Veena, Associate Professor, SRMIST 35

Statistical fundamentals and terminology for
model building and validation
• The Adjusted Coefficient of Determination (Adjusted R-squared) is an adjustment for the
Coefficient of Determination that takes into account the number of variables in a data set. It
also penalizes you for points that don't fit the model. ... Every time you add a data point in
regression analysis, R2 will increase.

Dr.S.Veena, Associate Professor, SRMIST 36

Statistical fundamentals and terminology for
model building and validation
• Maximum Likelihood estimate
Estimating the parameters with maximum likelihood

Dr.S.Veena, Associate Professor, SRMIST 37

Statistical fundamentals and terminology for
model building and validation
• Entropy – measure of impurity in the data
• If the sample is homogenous – entropy =0
• If the sample is equally divided – entropy =1

Dr.S.Veena, Associate Professor, SRMIST 38

Statistical fundamentals and terminology for
model building and validation

Dr.S.Veena, Associate Professor, SRMIST 39

Statistical fundamentals and terminology for
model building and validation
• The information gain is the amount of information gained
about a random variable or signal from observing another
random variable.
• Entropy is a measure of the uncertainty associated with a
random variable.

Dr.S.Veena, Associate Professor, SRMIST 40

Statistical fundamentals and terminology for
model building and validation
Gini index and entropy is the criterion for calculating information gain.
Both gini and entropy are measures of impurity of a node.
A node having multiple classes is impure whereas a node having only one class is pure.

Dr.S.Veena, Associate Professor, SRMIST 41

 Bias versus variance trade-off
• Bias and variance are inversely connected. It is impossible
to have an ML model with a low bias and a low variance. When
a data engineer modifies the ML algorithm to better fit a given
data set, it will lead to low bias—but it will increase variance

Dr.S.Veena, Associate Professor, SRMIST 42

Bias versus variance trade-off

Dr.S.Veena, Associate Professor, SRMIST 43

Bias versus variance trade-off

Dr.S.Veena, Associate Professor, SRMIST 44

Dr.S.Veena, Associate Professor, SRMIST 45
Train and test data

Dr.S.Veena, Associate Professor, SRMIST 46

Train and test data

Dr.S.Veena, Associate Professor, SRMIST 47

 Linear regression versus gradient descent
Linear Regression
• Linear regression analysis is used to predict the value of a
variable based on the value of another variable.
• The variable you want to predict is called the dependent
variable.
• The variable you are using to predict the other variable's value
is called the independent variable.

Dr.S.Veena, Associate Professor, SRMIST 48

 Linear regression versus gradient descent
• Gradient descent is an optimization algorithm used to find the
values of parameters (coefficients) of a function (f) that minimizes a
cost function (cost).
• Gradient descent is an optimization algorithm which is commonly-
used to train machine learning models and neural networks. Training
data helps these models learn over time, and the cost function within
gradient descent specifically acts as a barometer, gauging its
accuracy with each iteration of parameter updates.

Dr.S.Veena, Associate Professor, SRMIST 49

Linear regression

Dr.S.Veena, Associate Professor, SRMIST 50

Linear regression

Dr.S.Veena, Associate Professor, SRMIST 51

Linear regression versus gradient descent

Dr.S.Veena, Associate Professor, SRMIST 52

 Linear regression versus gradient descent
• The simple linear regression model is essentially a linear
equation of the form y = c + b*x; where y is the dependent
variable (outcome), x is the independent variable (predictor), b
is the slope of the line; also known as regression coefficient and
c is the intercept; labeled as constant.

Dr.S.Veena, Associate Professor, SRMIST 53

Linear regression versus gradient descent

Dr.S.Veena, Associate Professor, SRMIST 54

Dr.S.Veena, Associate Professor, SRMIST 55
 Linear regression versus gradient descent
• Mean squared Error calculation
• Mean squared error (MSE) measures the amount of error in
statistical models. It assesses the average squared difference
between the observed and predicted values.

Dr.S.Veena, Associate Professor, SRMIST 56

Dr.S.Veena, Associate Professor, SRMIST 57
 Machine learning losses
• Loss is the penalty for a bad prediction. That is, loss is a number indicating how bad the model's
prediction was on a single example. If the model's prediction is perfect, the loss is zero; otherwise,
the loss is greater.
• The loss function or cost function in machine learning is a function that maps the values of
variables onto a real number intuitively representing some cost associated with the variable values.
• Optimization methods are applied to minimize the loss function by changing the parameter values,
which is the central theme of machine learning.
• Zero-one loss: The simplest loss function is the zero-one loss. It literally counts how many mistakes an hypothesis function
h makes on the training set. For every single example it suffers a loss of 1 if it is mispredicted, and 0 otherwise.

• Zero-one loss is L0-1 = 1 (m <= 0); in zero-one loss, value of loss is 0 for m >= 0 whereas 1 for
m < 0.
• The difficult part with this loss is it is not differentiable, non-convex, and also NP- hard.
• Surrogate losses used for machine learning in place of zero-one loss
• Types are
• Squared loss (for regression)
• Hinge loss (SVM)
• Logistic/log loss (logistic regression)
Dr.S.Veena, Associate Professor, SRMIST 58
 Types of Losses
• Squared loss is a loss function that can be used in the learning setting in which we are predicting a
real-valued variable y given an input variable x.

• The hinge loss is a loss function used for training classifiers, most notably the SVM. ... A negative
distance from the boundary incurs a high hinge loss. This essentially means that we are on the
wrong side of the boundary, and that the instance will be classified incorrectly.

• Log Loss is the most important classification metric based on probabilities.For any given problem, a
lower log loss value means better predictions. Mathematical interpretation: Log Loss is the negative
average of the log of corrected predicted probabilities for each instance. Log-loss is indicative of how
close the prediction probability is to the corresponding actual/true value (0 or 1 in case of
binary classification). The more the predicted probability diverges from the actual value, the higher is
the log-loss value

Dr.S.Veena, Associate Professor, SRMIST 59

Some loss functions are as follows:

Dr.S.Veena, Associate Professor, SRMIST 60

Machine learning losses

Dr.S.Veena, Associate Professor, SRMIST 61

 When to stop tuning machine learning models

• While increasing the complexity of a model, the following stages occur:

• Stage 1: Underfitting stage - high train and high test errors (or low train and low test accuracy)
• Stage 2: Good fit stage (ideal scenario) - low train and low test errors (or high train and high test
accuracy)
• Stage 3: Overfitting stage - low train and high test errors (or high train and low test accuracy)

Dr.S.Veena, Associate Professor, SRMIST 62

When to stop tuning machine learning models

Dr.S.Veena, Associate Professor, SRMIST 63

 Train, validation, and test data
• Cross-validation is not popular in the statistical modeling world for many reasons;
• statistical models are linear in nature and robust, and do not have a high variance/overfitting
problem.
• Hence, the model fit will remain the same either on train or test data, which does not hold true in
the machine learning world.
• Also, in statistical modeling, lots of tests are performed at the individual parameter level apart from
aggregated metrics, whereas in machine learning we do not have visibility at the individual
parameter level

Dr.S.Veena, Associate Professor, SRMIST 64

Train, validation, and test data

Dr.S.Veena, Associate Professor, SRMIST 65

 Train, validation, and test data
>>> import pandas as pd
>>> from sklearn.model_selection import train_test_split
>>> original_data = pd.read_csv("mtcars.csv")
>>> def data_split(dat,trf = 0.5,vlf=0.25,tsf = 0.25):
... nrows = dat.shape[0]
... trnr = int(nrows*trf)
... vlnr = int(nrows*vlf)

The following Python code splits the data into training and the remaining
data. The remaining data will be further split into validation and test datasets:
 tr_data,rmng = train_test_split(dat,train_size =
trnr,random_state=42)
... vl_data, ts_data =
train_test_split(rmng,train_size =
vlnr,random_state=45)
... return (tr_data,vl_data,ts_data)
 
Dr.S.Veena, Associate Professor, SRMIST 66
 Train, validation, and test data
Implementation of the split function on the original data to create three datasets (by 50 percent, 25
percent, and 25 percent splits) is as follows:

>>> train_data, validation_data, test_data = data_split (original_data

,trf=0.5, vlf=0.25,tsf=0.25)
 

Dr.S.Veena, Associate Professor, SRMIST 67

 Cross-validation

• Cross-validation is a resampling method that uses different

portions of the data to test and train a model on different
iterations. It is mainly used in settings where the goal is
prediction, and one wants to estimate how accurately a
predictive model will perform in practice.

Dr.S.Veena, Associate Professor, SRMIST 68

 Cross-validation
• Example: In five-fold cross-validation, data will be divided into five parts, subsequently trained on
four parts of the data, and tested on the one part of the data. This process will run five times, in
order to cover all points in the data. Finally, the error calculated will be the average of all the errors

Dr.S.Veena, Associate Professor, SRMIST 69

 Grid search
• Grid-searching is the process of scanning the data to configure
optimal parameters for a given model. Depending on the type of model
utilized, certain parameters are necessary.
• Grid-searching can be applied across machine learning to calculate the
best parameters to use for any given model.
• Grid search in machine learning is a popular way to tune the
hyperparameters of the model in order to find the best
combination for determining the best fit

Dr.S.Veena, Associate Professor, SRMIST 70

 Grid search
• Grid search has been implemented using a decision tree classifier for classification purposes.
• Tuning parameters are the depth of the tree, the minimum number of observations in terminal
node, and the minimum number of observations required to perform the node split
# Grid search
>>> import pandas as pd
>>> from sklearn.tree import DecisionTreeClassifier
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.metrics import classification_report,confusion_matrix,accuracy_score
>>> from sklearn.pipeline import Pipeline
>>> from sklearn.grid_search import GridSearchCV
 
>>> input_data = pd.read_csv("ad.csv",header=None)
 
>>> X_columns = set(input_data.columns.values)
>>> y = input_data[len(input_data.columns.values)-1]
>>> X_columns.remove(len(input_data.columns.values)-1)
>>> X = input_data[list(X_columns)]

Dr.S.Veena, Associate Professor, SRMIST 71

 Grid search
Split the data into train and testing:
• >>> X_train, X_test,y_train,y_test = train_test_split(X,y,train_size
= 0.7,random_state=33)
•Create a pipeline to create combinations of variables for the grid search:
• >>> pipeline = Pipeline([
• ... ('clf', DecisionTreeClassifier(criterion='entropy')) ])
 
Combinations to explore are given as parameters in Python dictionary format:
• >>> parameters = {
• ... 'clf max_depth': (50,100,150),
• ... 'clf min_samples_split': (2, 3),
• ... 'clf min_samples_leaf': (1, 2, 3)}

Dr.S.Veena, Associate Professor, SRMIST 72

 Grid search
The n_jobs field is for selecting the number of cores in a computer; -1 means it uses all the cores in
the computer. The scoring methodology is accuracy, in which many other options can be chosen,
such as precision, recall, and f1:
• >>> grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1,
verbose=1, scoring='accuracy')
• >>> grid_search.fit(X_train, y_train)

 
Predict using the best parameters of grid search:
• >>> y_pred = grid_search.predict(X_test)

Dr.S.Veena, Associate Professor, SRMIST 73

 Grid search
•The output is as follows:
• >>> print ('\n Best score: \n', grid_search.best_score_)
• >>> print ('\n Best parameters set: \n')
• >>> best_parameters = grid_search.best_estimator_.get_params()
• >>> for param_name in sorted(parameters.keys()):
• >>> print ('\t%s: %r' % (param_name,
best_parameters[param_name]))
• >>> print ("\n Confusion Matrix on Test data
• \n",confusion_matrix(y_test,y_pred))
• >>> print ("\n Test Accuracy \
n",accuracy_score(y_test,y_pred))
• >>> print ("\nPrecision Recall f1 table \
n",classification_report(y_test, y_pred))

Dr.S.Veena, Associate Professor, SRMIST 74

Grid search

Dr.S.Veena, Associate Professor, SRMIST 75

 Machine learning model overview

• Supervised learning: This is where an instructor provides feedback to a student on whether they have performed well in an examination or not. In
which target variable do present and models do get tune to achieve it. Many machine learning methods fall in to this category
• Classification problems
• Logistic regression
• Lasso and ridge regression
• Decision trees (classification trees)
• Bagging classifier
• Random forest classifier
• Boosting classifier (adaboost, gradient boost, and xgboost)
• SVM classifier
• Recommendation engine
• Regression problems
• Linear regression (lasso and ridge regression)
• Decision trees (regression trees)
• Bagging regressor
• Random forest regressor
• Boosting regressor - (adaboost, gradient boost, and xgboost)
• SVM regressor

Dr.S.Veena, Associate Professor, SRMIST 76

 Machine learning model overview

• Unsupervised learning: Similar to the teacher-student analogy, in which the instructor does not
present and provide feedback to the student and who needs to prepare on his/her own.
Unsupervised learning does not have as many are in supervised learning:
• Principal component analysis (PCA)
• K-means clustering

• Reinforcement learning: This is the scenario in which multiple decisions need to be taken by an
agent prior to reaching the target and it provides a reward, either +1 or -1, rather than notifying
how well or how badly the agent performed across the path
• Markov decision process
• Monte Carlo methods
• Temporal difference learning

Dr.S.Veena, Associate Professor, SRMIST 77

 Machine learning model overview

Logistic regression:
• This is the problem in which outcomes are discrete classes rather than continuous values.
• For example, a customer will arrive or not, he will purchase the product or not, and so on.
• In statistical methodology, it uses the maximum likelihood method to calculate the parameter of
individual variables.
• In contrast, in machine learning methodology, log loss will be minimized with respect to β coefficients
(also known as weights).
• Logistic regression has a high bias and a low variance error
Linear regression:
• This is used for the prediction of continuous variables such as customer income and so on.
• It utilizes error minimization to fit the best possible line in statistical methodology.
• However, in machine learning methodology, squared loss will be minimized with respect to β coefficients.
• Linear regression also has a high bias and a low variance error

Dr.S.Veena, Associate Professor, SRMIST 78

 Machine learning model overview

Lasso and ridge regression:

• This uses regularization to control overfitting issues by applying a penalty on coefficients.
• In ridge regression, a penalty is applied on the sum of squares of coefficients
• In lasso, a penalty is applied on the absolute values of the coefficients.
• Ridge regression tries to minimize the magnitude of coefficients, whereas lasso tries to eliminate them
Decision trees:
• Recursive binary splitting is applied to split the classes at each level to classify observations to their purest
class.
• The classification error rate is simply the fraction of the training observations in that region that do not
belong to the most common class.
• Decision trees have an overfitting problem due to their high variance in a way to fit
• Pruning is applied to reduce the overfitting problem by growing the tree completely.
• Decision trees have low a bias and a high variance error

Dr.S.Veena, Associate Professor, SRMIST 79

 Machine learning model overview

Bagging:
• This is an ensemble technique applied on decision trees in order to minimize the variance error and
at the same time not increase the error component due to bias.
• In bagging, various samples are selected with a subsample of observations and all variables
(columns), subsequently fit individual decision trees independently on each sample and later
ensemble the results by taking the maximum vote (in regression cases, the mean of outcomes
calculated
Random forest:
• In bagging, all the variables/columns are selected for each sample, whereas in random forest a few
subcolumns are selected.
• The reason behind the selection of a few variables rather than all was that during each independent
tree sampled, significant variables always came first in the top layer of splitting which makes all the
trees look more or less similar and defies the sole purpose of ensemble:
• Random forest has both low bias and variance errors

Dr.S.Veena, Associate Professor, SRMIST 80

 Machine learning model overview

Boosting:
• This is a sequential algorithm that applies on weak classifiers such as a decision stump (a one-level
decision tree or a tree with one root node and two terminal nodes) to create a strong classifier by
ensembling the results.
• The algorithm starts with equal weights assigned to all the observations, followed by subsequent
iterations where more focus was given to misclassified observations by increasing the weight of
misclassified observations and decreasing the weight of properly classified observations.
• In the end, all the individual classifiers were combined to create a strong classifier. Boosting might
have an overfitting problem, but by carefully tuning the parameters, we can obtain the best of the self
machine learning model
Support vector machines (SVMs):
• This maximizes the margin between classes by fitting the widest possible hyperplane between them.
• In the case of non-linearly separable classes, it uses kernels to move observations into higher-
dimensional space and then separates them linearly with the hyperplane there

Dr.S.Veena, Associate Professor, SRMIST 81

 Machine learning model overview

Recommendation engine:
• This utilizes a collaborative filtering algorithm to identify high-probability items to its respective
users, who have not used it in the past, by considering the tastes of similar users who would be
using that particular item.
• It uses the alternating least squares (ALS) methodology to solve this problem

Principal component analysis (PCA):

• This is a dimensionality reduction technique in which principal components are calculated in place
of the original variable.
• Principal components are determined where the variance in data is maximum; subsequently, the
top n components will be taken by covering about 80 percent of variance and will be used in further
modeling processes, or exploratory analysis will be performed as unsupervised learning

Dr.S.Veena, Associate Professor, SRMIST 82

 Machine learning model overview

K-means clustering:
• This is an unsupervised algorithm that is mainly utilized for segmentation exercise.
• K-means clustering classifies the given data into k clusters in such a way that, within the cluster,
variation is minimal and across the cluster, variation is maximal

Markov decision process (MDP):

• In reinforcement learning, MDP is a mathematical framework for modeling decision-making of an
agent in situations or environments where outcomes are partly random and partly under control.
• In this model, environment is modeled as a set of states and actions that can be performed by an
agent to control the system's state. The objective is to control the system in such a way that the
agent's total payoff is maximized

Dr.S.Veena, Associate Professor, SRMIST 83

 Machine learning model overview

Monte Carlo method:

• Monte Carlo methods do not require complete knowledge of the environment, in contrast with MDP.
• Monte Carlo methods require only experience, which is obtained by sample sequences of states, actions,
and rewards from actual or simulated interaction with the environment.
• Monte Carlo methods explore the space until the final outcome of a chosen sample sequences and update
estimates accordingly
Temporal difference learning:
• Temporal difference is a combination of both Monte Carlo and dynamic programming ideas.
• Similar to Monte Carlo, temporal difference methods can learn directly from raw experience without a
model of the environment's dynamics.
• Like dynamic programming, temporal difference methods update estimates based in part on other
learned estimates, without waiting for a final outcome.
• Temporal difference is the best of both worlds and is most commonly used in games such as AlphaGo
and so on

Dr.S.Veena, Associate Professor, SRMIST 84