Linear Regression

Linear Regression Algorithms Agenda

1. What is Machine Learning ?

2. Linear Regression & Algorithm

3. Algebra & Geometry

4. Linear Relationship with an Example

5. Evaluating the performance of Linear Models

6. Evaluation Metrics

7. Regularization Techniques

8. Bias Variance Trade-off

 Machine Learning Machine Learning process flow
 Subset of AI

 Enables system to learn from Data

Machine Learning

 Focuses on Dev. of Algos to identify

patterns, make predictions, extract
insights
ML in Data Science Types of Machine Learning
 Predictive Analytics

 Pattern Recognition

 Classification & Regression

 Optimization

 Automation
 Linear regression

Statistical method used to model the relationship between a dependent variable and
one or more independent variables.

The goal is to find the linear equation that best predicts the dependent variable from
Linear Regression

the independent variables.

1. Simple Linear Regression:

Involves one independent variable and one dependent variable. The relationship is
modeled using a straight line.

2. Multiple Linear Regression:

Involves two or more independent variables. The relationship is still linear, but it
extends into multiple dimensions.

Algorithm :

 A well-defined sequence of steps or instructions designed to perform a specific task or

solve a particular problem.

 Fundamental to CS & mathematics and are used in various fields to process data,
automate tasks, and solve complex problems efficiently.
 Characteristics equation of Geometries

Straight Line - Y = mX + C

Circle - X 2 + Y2 = R 2
Algebra & Geometry

- (X – h)2 + (Y – k)2 = R2

Parabola - Y2 = 4pX

- (Y-k)2 = 4p(X-h)

Ellipse - (X/a)2 + (Y/b)2 = 1

 Simple Linear Regression

Given n data points (x1,y1), (x2,y2),..., (xn,yn). The goal is to find a line y=mx+c that best fits the data.

 Calculate the Means: Compute the mean of x (x̄) and the mean of y (ȳ).
Linear Relationship

 Calculate the Slope (m):

 Calculate the Intercept (c) :

 Form the Regression Equation y = mx + c

Multiple Linear Regression

For multiple predictors, the model is y = b0 + b1 x1 +b 2x2 + ... + bkxk​.
1.Formulate the Design Matrix (X): This matrix includes all independent variables.
2.Compute the Coefficients (b) using the normal equation:

where X is the design matrix and y is the vector of observed values.

 Example of Linear Relationship
Suppose you are analyzing the relationship between the number of hours a student studies
(independent variable x) and their score on a test (dependent variable Y).

Calculation :
Linear Relationship

1. Calculate the Means

2. Calculate the Slope (m):

m = 7.5

3. Calculate the Intercept (C):

C = 41.5

4. Final Equation of the line (Characteristics)

Y = 7.5 x + 41.5
 Interpretation

• Slope (m = 7.5): For every additional hour studied, the test score increases by 7.5 points.

• Intercept (c = 41.5): If a student studies for 0 hours, the predicted test score is 41.5 points.
Linear Relationship

Using the Equation for Predictions

Let's predict the test score for a student who studies for 6 hours (x = 6) :

Y = 7.5 × 6 + 41.5 = 45 + 41.5

Y = 86.5

So, a student who studies for 6 hours is predicted to score 86.5 points on the test.
 1. Train Test Split
Description: Split into two : one for training the model & other for testing its perf.
Purpose: Helps evaluate how well the model can predict unseen data.
Evaluating Linear Models

2. Cross-Validation
Description: Involves partitioning the dataset into several subsets (folds) &
training/testing the model multiple times, each time using a different fold as the test
set.
Purpose: Provides a more reliable estimate of model performance and reduces the
risk of overfitting.
3. Residual Analysis
Description: Analyze the residuals (the differences between predicted and actual
values) to check for patterns.
Purpose: Residuals should be randomly distributed with no discernible pattern,
indicating that the model is a good fit.
•Residual Plots: Plot residuals against predicted values to check for homoscedasticity
(constant variance).
 4. Model Assumptions Check
Linear regression relies on several assumptions that should be checked:
Evaluating Linear Models

•Linearity: The relationship between predictors and the outcome should be linear.

•Independence: Observations should be independent of each other.

•Homoscedasticity: The variance of residuals should be constant across all levels of

the independent variables.

•Normality: Residuals should be approximately normally distributed.

•No Multicollinearity: Indepen. variables are not highly correlated with each other.

5. Feature Importance and Coefficients

Description: Examine the coefficients of the model to understand the importance of
each feature.

Purpose: Identifies which variables have the most influence on the predicted
outcome.
 1. MAE (Mean Absolute Error)

Measures average abs. diff. b/n predicted & actual values.

2. MSE (Mean Squared Error)

Evaluation Metrics

Measures average squared diff b/n predicted & actual values, penalizing larger errors
more than MAE.

3. RMSE (Root Mean Squared Error)

Provides error in the same units as the target variable, making it more interpretable.

4. R2 (R Squared)

Represents the proportion of variance in the dependent variable that can be explained
by the independent variables. Ranges from 0 to 1.
 Regularization Types
 Ridge and Lasso Regression are two regularization techniques used to address the
problem of multicollinearity and overfitting in linear regression models.
Regularization Techniques

 They add a penalty term to the ordinary least squares (OLS) objective function,
encouraging simpler models that generalize better.
Ridge Regression (L2 Regularization)
 Adds a penalty equivalent to the square of the magnitude of coefficients to the
loss function.
 It shrinks the coefficients, but never exactly to zero, which means it includes all
predictors in the model.
Characteristics :
Colsed-form solution  Shrinks coefficients

Ridge Objective  Used upon Multicollinearity

 Optimization : Closed-form
solution
 Bias-variance trade-off

(Introducing bias & red. Var.)

 Lasso Regression (L1 Regularization)
 Adds a penalty equivalent to the absolute value of the magnitude of coefficients
to the loss function.
Regularization Techniques

 It can shrink some coefficients to exactly zero, effectively performing feature

selection.

Characteristics :
 Shrinks coefficients : Performs feature selection and shrinks some coefficients to ‘0’
 Used when : We believe some predictors are irrelevant and want to simplify the model
 Bias-variance trade-off : Introducing bias by significantly reducing Variance)
 Optimization : Requires iterative algorithms
 Lasso Regression (L1 Regularization)
 Adds a penalty equivalent to the absolute value of the magnitude of coefficients
to the loss function.
Regularization Techniques

 It can shrink some coefficients to exactly zero, effectively performing feature

selection.

Characteristics :
 Shrinks coefficients : Performs feature selection and shrinks some coefficients to ‘0’
 Used when : We believe some predictors are irrelevant and want to simplify the model
 Bias-variance trade-off : Introducing bias by significantly reducing Variance)
 Optimization : Requires iterative algorithms
 Bias Variance Trade-off
 Trade off between two type of error viz., Bias Error & Variance Error which affects the
performance of the model
Bias Variance Trade-off

Trade-off

The trade-off between bias and variance can be summarized as follows:

 Low Bias, High Variance: Complex models (e.g., deep neural networks) that fit the training
data very well but perform poorly on unseen data due to capturing noise. Overfitting

 High Bias, Low Variance: Simple models (e.g., linear regression) that may not fit the
training data closely but generalize better to unseen data. Underfitting

 Low Bias, Low Variance: A Good model / A good Errors Trade-off

 Bias refers to the error introduced by approximating a real-world problem, which may be
complex, by a simplified model. High bias can lead to:
• Underfitting: The model is too simple to capture the underlying patterns in the data,
resulting in poor performance on both training and test sets.
Variance
Bias Variance Trade-off

Variance refers to the model's sensitivity to fluctuations in the training data. High variance
can lead to:
• Overfitting: The model captures noise and random fluctuations in the training data rather
than the true underlying pattern, resulting in excellent performance on the training set but
poor generalization to new data.