In [1]: import numpy as np

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

In [2]: plt.rcParams['figure.figsize'] = [19, 8]

In [3]: import warnings

warnings.filterwarnings('ignore')

Variables
A variable is nothing but data. In a dataframe the columns are called variables.

The variables are of 2 types:

Dependent Variable
Independent Variable

Dependent Variable:

The variable whose value is to be forecasted or predicted is the dependent variable.

The value of the dependent variable is dependent on the values of the independent
variables.

Dependent variables are also called response variables or target variables.

Dependent variables are generally represented by the letter y.

Independent Variables

Independent variables are used to explain the dependent variable.

Independent variables do not depend on any other variable.

Independent variables are also called feature variables or regressors.

Independent variables are represented by the letter x.

Regression
Regression is a statistical approach to analyze the relationship between a dependent
variable and one or more independent variables.

Regression is used to determine the most suitable function that characterizes the
relationship between dependent and independent variables.

Linear Regression
 Linear regression is a machine learning algorithm that depends on the linear relationship
between a dependent variable and one or more independent variables.

Two variables are in linear relationship when their values can be represented using a
straight line.

Simple Linear Regression

A simple linear regression model has one independent variable that has a linear
relationship with the dependent variable.

For more than one independent variables, it is called Multiple Linear regression.

The Linear Equation

In mathematics, a straight line is represented by the equation: y = mx + c, where m is
slope and c is a constant.

Using the above equation, the value of y (dependent variable) can be calculated using x
(independent variable).

In statistics, the equation is written as: y = β0 + β1 x + ϵ.

Here β0 is the intercept, β1 is the slope and ϵ is the error component.

Slope β1 = δy/δx.

Intercept β0 = Distance on y axis when value of x is 0.

The error component (term), ϵ, represents the distance of the actual value from the value
predicted by the regression line.

When the relationship between the independent variable (x) and the dependent variable
(y) is linear, Linear Regression model is applied to analyze the data.

Example

In [4]: data = {'voltage': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],

'current': [0, 1, 2, 2.99, 4, 5, 6, 6.99, 8, 9, 10]
}

In [5]: df = pd.DataFrame(data)

In [6]: df
Out[6]: voltage current

0 0 0.00

1 1 1.00

2 2 2.00

3 3 2.99

4 4 4.00

5 5 5.00

6 6 6.00

7 7 6.99

8 8 8.00

9 9 9.00

10 10 10.00

In [7]: plt.scatter(data=df, x="voltage", y="current", s=200)

plt.title("Voltage Vs Current")
plt.xlabel("Voltage")
plt.ylabel("Current")
plt.show()

Example

In [8]: homeprice = {'area': [2600, 3000, 3200, 3600, 4000], 'price': [550000, 565000, 6

In [9]: df = pd.DataFrame(homeprice)

In [10]: df
Out[10]: area price

0 2600 550000

1 3000 565000

2 3200 610000

3 3600 680000

4 4000 725000

In [11]: plt.scatter(data=df, x='area', y='price', s=200)

plt.title("House Area Vs Price")
plt.xlabel("Area")
plt.ylabel("Price")
plt.show()

From the above plot, the data points can be connected more or less using a straight line.
Hence we can apply Simple Linear regression machine learning model on the above data.

Ordinary Least Square Method

The regression line that best explains the trend in the data is the best fit line.

The line may pass through all of the points or non of the points.

The method aims at minimizing the sum of squares of the error terms i.e., it determines
the values of β0 and β1 at which the error terms are minimum.

β0 = ȳ − β1 x̄

β1 = Cov(x, y)/V ar(x)

β1 = ∑((x − x̄)(y − ȳ))/∑(x − x̄)2

Measures of Variation

Sum of Squares Total (SST)

 The sum of squares of total is the sum of squared difference between the target variable
and its mean value.

It is also known as Total Sum of Squares (TSS).

SST = ∑(y − ȳ)2

Sum of Squares Regression (SSR)

The sum of squares regression is the sum of squared difference between the predicted
value and the mean of the target variable.

It is also known as Regression Sum of Squares (RSS).

SSR = ∑(ypredict − ȳ)2

Sum of Squares of Error (SSE)

The sum of squares of error is the sum of squared difference between target variable and
its predicted value.

It is also known as Error Sum of Square.

SSE = ∑(y − ypredict )2

Total Variation

SST = SSR + SSE

Coefficient of Determination R2

Gives the total percentage of variation that is explained by the target variable.

R2 = SSR/SST

R2 = 1 − (SSE/SST )

Note: 0≤R2 ≤1

Case Study

Predict the Salary of an employee depending on the Years of Experience

Read the Data

In [12]: salary_df = pd.read_csv("./datasets/Salary_Data.csv")

Exploratory Data Analysis

In [13]: salary_df.shape
Out[13]: (30, 2)

In [14]: salary_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 30 entries, 0 to 29
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 YearsExperience 30 non-null float64
1 Salary 30 non-null int64
dtypes: float64(1), int64(1)
memory usage: 612.0 bytes

In [15]: salary_df.head()

Out[15]: YearsExperience Salary

0 1.1 39343

1 1.3 46205

2 1.5 37731

3 2.0 43525

4 2.2 39891

In [16]: salary_df.describe()

Out[16]: YearsExperience Salary

count 30.000000 30.000000

mean 5.313333 76003.000000

std 2.837888 27414.429785

min 1.100000 37731.000000

25% 3.200000 56720.750000

50% 4.700000 65237.000000

75% 7.700000 100544.750000

max 10.500000 122391.000000

Data Visualization

In [17]: plt.scatter(data=salary_df, x='YearsExperience', y='Salary', s=200)

plt.title("Salary based on the Years of experience")
plt.xlabel("Years of Experience")
plt.ylabel("Salary")
plt.show()
 The data shows linear relationship between YearsofExperience and Salary. Hence we can
use Simple Linear Regression.

Independent Variable: YearsExperience

Dependent Variable: Salary

In [18]: X = salary_df.loc[:, 'YearsExperience'].values

y = salary_df.loc[:, 'Salary'].values

Split the Data into Training and Test Data

The data will be divided into 2 parts: for training the model and for testing purpose.

Generally 70% of the rows are used for training and reamaining 30% for testing the
model.

In [19]: from sklearn.model_selection import train_test_split

In [20]: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_

In [21]: X_train.shape, X_test.shape, y_train.shape, y_test.shape

Out[21]: ((24,), (6,), (24,), (6,))

In [22]: X_train.reshape(-1, 1).shape

Out[22]: (24, 1)

Train the model

In [23]: from sklearn.linear_model import LinearRegression

In [24]: reg_model = LinearRegression()

In [25]: reg_model.fit(X_train.reshape(-1, 1), y_train.reshape(-1, 1))

Out[25]: ▾ LinearRegression i ?

LinearRegression()

Find the coefficient m i.e., slope

In [26]: reg_model.coef_

Out[26]: array([[9312.57512673]])

Find the intercept i.e., c

In [27]: reg_model.intercept_

Out[27]: array([26780.09915063])

Predict the Salary for the test data

In [28]: y_predicted = reg_model.predict(X_test.reshape(-1, 1))

In [29]: y_predicted

Out[29]: array([[ 40748.96184072],

[122699.62295594],
[ 64961.65717022],
[ 63099.14214487],
[115249.56285456],
[107799.50275317]])

In [30]: y_test

Out[30]: array([ 37731, 122391, 57081, 63218, 116969, 109431], dtype=int64)

Model Evaluation

In [31]: from sklearn.metrics import r2_score

The R Squared Value/R2 Score

In [32]: r_square = r2_score(y_test.reshape(-1, 1), y_predicted)

In [33]: r_square

Out[33]: 0.988169515729126

In [34]: print(f"Accuracy = {r_square:.2%}")

Accuracy = 98.82%

Data Visualization

In [35]: plt.scatter(x=X_test, y=y_test, color='red', s=300)

plt.scatter(x=X_test, y=y_predicted, color='green', s=300)
plt.title("Salary Test Data Vs Predicted")
plt.xlabel("Years of Experience")
plt.ylabel("Salary")
 plt.legend(["Test Value", "Predicted Value"], loc="lower right")

plt.show()

In [36]: sns.lmplot(data=salary_df, x='YearsExperience', y='Salary')

plt.show()

Multiple Linear Regression

Simple Linear Regression model takes only one independent variable to predict the
dependent variable value.
 Multiple Linear Regression model uses multiple independent variable to predict the
dependent variable value.

The equation of a Multiple Linear Regression model will be of the form:

y = m1 x1 + m2 x2 + m3 x3 +. . . +b

Here y is the dependent or target variable, x1 , x2 , x3 are independent variables,

m1 , m2 , m3 are coefficients of the independent variables and b is the intercept.

The relationship between y and x1 , x2 , x3 should be linear.

Case Study

Home price data prediction using area, no. of bedrooms, and age

Read the data

In [37]: homeprice_df = pd.read_csv('./datasets/homeprices.csv')

Exploratory Data Analysis

In [38]: homeprice_df.shape

Out[38]: (6, 4)

In [39]: homeprice_df.head()

Out[39]: area bedrooms age price

0 2600 3.0 20 550000

1 3000 4.0 15 565000

2 3200 NaN 18 610000

3 3600 3.0 30 595000

4 4000 5.0 8 760000

In [40]: homeprice_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6 entries, 0 to 5
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 area 6 non-null int64
1 bedrooms 5 non-null float64
2 age 6 non-null int64
3 price 6 non-null int64
dtypes: float64(1), int64(3)
memory usage: 324.0 bytes

In [41]: homeprice_df.duplicated().sum()
Out[41]: 0

In [42]: homeprice_df['bedrooms'].skew()

Out[42]: 0.0

In [43]: homeprice_df['bedrooms'].fillna(homeprice_df['bedrooms'].mean(), inplace=True)

In [44]: homeprice_df.isnull().sum()

Out[44]: area 0
bedrooms 0
age 0
price 0
dtype: int64

Data Visualization

In [45]: sns.pairplot(data=homeprice_df)

plt.show()
 Model Training

In [46]: X = homeprice_df.loc[:, 'area':'age'].values

y = homeprice_df['price'].values

In [47]: from sklearn.linear_model import LinearRegression

In [48]: linear_model = LinearRegression()

In [49]: linear_model.fit(X, y)

Out[49]: ▾ LinearRegression i ?

LinearRegression()

In [50]: linear_model.coef_

Out[50]: array([ 142.895644 , -48591.66405516, -8529.30115951])

In [51]: linear_model.intercept_

Out[51]: 485561.8928233979

In [52]: y_predict = linear_model.predict(X)

y_predict

Out[52]: array([540729.55186458, 591942.6512065 , 594933.87652772, 598332.18426826,

745951.73926671, 803109.99686623])

Model Evaluation

In [53]: from sklearn.metrics import r2_score

In [54]: r2_score(y, y_predict)

Out[54]: 0.9760698937818199

In [55]: linear_model.score(X, y)

Out[55]: 0.9760698937818199

Polynomial Linear Regression

Simple and Multiple Linear Regression models are used only when the data points are in
a straight line.

It is not always possible to expect the data points to be in the straight line. They may
form a curved line.

Polynomial Linear Regression is used on non-linear data.

Polynomial Linear Regression is a special case of Multiple Linear Regression.

In Multiple Linear Regression, each term is without any power value but in Polynomial
Linear Regression, each term is raised to a power.

The first term has power 1, the second term has power 2, and the nth term has the power
n.

The equation of a Multiple Linear Regression model will be of the form:

y = m1 x1 + m2 x2 2 + m3 x3 3 +. . . +b

Here y is the dependent or target variable, x1 , x2 , x3 are independent variables,

m1 , m2 , m3 are coefficients of the independent variables and b is the intercept.

Case Study

Bias and Variance

While building a machine learning model, the data is divided into 2 parts:

training data, to train the model

test data, to test the model

A machine learning model should be able to yield correct results with both training and
test data. Then only the model is accurate.

While using a machine learning model, the training data points would have a lot of
deviations from the regression line. This shows the inability of the model to accurately
represent the relationship between the data points. This is called bias.

Bias represents the deviations from the original data and predicted values by the model.

On the other hand, the model may fit the test data accurately on the regression line.

The difference in fits between the datasets is called variance.

When the model works well on train data, it represents low bias.

When the model failed on test (new) data, it represents high variance.

Bias and variance are inversely proportional to each other. What is needed is the low bias
and low variance.

If the model shows high bias and low variance, it is called underfitting.

If the model shows low bias and high variance, it is called overfitting.

Sum of Squared Residual

The Simple Linear Regression model uses a straight line to fit the data.
There are chances of some deviations from the straight line to the actual data. These
deviations must be minimized.

To minimize the deviations, calculate the sum of squares of deviations and add that to
the equation of the line.

The equation of the linear regression line is given by:

y = β0 + β1 x + ϵ

ϵ = y − β0 − β1 x

ϵ = y − (β0 + β1 x)

ϵ = yactual − ypredict

The error term exist for every observation in the data.

Squared Error:

ϵ2i = (yactual − ypredict )2

Sum of Squared Error:

Sum of Squared Error = ∑ϵ2i

Why to take the square of the deviations?

When deviations are taken alone, the positive and negative deviations may cancel out.
But when the deviations are squared, the negative deviation becomes positive and add
to the overall deviation.

Regularization
Regularization is a technique to minimize the variance or overfitting to achieve better
accuracy.

When regularization is used in Linear Regression model, there would be other variations
like:

Ridge Regression
Lasso Regression
ElasticNET Regression

The above 3 models are developed to minimize the variance.

Ridge Regression
To control the impact on bias and variance, a regularization parameter by name λ is
used.
When λ value is increased, it reduces the variance. But if the value of λ is increased too
much, it will increase bias also. Hence it is importance to tune λ to the correct value to
keep both the bias and the variance low.

The regularization pameter λ is also known as the penalty parameter.

2 2
Ridge Regression = β0 + β1 x + ∑ϵi + λβ1

The term λ X square of the slope is called L2 Penalty term.

λ value can be anything from 0 to positive infinity.

The addition of penalty term is called Regularization.

When λ is 0, the Ridge Regression will be equal to Simple Linear Regression. Hence
Simple Linear Regression and Ridge Regression can be used on any (train or test) of the
datasets.

Ridge regularization is used when there is a lot of features in the dataset and all features
have small coefficients.

Lasso Regression
LASSO stands for Least Absolute Shrinkage and Selection Operator.

Lasso Regression is same as Ridge Regression except that the penalty term is calculated
by: λ X Modulus of slope.

2
Lasso Regression equation = β0 + β1 x + ∑ϵi + λ|β1 |

Lasso Regression has L1 Penalty term. L1 indicates λ|β1 |.

Lasso Regression offers some bias but very low variance. Hence predictions will be more
accurate than Ridge Regression.

Ridge Regression considers all the columns in the dataset to predict the output. But
Lasso Regression selects only the columns that influence maximum on the output. Hence
Lassor Regression is generally used in feature selection.

ElasticNET Regression
ElasticNET Regression is used when a dataset has large number of columns and large
volume of data.

ElasticNET Regression is a combination of Ridge and Lasso Regressions.

2 2
ElasticNET Regression equation = β0 + β1 x + ∑ϵi + +λ1 β1 + λ2 |β1 |

Case Study
In [56]: houseprice_df = pd.read_csv("./datasets/Melbourne_housing_FULL.csv")

In [57]: houseprice_df.shape

Out[57]: (34857, 21)

In [58]: houseprice_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 34857 entries, 0 to 34856
Data columns (total 21 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Suburb 34857 non-null object
1 Address 34857 non-null object
2 Rooms 34857 non-null int64
3 Type 34857 non-null object
4 Price 27247 non-null float64
5 Method 34857 non-null object
6 SellerG 34857 non-null object
7 Date 34857 non-null object
8 Distance 34856 non-null float64
9 Postcode 34856 non-null float64
10 Bedroom2 26640 non-null float64
11 Bathroom 26631 non-null float64
12 Car 26129 non-null float64
13 Landsize 23047 non-null float64
14 BuildingArea 13742 non-null float64
15 YearBuilt 15551 non-null float64
16 CouncilArea 34854 non-null object
17 Lattitude 26881 non-null float64
18 Longtitude 26881 non-null float64
19 Regionname 34854 non-null object
20 Propertycount 34854 non-null float64
dtypes: float64(12), int64(1), object(8)
memory usage: 5.6+ MB

In [59]: houseprice_df.head()
Out[59]: Suburb Address Rooms Type Price Method SellerG Date Distan

68 Studley
0 Abbotsford 2 h NaN SS Jellis 3/09/2016
St

85 Turner
1 Abbotsford 2 h 1480000.0 S Biggin 3/12/2016
St

25
2 Abbotsford Bloomburg 2 h 1035000.0 S Biggin 4/02/2016
St

18/659
3 Abbotsford 3 u NaN VB Rounds 4/02/2016
Victoria St

5 Charles
4 Abbotsford 3 h 1465000.0 SP Biggin 4/03/2017
St

5 rows × 21 columns

 

In [60]: houseprice_df = houseprice_df.loc[:, ['Suburb', 'Rooms', 'Type', 'Method', 'Sell

'Bedroom2', 'Bathroom', 'Car', 'Landsize',
'Regionname', 'Propertycount',
'Price']]

In [61]: houseprice_df.shape

Out[61]: (34857, 15)

In [62]: houseprice_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 34857 entries, 0 to 34856
Data columns (total 15 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Suburb 34857 non-null object
1 Rooms 34857 non-null int64
2 Type 34857 non-null object
3 Method 34857 non-null object
4 SellerG 34857 non-null object
5 Distance 34856 non-null float64
6 Bedroom2 26640 non-null float64
7 Bathroom 26631 non-null float64
8 Car 26129 non-null float64
9 Landsize 23047 non-null float64
10 BuildingArea 13742 non-null float64
11 CouncilArea 34854 non-null object
12 Regionname 34854 non-null object
13 Propertycount 34854 non-null float64
14 Price 27247 non-null float64
dtypes: float64(8), int64(1), object(6)
memory usage: 4.0+ MB

In [63]: houseprice_df['Distance'].fillna(0, inplace=True)

houseprice_df['Bedroom2'].fillna(0, inplace=True)
 houseprice_df['Bathroom'].fillna(0, inplace=True)

houseprice_df['Car'].fillna(0, inplace=True)

houseprice_df['Propertycount'].fillna(0, inplace=True)

In [64]: houseprice_df['Landsize'].fillna(houseprice_df['Landsize'].mean(), inplace=True)

houseprice_df['BuildingArea'].fillna(houseprice_df['BuildingArea'].mean(), inpla

In [65]: houseprice_df.dropna(inplace=True)

In [66]: houseprice_df.isnull().sum()

Out[66]: Suburb 0
Rooms 0
Type 0
Method 0
SellerG 0
Distance 0
Bedroom2 0
Bathroom 0
Car 0
Landsize 0
BuildingArea 0
CouncilArea 0
Regionname 0
Propertycount 0
Price 0
dtype: int64

In [67]: houseprice_df.duplicated().sum()

Out[67]: 50

In [68]: houseprice_df = pd.get_dummies(data=houseprice_df, drop_first=True, dtype=np.int

In [69]: houseprice_df.head()

Out[69]:
Rooms Distance Bedroom2 Bathroom Car Landsize BuildingArea Propertycount

1 2 2.5 2.0 1.0 1.0 202.0 160.2564 4019.0

2 2 2.5 2.0 1.0 0.0 156.0 79.0000 4019.0

4 3 2.5 3.0 2.0 0.0 134.0 150.0000 4019.0

5 3 2.5 3.0 2.0 1.0 94.0 160.2564 4019.0

6 4 2.5 3.0 1.0 2.0 120.0 142.0000 4019.0

5 rows × 745 columns

 

In [70]: y = houseprice_df['Price']
 X = houseprice_df.drop('Price', axis=1)

In [71]: from sklearn.model_selection import train_test_split

In [72]: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_

Simple Linear Regression

In [73]: from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet

In [74]: linear_model = LinearRegression()

In [75]: linear_model.fit(X_train, y_train)

Out[75]: ▾ LinearRegression i ?

LinearRegression()

In [76]: linear_model.score(X_train, y_train)

Out[76]: 0.6827792395792723

In [77]: linear_model.score(X_test, y_test)

Out[77]: 0.13853683161540487

Ridge Regression

In [78]: ridge_model = Ridge(alpha=1.0, max_iter=100)

In [79]: ridge_model.fit(X_train, y_train)

Out[79]: ▾ Ridge i ?

Ridge(max_iter=100)

In [80]: ridge_model.score(X_train, y_train)

Out[80]: 0.6796668251040214

In [81]: ridge_model.score(X_test, y_test)

Out[81]: 0.6701765758295284

Lasso Regression

In [82]: lasso_model = Lasso(alpha=50, max_iter=100)

In [83]: lasso_model.fit(X_train, y_train)

Out[83]: ▾ Lasso i ?

Lasso(alpha=50, max_iter=100)

In [84]: lasso_model.score(X_train, y_train)

Out[84]: 0.6767356948457683

In [85]: lasso_model.score(X_test, y_test)

Out[85]: 0.6637669697137103

ElasticNET Regression

In [86]: elastic_model = ElasticNet(alpha=100, l1_ratio=0.5)

In [87]: elastic_model.fit(X_train, y_train)

Out[87]: ▾ ElasticNet i ?

ElasticNet(alpha=100)

In [88]: elastic_model.score(X_train, y_train)

Out[88]: 0.06482342181996403

In [89]: elastic_model.score(X_test, y_test)

Out[89]: 0.07381459514861177

In [ ]: