Logistic Regression

Logistic regression is a statistical analysis method used to predict a data value based on prior
observations of a data set. Logistic regression has become an important tool in the discipline
of machine learning. The approach allows an algorithm being used in a machine learning
application to classify incoming data based on historical data. A logistic regression model
predicts a dependent data variable by analyzing the relationship between one or more existing
independent variables. For example, a logistic regression can be used to predict whether a
political candidate will win or lose an election or whether a high school student will be
admitted to a particular college. The purpose of logistic regression is to estimate the
probabilities of events, including determining a relationship between features and the
probabilities of particular outcomes.Multinomial logistic regression can be used to classify
subjects into groups based on a categorical range of variables to predict behavior.
Binary logistic regression is most useful when we want to model the event probability for a
categorical response variable with two outcomes.

Figure 1 linear vs logistic regression

We must wonder how logistic regression squeezes the output of linear regression between 0
and 1. We start by mentioning the formula of logistic function:
1
p= −(β0 +β 1 x)
1+e
We know the equation of the best fit line in linear regression is:
y=β 0 + β 1 x
Where:
 x is the input value
 y is the predicted output
 β 0 is the bias or intercept term
 β 1 is the coefficient for the single input value (x)
Instead of y we are taking probabilities (P). But there is an issue here, the value of (P) will
exceed 1 or go below 0 and we know that range of Probability is (0-1). To overcome this
issue we take “odds” of P:
p=β 0 + β 1 x

p
=β 0 + β 1 x
(1− p)
We know that odds can always be positive which means the range will always be (0,+∞ ).
Odds are nothing but the ratio of the probability of success and probability of failure. The
problem here is that the range is restricted and we don’t want a restricted range because if we
do so then correlation will decrease. By restricting the range we are actually decreasing the
number of data points and of course, if we decrease our data points, correlation will decrease.
It is difficult to model a variable that has a restricted range. To control this we take the log of
odds which has a range from (-∞,+∞).

log ( 1−pp )=β + β x 0 1

Now we just want a function of P because we want to predict probability right? not log of
odds. To do so we will multiply by exponent on both sides and then solve for P.

exp [log ( 1−p p )]=exp ⁡(β¿¿ 0+ β x)¿ 1

p
ln ⁡[ ]
e 1− p
=e ⁡( β¿ ¿ 0+ β x)¿
1

p ⁡( β ¿ 0+ β x)¿
=e ¿ 1

(1− p)
 ⁡( β¿ ¿0 + β1 x) ¿

p=e ⁡(β ¿ ¿0+ β 1 x)− p e ¿

p= p ¿
⁡( β¿ ¿ 0+ β1 x)
e ⁡(β ¿¿ 0+ β1 x )¿
1= −e ¿
p
⁡( β¿ ¿ 0+ β1 x)
e ⁡(β ¿¿ 0+ β1 x )¿
1= −e ¿
p
⁡( β ¿ 0+ β1 x)
e ¿
p= ¿
1+e ⁡( β¿¿ 0+ β x) ¿ 1

1
p= − ⁡(β ¿¿ 0+ β1 x)
1+e ¿
Now we have our logistic function, also called a sigmoid function. The graph of a sigmoid
function is as shown below. It squeezes a straight line into an S-curve.

Illustrate with an example

Table 1 simple data set for number of hours study and probability of passing
Hours Pass
 0.5 0
0.75 0
1 0
1.25 0
1.5 0
1.75 0
1.75 1
2 0
2.25 1
2.5 0
2.75 1
3 0
3.25 1
3.5 0
4 1
4.25 1
4.5 1
4.75 1
5 1
5.5 1

First we need to calculate intercept (b) and slope (m)

m=
∑ ( x i−x ) ( y i − y)
∑ (x i−x )2

b= y−m∗x
x=independentvariables
x=average of independent variables
y=dependent variables
y=average of depndent variables
Now put x=0.5
y=0.2345∗0.5+¿(-0.15366)
y=0.11725-0.15366
y=−0.0366
Now calculate exp(value of logit) also called odds

Table 2 Calculation odds values

 Hours Pass Logit Odds
0.5 0 -0.0366 0.96406168
0.75 0 0.02205 1.0222949
1 0 0.0807 1.08404563
1.25 0 0.13935 1.14952636
1.5 0 0.198 1.21896239
1.75 0 0.25665 1.29259264
1.75 1 0.25665 1.29259264
2 0 0.3153 1.37067045
2.25 1 0.37395 1.45346448
2.5 0 0.4326 1.54125959
2.75 1 0.49125 1.63435789
3 0 -0.6156 0.54031661
3.25 1 0.60855 1.83776471
3.5 0 0.6672 1.94877311
4 1 0.7845 2.19131101
4.25 1 0.84315 2.32367504
4.5 1 0.9018 2.46403438
4.75 1 0.96045 2.612872
5 1 1.0191 2.77070001
5.5 1 1.1364 3.11553224

odds= EXP(-0.0366) =0.96406168

Now finally calculate the probability of passing by using the formula

⁡( odds)
pro=
(1+ odds)

0.96406
pro=
(1+0.96406)

0.96406
pro=
(1.96406)
 0.96406
pro=
( 1.96406 )

pro=0.49085

Table 3 Calculation of probability of passing

Hours Pass Logit Odds pro

0.5 0 -0.0366 0.96406168 0.490851021
0.75 0 0.02205 1.0222949 0.505512277
1 0 0.0807 1.08404563 0.520164058
1.25 0 0.13935 1.14952636 0.534781235
1.5 0 0.198 1.21896239 0.549338915
1.75 0 0.25665 1.29259264 0.56381261
1.75 1 0.25665 1.29259264 0.56381261
2 0 0.3153 1.37067045 0.578178401
2.25 1 0.37395 1.45346448 0.592413092
2.5 0 0.4326 1.54125959 0.606494353
2.75 1 0.49125 1.63435789 0.620400856
3 0 -0.6156 0.54031661 0.350782823
3.25 1 0.60855 1.83776471 0.647609967
3.5 0 0.6672 1.94877311 0.660875909
4 1 0.7845 2.19131101 0.686649156
4.25 1 0.84315 2.32367504 0.699128227
4.5 1 0.9018 2.46403438 0.711319263
4.75 1 0.96045 2.612872 0.723211894
5 1 1.0191 2.77070001 0.734797253
5.5 1 1.1364 3.11553224 0.757018062