Week 3

Maximum Likelihood Estimator

Notes on Statistics before we start:
• What is Probability Density Function?

Example Normal Distribution:

1 1
exp (− 2 𝑧 2 )
𝜎√2𝜋 2𝜎

𝑧
1 1 2
𝐶𝐷𝐹 → ∫ exp (− 2 𝑧 ) 𝑑𝑧 → 𝑇𝑎𝑏𝑙𝑒 𝑍
−∞ 𝜎 √2𝜋 2𝜎
For Sure here 𝜎 = 1 if Standard normal and z is weighted by
standard deviation

Example Poisson Distribution:

𝜆𝑥 𝑒 −𝜆
𝑥!

Example Binomial Simplified (Bernoulli) Distribution:

𝑃 𝑥 (1 − 𝑃)1−𝑥
 𝑋 Example 𝐷𝑖𝑠𝑡 𝑃𝐷𝐹
𝑋 → −∞: +∞ Returns of 𝑁𝑜𝑟𝑚𝑎𝑙 See above
(𝑛 > 30) Stock Market
(𝑛 > 30)

𝑋 → −∞: +∞ Returns of 𝑆𝑡𝑢𝑑𝑒𝑛𝑡

(𝑛 ≤ 30) Stock Market
(𝑛 ≤ 30)

𝑋 → 0,1 Distribution of 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 See above

𝐵𝑖𝑛𝑎𝑟𝑦 Dividends: (𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖)
0: if no Div.
1: if Div.

𝑋 Covid19 cases Poisson See above

→ 𝑟𝑒𝑎𝑐ℎ𝑒𝑠 𝑝𝑒𝑎𝑘 𝑡ℎ𝑒𝑛 𝑍𝑒𝑟𝑜

• What is a Joint Distribution in Statistics?

When we connect Distributions. When we need to create a joint
distribution, we multiply their probability density function together.
• What is Maximum Likelihood Estimator?

1
𝑁𝑜𝑟𝑚𝑎𝑙 → 𝑃𝜃 = ..
𝜎…
𝑆𝑡𝑢𝑑𝑒𝑛𝑡
Θ= 𝑃𝑜𝑖𝑠𝑠𝑜𝑛 → 𝑃 𝜃
𝑊𝑒𝑖𝑏𝑢𝑙𝑙
..
( .. )

We assume a data D follows one 𝜃𝑖 ∈ Θ with its 𝑃𝜃𝑖 .

Example.
You have data on returns of stock market for 100 months.
This Data D~ 𝜃𝑖 ∈ Θ: 𝑃𝜃

1 1 2
𝐷~𝑁𝑜𝑟𝑚𝑎𝑙 ∈ Θ: 𝑃𝜃 = exp (− 𝑧 )
𝜎√2𝜋 2𝜎 2

1 1 2
𝑃𝑟𝑜𝑏 = exp (− 𝑧 )
𝜎√2𝜋 2𝜎 2
 1 1 2 1 1 2
𝐿= exp (− 2 𝑧 ) × exp (− 2 𝑧 )
𝜎√2𝜋 2𝜎 𝜎√2𝜋 2𝜎
1 1 1 1
× exp (− 2 𝑧 2 ) × exp (− 2 𝑧 2 ) × … .
𝜎√2𝜋 2𝜎 𝜎√2𝜋 2𝜎
1 1 2
× exp (− 2 𝑧 )
𝜎√2𝜋 2𝜎

MLE is the Likelihood Maximized

Clue Keywords:
• Probability Density Function.
• Likelihood Function: Joint PDF

Maximum Likelihood Estimator.

It is an estimator that assumes for a given data 𝐷, it can follow a
specific distribution 𝜃 in statistics that belongs to Θ with its
probability density function. This estimator creates a Joint
Distribution (Likelihood Function) and at its maximum (once
maximized) we find the best parameters.
Now we will start apply/understand MLE assuming a normal distribution.
I. Conceptual Part
• OLS is a special case of MLE where the Data ~ Normality.
• The Minimum ∑𝜀 2 of OLS will corresponds to the Maximum of
Likelihood function, under normality conditions.

II. Mathematical Part

𝐷 𝑤𝑖𝑡ℎ 𝑁 𝑜𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠~𝑁𝑜𝑟𝑚𝑎𝑙
𝜀~𝑁(0, 𝜎 2 )
For Simple Form (One X)
𝑵 𝟏 𝟏 𝟐
𝑳= 𝚷𝒊=𝟏 𝒆𝒙𝒑 (− 𝟐 𝜺 )
𝝈√𝟐𝚷 𝟐𝝈

𝟏 𝟏
𝑵
𝑳 = 𝚷𝒊=𝟏 𝒆𝒙𝒑 (− (𝒀 − ̂ )𝟐 )
𝒀
𝝈√𝟐𝚷 𝟐𝝈𝟐
 𝟏 𝟏
𝑳= 𝑵
𝚷𝒊=𝟏 𝒆𝒙𝒑 (− 𝟐 (𝒀 − (𝜶 ̂ 𝑿)𝟐 )
̂+𝜷
𝝈√𝟐𝚷 𝟐𝝈

𝟏 𝟏
𝑳= 𝑵
𝚷𝒊=𝟏 𝒆𝒙𝒑 (− 𝟐 (𝒀 − 𝜶 ̂ 𝑿)𝟐 )
̂−𝜷
𝝈√𝟐𝚷 𝟐𝝈

For Matrix Form (Multiple X)

𝟏 𝟏
𝑵
𝑳 = 𝚷𝒊=𝟏 𝒆𝒙𝒑 (− (𝒀 − ̂ )𝟐 )
𝒀
𝝈√𝟐𝚷 𝟐𝝈𝟐

𝟏 𝟏
𝑵
𝑳 = 𝚷𝒊=𝟏 𝒆𝒙𝒑 (− ̂ )𝟐 )
(𝒀 − 𝑿𝜷
𝟐𝝈 𝟐
𝝈√𝟐𝚷
See Appendix
• 𝚷 joins a constant by Power
• 𝚷 joins an Exponential by summation

𝑵
𝟏 𝟏
𝑳=( ) 𝒆𝒙𝒑 (− ∑(𝒀 − ̂ )𝟐 )
𝑿𝜷
𝝈√𝟐𝚷 𝟐𝝈𝟐
Before we derive the likelihood function, we can follow the recent
updates in Python, R, Stata, SAS: we take the Log then we derive

𝒍 → 𝑳𝒐𝒈(𝑳)
 𝑵
𝟏 𝟏
𝑳=( ̂ )𝟐 )
) 𝒆𝒙𝒑 (− 𝟐 ∑(𝒀 − 𝑿𝜷
⏟𝝈√𝟐𝚷 ⏟ 𝟐𝝈
𝒂 𝒃

𝟏 𝟏
𝒍 = 𝑵. 𝒍𝒐𝒈 ( ̂ )𝟐 )
) + 𝒍𝒐𝒈 𝒆𝒙𝒑 (− 𝟐 ∑(𝒀 − 𝑿𝜷
𝝈√𝟐𝚷 𝟐𝝈

𝟏 𝟏
𝒍 = 𝑵. 𝒍𝒐𝒈 ( ̂ )𝟐
) − 𝟐 ∑(𝒀 − 𝑿𝜷
𝝈√𝟐𝚷 𝟐𝝈

𝟏
̂ )𝟐
𝒍 = 𝑵[𝒍𝒐𝒈 𝟏 − 𝒍𝒐𝒈 𝝈√𝟐𝚷] − 𝟐 ∑(𝒀 − 𝑿𝜷
𝟐𝝈
Log Likelihood Function of Normal Distribution
𝟏
̂ )𝟐
𝒍 = −𝑵(𝒍𝒐𝒈 𝝈√𝟐𝚷) − 𝟐 ∑(𝒀 − 𝑿𝜷
𝟐𝝈

𝟏
𝒍 = −𝑵𝒍𝒐𝒈 𝝈 − 𝑵 𝒍𝒐𝒈 √𝟐𝚷 − ∑(𝒀 − ̂ )𝟐
𝑿𝜷
𝟐𝝈𝟐

𝝏𝒍
̂
𝝏𝜷

𝝏𝒍
̂
𝝏𝝈
Appendix Math Notes:

Note 1. Product Operator

3
𝑎 × 𝑎 × 𝑎 = 𝑎3 = Π𝑖=1 𝑎𝑖

Note 2. The Product of Exponential is the Exponential of a Sum

Π exp(3) = exp(∑3)
exp(3 + 3 + 3) = 8103
exp(3) × exp(3) × exp(3) = 8103
When the Π joins an exponential it turns to summation.
When the Π joins a constant it turns to power

Note 3. Econometrically Speaking, MLE is assumed to the 𝜀

Normality 𝜀~𝑁(0, 𝜎 2 )

Note 4. Normal distribution PDF

1 1
𝑃𝐷𝐹 = exp (− 2 (𝑥 − 𝜇𝑥 )2 )
𝜎√2𝜋 2𝜎
From an Econometric point of view, we assume our variable is not 𝑥 but
𝜀 where they follow normality with mean 𝜇𝜀 = 0 and variance is 𝜎 2
1 1
𝑃𝐷𝐹 = exp (− 2 (𝜀 )2 )
𝜎√2𝜋 2𝜎
Note 5. Review of Log Properties

log(𝑎𝑏) = log(𝑎 ) + log(𝑏)

𝑎
log ( ) = log(𝑎 ) − log(𝑏)
𝑏

log 𝑒 𝑎 = 𝑎 log 𝑒 = 𝑎

exp(𝑎) = 𝑒 𝑎