Module 32

Limiting Distributions

() Module 32 Limiting Distributions 1 / 38

T = (T1 , . . . , Tn ): a r.v. having joint p.m.f./p.d.f. fT (·);

h : Rn → R: a given function;

Distribution of Xn = h(T ) is desired;

Very often it is not possible to derive the expression for distribution

(i.e., p.m.f. or p.d.f.) of Xn .

() Module 32 Limiting Distributions 1 / 38

Example 1.

Let T1 , . . . , Tn be a random sample from B(a, b) distribution (beta

n
distribution), where −∞ < a < b < ∞. Let T̄n = n1
P
Ti . The form of
i=1
p.d.f. (or d.f.) of T̄n is so complicated (it involves multiple integrals which
cannot be expressed in closed form) that hardly anybody would be
interested in using it. It will be helpful to approximate distribution of T̄n
by a distribution which is mathematically tractable.

() Module 32 Limiting Distributions 2 / 38

Convergence in Distribution and Probability
{Xn }n≥1 : a sequence of r.v.s;

Fn : d.f. of Xn , n = 1, 2, . . .;

An approximation to distribution Xn (i.e., Fn ) is desired for large

values of n (i.e., as n → ∞);

it may be tempting to approximate Fn (·) by

F (x) = lim Fn (x), x ∈ R.
n→∞

Question: If Fn ’s are d.f.s, does

F (x) = lim Fn (x), x ∈ R
n→∞
define a d.f.?
Answer: No.
() Module 32 Limiting Distributions 3 / 38
Example 2.

(i) Suppose that

 1
P Xn = = 1, n = 1, 2, . . . .
n
Then, (
1
0, if x < n
Fn (x) = P(Xn ≤ x) = 1
.
1, if x ≥ n

and (
0, if x ≤ 0
F (x) = lim =
n→∞ 1, if x > 0
is not a d.f. (it is not right continuous at x = 0).

() Module 32 Limiting Distributions 4 / 38

However, F can be converted to a d.f.
(
∗ 0, if x < 0
F (x) =
1, if x ≥ 0

by changing value of F at x = 0.

1
Since lim = 0, a natural approximation of Fn seems to be the d.f. of
n→∞ n
r.v. X degenerate at 0 (i.e., F ∗ ).

() Module 32 Limiting Distributions 5 / 38

(ii) Let Xn ∼ N(0, n1 ), n = 1, 2, . . .. Then
√
Fn (x) = P(Xn ≤ x) = Φ( nx)

0, if x < 0

F (x) = lim Fn (x) = 12 , if x = 0 .
n→∞ 
1, if x > 0


Clearly, F is not a d.f. (it is not right continuous at x = 0). However,

F can be converted to a d.f.
(
0, if x < 0
F ∗ (x) =
1, if x ≥ 0

by changing value of F at x = 0. Since E (Xn ) = 0 and

Var(Xn ) = n1 → 0, as n → ∞, a natural approximation of Fn seems to
be the d.f. of r.v. X degenerate at 0 (i.e., F ∗ ).

() Module 32 Limiting Distributions 6 / 38

(iii) Suppose that

1
P(Xn = 0) = 1 − P(Xn = n) = , n = 1, 2, . . . .
n
Then, (
0, if x < n
Fn (x) = P(Xn ≤ x) = .
1, if x ≥ n
and
F (x) = lim Fn (x) = 0, ∀x ∈ R.
n→∞

Here, F (x) cannot be converted to a d.f. by changing its value at

countable number of points.

() Module 32 Limiting Distributions 7 / 38

The above examples suggest that if a sequence {Fn }n≥1 of d.f.s
converges at every point then it may be two restrictive to require that
{Fn }n≥1 converges to a d.f. (i.e., to require that
lim Fn (x) = F (x), ∀x ∈ R, for some d.f. F .)
n→∞

() Module 32 Limiting Distributions 8 / 38

Definition 1:

Let X be a r.v. with d.f. F .

(a) The sequence {Xn }n≥1 is said to converge in distribution to X , as
d
n → ∞ (written as Xn → X , as n → ∞), if

lim Fn (x) = F (x), ∀ x ∈ CF ,

n→∞

where CF is the set of continuity points of F . The d.f. F (or, the

corresponding p.m.f./p.d.f.) is called the limiting distribution of Xn ,
as n → ∞.

(b) Let c ∈ R. The sequence {Xn }n≥1 is said to converge in probability

p d
to c, as n → ∞ (written as Xn → c, as n → ∞), if Xn → X , as
n → ∞, where X is a r.v. degenerate at c (i.e., P(X = c) = 1).

() Module 32 Limiting Distributions 9 / 38

Remark 1:
(i) Since CFc is countable,
d
Xn → X , as n → ∞ ⇒ lim Fn (x) = F (x),
n→∞

everywhere except possibly at a countable number of points.

(ii) For c ∈ R,
(
p 0, if x < c
Xn → c, as n → ∞ ⇔ For x 6= c, lim Fn (x) = .
n→∞ 1, if x > c

(iii) If r.v. X is continuous, then

d
Xn → X ⇔ lim Fn (x) = F (x), ∀ x ∈ R;
n→∞

(iv) For c ∈ R,
p p
Xn → c ⇔ Xn − c → 0.
() Module 32 Limiting Distributions 10 / 38
Example 3

Let
1
P(Xn = 0) = 1 − P(Xn = n) = , n = 1, 2, . . . .
n
Then,

0, if x < 0

Fn (x) = n1 , if 0 ≤ x < 1
n

1, if x ≥ n1 .


(
n→∞ 0, if x ≤ 0
−→ (not a d.f.)
1, if x > 0.
p
Clearly, Xn → 0.

() Module 32 Limiting Distributions 11 / 38

Example 4

Let X1 , X2 , . . . be i.i.d. r.v.s with X1 ∼ U(0, 1), θ > 0. Find limiting

distribution of Xn:n = max{X1 , . . . , Xn } and Yn = n(θ − Xn:n ).
Solution For x ∈ R,

FXn:n (x) = P(max{X1 , . . . , Xn } ≤ x)

= P(Xi ≤ x, i = 1, . . . , n)
Yn
= P(Xi ≤ x)
i=1
h in
= F (x) ,

where F (x) is the d.f. of X .

() Module 32 Limiting Distributions 12 / 38

We have 
0,
 if x < 0
F (x) = P(X1 ≤ x) = xθ , if 0 ≤ x < θ .

1, if x ≥ θ


Thus,


0,

 if x < 0
x n
θ , if 0 ≤ x < θ
FXn:n (x) =

1, if x ≥ θ


(
n→∞ 0, if x < θ
−→
1, if x ≥ θ
p
⇒ Xn:n → θ.

() Module 32 Limiting Distributions 13 / 38

Also,
 
FYn (x) = P n(θ − Xn:n ) ≤ x
 x
= P Xn:n ≥ θ −
n
 x
= 1 − FXn:n θ −
n

0,
 if x ≤ 0
x

n
= 1 − 1 − nθ , if 0 < x < nθ

1, if x > nθ.

(
n→∞ 0, if x ≤ 0
−→ − θx
.(d.f. of Exp(θ))
1 − e , if x > 0
= G (x), say

d
Thus, Xn → X , where X ∼ Exp(θ).

() Module 32 Limiting Distributions 14 / 38

Result 1.
Let {Xn }n≥1 be a sequence of r.v.s and let c be a real constant. Then
p
 
Xn → c, as n → ∞ ⇔ ∀  > 0, lim P |Xn − c| ≥  = 0,
n→∞
Proof:
p
Suppose that Xn → c, as n → ∞, i.e., lim Fn (x) = F (x), ∀ x 6= c, where
n→∞
(
0, if x < c
F (x) = .
1, if x ≥ c
Fix  > 0. Then
  h i
lim P |Xn − c| ≥  = lim P(Xn ≤ c − ) + P(Xn ≥ c + )
n→∞ n→∞
h i
= lim Fn (c − ) + 1 − Fn ((c + )−)
n→∞
= F (c − ) + 1 − F ((c + )−)
= 0 + 1 − 1 = 0.
() Module 32 Limiting Distributions 15 / 38
Conversely suppose that
 
lim P |Xn − c| ≥  = 0, ∀ > 0
n→∞
h i
⇒ lim Fn (c − ) + 1 − Fn ((c + )−) = 0, ∀ > 0
n→∞
⇒ lim Fn (c − ) = 0, ∀ > 0 and lim Fn ((c + )−) = 1, ∀ > 0
n→∞ n→∞
⇒ lim Fn (c − ) = 0, ∀ > 0 and lim Fn (c + ) = 1, ∀ > 0
n→∞ n→∞
(since Fn (c + ) ≥ Fn (c + )−)
⇒ lim Fn (x) = 0, ∀x < c and lim Fn (x) = 1, ∀x > c
n→∞ n→∞
⇒ ∀x 6= c, lim Fn (x) = F (x),
n→∞
where
(
0, if x < c
F (x) =
1, if x ≥ c

d
. Thus, Xn → X .
() Module 32 Limiting Distributions 16 / 38
Remark 2.

p
Above result suggests that if Xn −→ c, as n → ∞, then Xn is
stochastically (in probability) very close to c for large values of n. Such an
interpretation does not hold for the concept of convergence in distribution.
d
Specifically, if Xn −→ X , as n → ∞ (where X is some non-degenerate
r.v.), then it can not be inferred that Xn is getting close to X (as n → ∞)
in any sense. All we know in that case is, for large values of n, the
distribution of Xn is getting close to that of X .

() Module 32 Limiting Distributions 17 / 38

Result 2.
Let {Xn }n≥1 be a sequence of r.v.s with E (Xn ) = µn ∈ (−∞, ∞) and
Var(Xn ) = σn2 ∈ (0, ∞), n = 1, 2, . . .. Suppose that lim µn = µ ∈ R, and
n→∞
p
lim σn2 = 0. Then Xn −→ µ, as n −→ ∞.
n→∞

Proof.
Fix  > 0. Then
  E (X − µ)2

n
0 ≤ P |Xn − µ| ≥  ≤ .
2
Also,

E (Xn − µ)2 = E (Xn − µn + µn − µ)2

= E (Xn − µn )2 + (µn − µ)2

= σn2 + (µn − µ)2 .

() Module 32 Limiting Distributions 18 / 38

Then,
  σ 2 + (µ − µ)2
n
0 ≤ P |Xn − µ| ≥  ≤ n
2
n→∞
−→ 0
 
⇒ lim P |Xn − µ| ≥  = 0
n→∞
p
⇒ Xn → µ, as n → ∞.

We state the following useful result without providing it proof.

() Module 32 Limiting Distributions 19 / 38

Result 3.

Let {Xn }n≥1 be a sequence of r.v.s with m.g.f.s {Mn (·)}n≥1 and let X be
another r.v. with m.g.f. M(·). Suppose there exists an h > 0 such that
Mn (·), n = 1, 2, . . . and M(·) are finite on (−h, h) and

lim Mn (t) = M(t), ∀t ∈ (−h, h).

n→∞

d
Then Xn → X .

() Module 32 Limiting Distributions 20 / 38

Lemma 1.

Let {cn }n≥1 be a sequence of real numbers such that lim cn = c. Then,
n→∞
 cn n
lim 1+ = ec .
n→∞ n
 
x
Hint: For any x ∈ R, ln(1 + x) = 1+ξx , for some ξx between 0 and x.

() Module 32 Limiting Distributions 21 / 38

Result 4 (Poisson Approximation to Binomial
Distribution)
Let Xn ∼ Bin(n, θn ), where θn ∈ (0, 1), n = 1, 2, . . . and
d
lim (nθn ) = θ > 0. Then Xn → X , where X ∼ Poisson(θ), the Poisson
n→∞
distribution with mean θ.

Proof. We know that the m.g.f. of X is

t −1)
M(t) = e θ(e , t∈R

and the m.g.f. of Xn (n = 1, 2, . . .) is

Mn (t) = (1 − θn + θn e t )n
cn (t) n
 
= 1+ , t ∈ R,
n

() Module 32 Limiting Distributions 22 / 38

where

cn (t) = nθn (e t − 1), t ∈ R

n→∞
−→ θ(e t − 1), t ∈ R.

Thus,
t −1)
lim Mn (t) = e θ(e , t∈R
n→∞

d
⇒ Xn → X .

() Module 32 Limiting Distributions 23 / 38

Result 5 (Weak Law of Large Numbers (WLLN) and
the Central Limit Theorem (CLT))

n
1 P
Let {Xn }n≥1 be a sequence of i.i.d. r.v.s and let X̄n = n Xi , n = 1, 2, . . .
i=1

(i) (WLLN) Suppose that E (X1 ) = µ is finite. Then

p
X̄n → µ, as n → ∞,

(ii) (CLT) Suppose that 0 < Var(X1 ) = σ 2 < ∞. Then

√
def n(X̄n − µ) d
Zn = → Z ∼ N(0, 1), as n → ∞.
σ

() Module 32 Limiting Distributions 24 / 38

Proof.

(i) For simplicity we will assume that Var(X1 ) = σ 2 < ∞. Then

n
1 X  1X n
E (X̄n ) = E Xi = E (Xi ) = E (X1 ) = µ,
n n
i=1 i=1
n n
1 X  1 X σ2
Var(X̄n ) = Var Xi = 2 Var(Xi ) = → ∞, as n → ∞.
n n n
i=1 i=1

p
Thus, X̄n → µ, by Result 2.

() Module 32 Limiting Distributions 25 / 38

(ii) For simplicity assume that the common m.g.f. M(·) of X1 , X2 , . . . is
finite in an interval (−h, h), for some h > 0. Let Yi = Xiσ−µ ,
i = 1, 2, . . ., so that {Yn }n≥1 is a sequence of i.i.d. r.v.s with
E (Y1 ) = 0 and Var(Y1 ) = 1. Also,
n n
1X 1X
X̄n = Xi = µ + σ Yi = µ + σ Ȳn
n n
i=1 i=1
n
√ 1X
Zn = nȲn , n = 1, 2, . . . , where Ȳn = Yi .
n
i=1

The common m.g.f. of Y1 , Y2 , . . . is

t(X1 −µ)
MY (t) = E (e )σ

µt t
= e − σ MX1 ( ), −hσ < t < hσ.
σ

() Module 32 Limiting Distributions 26 / 38

Then
(1)
MY (0) = E (Y1 ) = 0
(2)
MY (0) = E (Y12 ) = 1.

Let ψ2 : (−hσ, hσ) → R be such that

(1) t 2  (2) 
MY (t) = MY (0) + tMY (0) + MY (0) + ψ2 (t) , −hσ < t < hσ;
2
i.e., for t ∈ (−hσ, hσ)
(1)
MY (t) − MY (0) − tMY (0) (2)
ψ2 (t) = 2
− MY (0),
t /2
(1) (1)
MY (t) − MY (0) (2)
⇒ lim ψ2 (t) = lim − MY (0), (L’ Hospital Rule)
t→0 t→0 t
(2) (2)
= MY (0) − MY (0)
=0

() Module 32 Limiting Distributions 27 / 38

 √ n
n(X̄n −µ)
√1
P
The m.g.f. of Zn = σ = n
Yi is
i=1

n
√t
P

n
Yi 
MZn (t) = E e i=1

n
Y  t 
= MY √
n
i=1
h  t in
= MY √
n
h t (1) t 2  (2)  t in
= MY (0) + √ MY (0) + MY (0) + ψ2 √
n 2n n
h t 2   t i n √ √
= 1+ 1 + ψ2 √ , t ∈ (− nhσ, nhσ).
2n n

() Module 32 Limiting Distributions 28 / 38

Clearly,
t2
lim MZn (t) = e 2 = K (t), ∀t ∈ R.
n→∞

Note that K (t), t ∈ R, is the m.g.f. of Z ∼ N(0, 1). Now the assertion
follows from Result 3.

() Module 32 Limiting Distributions 29 / 38

Remark 3:

(i) The WLLN implies that the sample mean based on a random sample
from any parent distribution can be made arbitrary close to the
population mean (in probability) by choosing sufficiently large sample
sizes.

(ii) The CLT states that, irrespective of the nature of the parent
distribution, the probability distribution of a normalized version of the
sample mean, based on a random sample of large size, is
approximately normal.

() Module 32 Limiting Distributions 30 / 38

Justification of Relative Frequency Method of
Assigning Probabilities

(Ω, P(Ω), P): Probability space associated with a random experiment

E.

We are interested in assigning probability, say P(E ), to an event

E ∈ P(Ω).

Repeat the experiment E a large (say N) number of times.

Define, for i = 1, . . . , N
(
1, if ith trial results in E
Yi =
0, otherwise.

() Module 32 Limiting Distributions 31 / 38

Clearly Y1 , . . . , YN are i.i.d. r.v.s with µ = E (Y1 ) = P(E ).

N
P
fN (E ) = No. of times event E occurs in first N trials = Yi .
i=1

N
fN (E ) 1 X
rN (E ) = = Yi = ȲN .
N N
i=1

By WLLN
p
rN (E ) = ȲN → µ = P(E ).

Thus, the WLLN justifies the relative frequency approach to assign

probabilities.

() Module 32 Limiting Distributions 32 / 38

Result 6.

Let {Xn }n≥1 and {Yn }n≥1 be sequences of r.v.s and let X be another r.v.
p
(i) g : R → R is continuous at c ∈ R and Xn → c, as n → ∞,
p
⇒ g (Xn ) → g (c), as n → ∞.

p p
(ii) h : R2 → R is continuous at (c1 , c2 ) ∈ R2 , Xn → c1 , and Yn → c2 , as
n → ∞,
p
⇒ h(Xn , Yn ) → h(c1 , c2 ), as n → ∞.

() Module 32 Limiting Distributions 33 / 38

 d
(iii) g : R → R is continuous on SX (support of X ) and Xn → X , as
n → ∞,
d
⇒ g (Xn ) → g (X ), as n → ∞.

d
(iv) h : R2 → R is continuous on D = {(x, b) : x ∈ SX }, Xn → X , and
p
Yn → b, as n → ∞
d
⇒ h(Xn , Yn ) → h(X , b), as n → ∞.

() Module 32 Limiting Distributions 34 / 38

Remark 4.

(i) Let {Xn }n≥1 be a sequence of i.i.d. r.v.s with E (X1 ) = µ and
Var(X1 ) = σ 2 . The CLT asserts that, as n → ∞,
√
n(X̄n − µ) d
−→ Z ∼ N(0, 1)
√ σ
1 n(X̄n − µ) d
⇒√ −→ 0 × Z = 0
n σ
X̄n − µ p
⇒ −→ 0
σ
p
⇒ X̄n −→ µ.

Thus, under finiteness of variance, the CLT is a stronger result than

WLLN.

() Module 32 Limiting Distributions 35 / 38

(ii) For real constants c and d (d 6= 0)
p p
Xn −→ c, Yn −→ d ⇒
p p Xn p c
Xn ± Yn −→ c ± d, Xn Yn −→ cd, −→ .
Yn d

(iii) For real constant c

d p
Xn −→ X , Yn −→ c ⇒
d d Yn d X
Xn ± Yn −→ X ± c, Xn Yn −→ cX , −→ ,
Xn c
(provided c 6= 0).

() Module 32 Limiting Distributions 36 / 38

Take Home Problems
Let {Xn }n≥1 be a sequence of i.i.d. r.v.s with finite mean µ. Let
n n
X̄n = n1 1 P
Xi and Sn2 = n−1 (Xi − X̄n )2 , n = 1, 2, . . . be sequences of
P
i=1 i=1 √
n(X̄n −µ)
sample means and sample variances, respectively. Define Tn = Sn ,
n = 1, 2, . . ..
p p
(a) If σ 2 = Var(X1 ) ∈ (0, ∞), then show that Sn2 −→ σ 2 , Sn −→ σ and
d
Tn −→ Z ∼ N(0, 1), as n → ∞;

4
(b) Suppose that the kurtosis γ1 = E ((Xσ1 −µ)
4
)
< ∞. Then show that
√ d
n(Sn2 − σ 2 ) −→ W N(0, (γ1 − 1)σ 4 ), as n → ∞.

(c) Show that the Student t-distribution with large degrees of freedom
(i.e., as degrees of freedom ν → ∞ ) can be approximated by a
N(0, 1) distribution.
() Module 32 Limiting Distributions 37 / 38
 Thank you for your patience

() Module 32 Limiting Distributions 38 / 38