36-752 Spring 2017

Advanced Probability Overview

Lecture Notes Set 1: Course Overview, -Fields, and Measures

Instructor: Jing Lei

Associated reading: Sec 1.1-1.4 of Ash and Doleans-Dade; Sec 1.1 and A.1 of Durrett.

1 Introduction
How is this course different from your earlier probability courses? There are some problems
that simply cant be handled with finite-dimensional sample spaces and random variables
that are either discrete or have densities.

Example 1 Try to express the strong law of large numbers without using an infinite-dimensional
space. Oddly enough, the weak law of large numbers requires only a sequence of finite-
dimensional spaces, but the strong law concerns entire infinite sequences.

Example 2 Consider a distribution whose cumulative distribution function (cdf ) increases

continuously part of the time but has some jumps. Such a distribution is neither discrete nor
continuous. How do you define the mean of such a random variable? Is there a way to treat
such distributions together with discrete and continuous ones in a unified manner?

General Measures Both of the above examples are accommodated by a generalization

of the theories of summation and integration. Indeed, summation becomes a special case of
the more general theory of integration. It all begins with a generalization of the concept of
size of a set.

Example 3 One way to measure the size of a set is to count its elements. All infinite sets
would have the same size (unless you distinguish different infinite cardinals).

Example 4 Special subsets of Euclidean spaces can be measured by length, area, volume,
etc. But what about sets with lots of holes in them? For example, how large is the set of
irrational numbers between 0 and 1?

We will use measures to say how large sets are. First, we have to decide which sets we will
measure.

1
2 -fields
Definition 1 (fields and -fields) Let be a set. A collection F of subsets of is called
a field if it satisfies

F,

for each A F, AC F,

for all A1 , A2 F, A1 A2 F.

A field F is a -field if, in addition, it satisfies

S
for every sequence {Ak }
k=1 in F, k=1 Ak F.

We will define measures on fields and -fields.

Definition 2 (Measurable Space) A set together with a -field F is called a measur-

able space (, F), and the elements of F are called measurable sets .

Example 5 (Intervals on R1 ) Let = IR and define U to be the collection of all unions of

finitely many disjoint intervals of the form (a, b] or (, b] or (a, ) or (, ), together
with . Then U is a field.

Example 6 (Power set) Let be an arbitrary set. The collection of all subsets of is a
-field. It is denoted 2 and is called the power set of .

Example 7 (Trivial -field) Let be an arbitrary set. Let F = {, }. This is the

trivial -field.

Exercise 1 Let F1 , F2 , . . . be classes of sets in a common space such that Fn Fn+1 for
each n. Show that if each Fn is a field, then n=1 Fn is also a field.

If each Fn is a -field, then is

n=1 Fn also necessarilya -field? Think about the following


case: is the set of nonnegative integers and Fn {0}, {1}, . . . , {n} .

2
Generated -fields A field is closed under finite set theoretic operations whereas a -field
is closed under countable set theoretic operations. In a problem dealing with probabilities,
one usually deals with a small class of subsets A, for example the class of subintervals of
(0, 1]. It is possible that if we perform countable operations on such a class A of sets, we
might end up operating on sets outside the class A. Hence, we would like to define a class
denoted by (A) in which we can safely perform countable set-theoretic operations. This
class (A) is called the -field generated by A, and it is defined as the intersection of all
the -fields containing A (exercise: show that this is a -field). (A) is the smallest -field
containing A.

Example 8 Let C = {A} for some nonempty A that is not itself . Then (C) = {, A, AC , }.

Example 9 Let = IR and let C be the collection of all intervals of the form (a, b]. Then
the field generated by C is U from Example 5 while (C) is larger.

Example 10 (Borel -field) Let be a topological space and let C be the collection of open
sets. Then (C) is called the Borel -field. If = IR, the Borel -field is the same as (C)
in Example 9. The Borel -field of subsets of IRk is denoted B k .

Exercise 2 Give some examples of classes of sets C such that (C) = B 1 .

Exercise 3 Are there subsets of IR which are not in B 1 ?

3 Measures
Notation 11 (Extended Reals) The extended reals is the set of all real numbers together
+
with and . We shall denote this set IR. The positive extended reals, denoted IR is
+0
(0, ], and the nonnegative extended reals, denoted IR is [0, ].

+0
Definition 3 Let (, F) be a measurable space. Let : F IR satisfy

() = 0,
S P
for every sequence {Ak }
k=1 of mutually disjoint elements of F, ( k=1 Ak ) = k=1 (Ak ).

Then is called a measure on (, F) and (, F, ) is a measure space. If F is merely

a field, then a that satisfies the above two conditions whenever
S
k=1 Ak F is called a
measure on the field F.

3
Example 12 Let be arbitrary with F the trivial -field. Define () = 0 and () = c
for arbitrary c > 0 (with c = possible).

Example 13 (Counting measure) Let be arbitrary and F = 2 . For each finite subset
A of , define (A) to be the number of elements of A. Let (A) = for all infinite subsets.
This is called counting measure on .

Definition 4 (Probability measure) Let (, F, P ) be a measure space. If P () = 1,

then P is called a probability, (, F, P ) is a probability space, and elements of F are called
events.

Sometimes, if the name of the probability P is understood or is not even mentioned, we will
denote P (E) by Pr(E) for events E.
Infinite measures pose a few unique problems. Some infinite measures are just like finite
ones.

Definition 5 (-finite measure) Let (, F, ) be a measure space, and let C F. Sup-

pose that there exists a sequence {An }
n=1 of elements of C such that (An ) < for all n
S
and = n=1 An . Then we say that is -finite on C. If is -finite on F, we merely say
that is -finite.

Example 14 Let = ZZ with F = 2 and being counting measure. This measure is

-finite. Counting measure on an uncountable space is not -finite.

Exercise 4 Prove the claims in Example 14.

3.1 Basic properties of measures

There are several useful properties of measures that are worth knowing.
First, measures are countably subadditive in the sense that

!
[ X
An (An ), (5)
n=1 n=1

for arbitrary sequences {An } n=1 . The proof of this uses a Sstandard trick for dealing with
n1
countable sequences of sets. Let B1 = A1 and let Bn = An \ i=1 Bi for n > 1. The Bn s are
disjoint and have the same finite and countable unions as the An s. The proof of Equation
5 relies on the additional fact that (Bn ) (An ) for all n.
Next, if (An ) = 0 for all n, it follows that (
S
n=1 An ) = 0. This
T gets used a lot in proofs.
Similarly, if is a probability and (An ) = 1 for all n, then ( n=1 An ) = 1.

4
Definition 6 (Almost sure/almost everywhere) Suppose that some statement about el-
ements of holds for all AC where (A) = 0. Then we say that the statement holds
almost everywhere, denoted a.e. []. If P is a probability, then almost everywhere is often
replaced by almost surely, denoted a.s. [P ].

Example 15 Let (, F, P ) be a probability space. Let {Xn }n=1 be a sequence of functions

a.s.
from to IR. To say that Xn converges to X a.s. [P ] (denoted Xn X) means that there
is a set A with P (A) = 0 and limn Xn () = X() for all AC .

Proposition 6 (Linearity) If 1 , 2 , . . . are all measures on (, F) and if {an }

n=1 is a
P
sequence of positive numbers, then n=1 an n is a measure on (, F).

Exercise 7 Prove Proposition 6.

3.2 Monotone sequences of sets and limits of measure

Definition 7 (Monotone sequences of sets) Let (, F, ) be a measure space. A se-
quence {An }
n=1 of elements of F is called monotone increasing if An An+1 for each n. It
is monotone decreasing if An An+1 for each n.

Lemma 16 Let (, F, ) be a measure space. Let {An } n=1 be a monotone sequence of

elements of F. Then limn (An ) = (limn An ) if either of the following hold:

the sequence is increasing,

the sequence is decreasing and (Ak ) < for some k.

Bn = An \ An1 for
Proof: Define AS= limn An . In the first case, write B1 = A1 and P
n > 1. Then An = k=1 Bk for all n (including n = ). Then (An ) = nk=1 (Bk ), and
n

 
X n
X
lim An = (A ) = (Bk ) = lim (Bk ) = lim (An ).
n n n
k=1 k=1

In the second case, write Bn = An \ An+1 for all n k. Then, for all n > k,
n1
[
Ak \ An = Bi ,
i=k

[
Ak \ A = Bi .
i=k

5
By the first case,

!
[
lim (Ak \ An ) = Bi = (Ak \ A ).
n
i=k

Because An Ak for all n > k and A Ak , it follows that

(Ak \ An ) = (Ak ) (An ),

(Ak \ A ) = (Ak ) (A ).

It now follows that limn (An ) = (A ).

Exercise 8 Construct a simple counterexample to show that the condition (Ak ) < is
required in the second claim of Lemma 16.

3.3 Uniqueness of Measures

There is a popular method for proving uniqueness theorems about measures. The idea is to
define a function on a convenient class C of sets and then prove that there can be at most
one extension of to (C).

Example 17 Suppose it is given that for any a IR,

Z a
1
exp u2 /2 du.


P ((, a]) =
2
Does that uniquely define a probability measure on the class of Borel subsets of the line, B 1 ?

Definition 8 (-system and -system) A collection A of subsets of is a -system if,

for all A1 , A2 A, A1 A2 A. A class C is a -system if

C,

for each A C, AC C,
S
for each sequence {An }
n=1 of disjoint elements of C, n=1 An C.

Example 18 The collection of all intervals of the form (, a] is a -system of subsets of

IR. So too is the collection of all intervals of the form (a, b] (together with ). The collection
of all sets of the form {(x, y) : x a, y b} is a -system of subsets of IR2 . So too is the
collection of all rectangles with sides parallel to the coordinate axes.

Some simple results about -systems and -systems are the following.

6
Proposition 9 If is a set and C is both a -system and a -system, then C is a -field.

Proposition 10 Let be a set and let be a -system of subsets. If A and A B

then A B C .

Exercise 11 Prove Propositions 9 and 10.

Lemma 19 ( theorem) Let be a set and let be a -system and let be a -

system that contains . Then () .

Proof: Define () to be the smallest -system containing . For each A , define GA

to be the collection of all sets B such that A B ().
First, we show that GA is a -system for each A (). To see this, note that A (),
so GA . If B GA , then A B (), and Proposition 10 says that A B C (),
so B C GA . Finally, {Bn } n=1 GA with the Bn disjoint implies
Sthat A BSn () with
ABn disjoint, so their union is in (). But their union is A( n=1 Bn ). So n=1 Bn GA .
Next, we show that () GC for every C (). Let A, B , and notice that
A B , so B GA . Since GA is a -system containing , it must contain (). It follows
that A C () for all C (). If C (), it then follows that A GC . So, GC
for all C (). Since GC is a -system containing , it must contain ().
Finally, if A, B (), we just proved that B GA , so A B () and hence () is
also a -system. By Proposition 9, () is a -field containing and hence must contain
(). Since () , the proof is complete.

The uniqueness theorem is the following.

Theorem 20 (Uniqueness theorem) Suppose that 1 and 2 are measures on (, F) and

F = (), for a -system . If 1 and 2 are both -finite on and they agree on , then
they agree on F.

Proof: First, let C be such that 1 (C) = 2 (C) < , and define GC to be the
collection of all B F such that 1 (B C) = 2 (B C). It is easy to see that GC is a
-system that contains , hence it equals F by Lemma 19. (For example, if B GC ,
1 (B C C) = 1 (C) 1 (B C) = 2 (C) 2 (B C) = 2 (B C C),
so B C GC .)
Since 1 and 2 are -finite,
S there exists a sequence {Cn }
n=1 such that 1 (Cn ) =
2 (Cn ) < , and = n=1 Cn . (Since is only a -system, we cannot assume that the Cn
are disjoint.) For each A F,
n
!
[
j (A) = lim j [Ci A] for j = 1, 2.
n
i=1

7
Since j ( ni=1 [Ci A]) can be written as a linear combination of values of j at sets of the
S
form A C, where Sn C is the intersection
Sn of finitely many of C1 , . . . , Cn , it follows from
A GC that 1 ( i=1 [Ci A]) = 2 ( i=1 [Ci A]) for all n, hence 1 (A) = 2 (A).

Exercise 12 Return to Example 17. You should now be able to answer the question posed
there.

Exercise 13 Suppose that = {a, b, c, d, e} and I tell you the value of P ({a, b}) and
P ({b, c}). For which subset of do I need to define P () in order to have a unique ex-
tension of P to a -field of subsets of ?

4 Lebesgue Measure and Caratheodorys Extension The-

orem
Let F be a cdf (nondecreasing, right-continuous, limits equal 0 and 1 at and respec-
tively). Let U be the field inPExample 5 (unions of finitely many Sn disjoint intervals). Define
n
: U [0, 1] by (A) = k=1 F (bk ) F (ak ) when A = k=1 (ak , bk ] and {(ak , bk ]} are
disjoint. This set-function is well-defined and finitely additive.
Is countably additive as probabilities are supposed to be? That is, if A =
S
i=1 Ai where
the Ai s are disjoint, each Ai is a union of finitely many disjoint intervals, and APitself is the
union of finitely many disjoint intervals (ak , bk ] for k = 1, . . . , n, does (A) = i=1 (Ai )?
First, take the collection of intervals that go into all of the Ai s and split them, if necessary,
so that each is a subset of at most one of the (ak , bk ] intervals. Then apply the following
result to each (ak , bk ].
S
Lemma 21 Let (a, b] = k=1 (ck , dk ] with the (ck , dk ]s disjoint. Then F (b) F (a) =
P
k=1 F (dk ) F (ck ).

that F (b)F (a) nk=1 F (dk )F (ck )

Proof: Since (a, b] nk=1 (ck , dk ] for all n, it follows P
S P
(because (ck , dk )s are disjoint), hence F (b) F (a) k=1 F (dk ) F (ck ). We need to prove
the opposite inequality.
Suppose first that both a and b are finite. Let  > 0. For each k, there is ek > dk such that

F (dk ) F (ek ) F (dk ) + k .
2

S is f > a such that F (a) F (f ) . Now, the interval [f, b] is compact and
Also, there
[f, b] k=1S(ck , ek ). So there are finitely many (ck , ek )s (suppose they are the first n) such
that [f, b] nk=1 (ck , ek ). Now,
n
X n
X
F (b) F (a) F (b) F (f ) +   + F (ek ) F (ck ) 2 + F (dk ) F (ck ).
k=1 k=1

8
Here we have to work with finitely many (ck , ek )s because we do not yet have countable
P
sub-additivity. It follows that F (b) F (a) 2 + k=1 F (dk ) F (ck ). Since this is true for
all  > 0, it is true for  = 0.
If = a < b < , let g > be such that F (g) < . The above argument shows that

X
X
F (b) F (g) F (dk g) F (ck g) F (dk ) F (ck ).
k=1 k=1
P
Since limg F (g) = 0, it follows that F (b) k=1 F (dk ) F (ck ). Similar arguments
work when a < b = and = a < b = .

In Lemma 21 you can replace F by an arbitrary nondecreasing right-continuous function

with only a bit more effort. (See the supplement following at the end of this lecture.)
The function defined in terms of a nondecreasing right-continuous function is a measure
on the field U. There is an extension theorem that gives conditions under which a measure
on a field can be extended to a measure on the generated -field. Furthermore, the extension
is unique.

Example 22 (Lebesgue measure) Start with the function F (x) = x, form the measure
on the field U and extend it to the Borel -field. The result is called Lebesgue measure,
and it extends the concept of length from intervals to more general sets.

Example 23 Every distribution function for a random variable has a corresponding proba-
bility measure on the real line.

Theorem 24 (Caratheodory Extension) Let be a -finite measure on the field C of

subsets of . Then has a unique extension to a measure on (C).

Exercise 14 In this exercise, we prove Theorem 24. Note that the uniqueness of the exten-
sion is a direct consequence of Theorem 20. We only need to prove the existence.
First, for each B 2 , define

X

(B) = inf (Ai ), (15)
i=1

{Ai }such that B

S
where the inf is taken over all i=1 i=1 Ai and Ai C for all i. Since C
is a field, we can assume that the Ai s are mutually disjoint without changing the value of
(B). Let

A = {B 2 : (C) = (C B) + (C B C ), for all C 2 }.

Now take the following steps:

9
1. Show that extends , i.e. that (A) = (A) for each A C.

2. Show that is monotone and subadditive.

3. Show that C A.

4. Show that A is a field.

5. Show that is finitely additive on A.

6. Show that A is a -field.

7. Show that is countably additive on A.

10
 Supplement: Measures from Increasing Functions
Lemma 21 deals only with functions F that are cdfs. Suppose that F is an unbounded
S If < a < b < , then the proof
nondecreasing function that is continuous from the right.
of Lemma 21 still applies. Suppose that (, b] = k=1 (ck , dk ] withPb < nd all (ck , dk ]
disjoint. Suppose that limx F (x) = . We want to show that k=1 F (dk ) F (ck ) =
. If one ck = , the proof is immediate, so assume that all ck > . Then there must
be a subsequence {kj } 0 0
j=1 such that limj ckj = . For each j, let {(cj,n , dj,n ]}n=1 be the
subsequence of intervals that cover (ckj , b]. For each j, the proof of Lemma 21 applies to
show that
X
F (b) F (ckj ) = F (d0j,n ) F (c0j,n ). (16)
n=1

As j , the left side of Equation 16 goes to while the right side eventually includes
every interval in the original collection.
A similar proof works for an interval of the form (a, ) when limx F (x) = . A combi-
nation of the two works for (, ).

11