1.

Probability Theory

1.1 Probability Theory as a Set of Outcome

 Terminology :

- Experiment: a probabilistic model, the output is not deterministic

- Sample space(Ω ¿: the set of all possible outcomes

- Sample points(ω ¿ : the elements of the sample space ω j , j=1,2 , …

- Events: subsets of sample space

%%% kim’s Comment

 Space:

Euclidean space / Linear space / Probability space / ….

 Deterministic: the output is determined, the fixed number, quantitative.

Probabilistic (Stochastic): the output is not determined.

%%%

Def. 1.1 An event is an outcome or a collection of outcomes. It is a set, and hence we

use set notation to denote an event, A ⊂Ω , called set A is a subset of Ω

 Roll a fair die Ω={ 1,2,3,4,5,6 },

 Roll an odd die Ω={ 1,2,3,4,6 }

 Roll a fair die: Is this Ω={ 1,2,3,4,5,6 ,7 } ?  No! there is no “7”

outcome of the experiment

1.2 Set Theory

Def. 1.4.

1) The sets A and B are equal to sets or identical sets iff A and B have the same
elements. We denote equality by writing A=B

2) A is included in B or A is a subset of B iff ω ∈ A implies ω ∈ B . In such cases, we

write A ⊂ B .

Prop. 1.6.

1) A=B iff A ⊂ B∧B⊂ A

2) If A ⊂B∧B⊂C , then A ⊂C

 Union A B:={ω :ω ∈ A∨ω ∈ B } --> logic as “or”

 Intersection A ∩ B:={ω : ω ∈ A∧ω ∈ B }  logic as “and”

Complement, A := { ω :ω ∉ A }
c


 Relative complement (difference) A−B :={ω :ω ∈ A∧ω ∉ B }.

 Prop.1.7.

1) A ⊂B ; then Bc ⊂ A c

2) ( A ∩ B )c = Ac ∪ B c

3) ( A B )c = Ac ∩B c
 Proof

1) By Contradiction: proof the claim is not correct.

a) ω ∈ B c and ω ∉ A c ↔ ω ∈ B c and ω ∈ A
b) If ω ∈ A , by assumption ω ∈ B

c) b) is contradict to a) since ω ∉ B  the proof is complete. QED

2) First we prove → direction i.e., if ( A ∪B ) ⊂ ( A ∩ B )

c c c

2.1) we know ( A ∩ B ) ⊂ A and ( A ∩ B ) ⊂ B .

2.2) Since ifA ⊂ B ; then Bc ⊂ A c (the first part of this proposition), Ac ⊂ ( A ∩ B )c

and B ⊂ ( A ∩ B ) . Hence ( A c ∪B c ) ⊂ ( A ∩ B )
c c c

2.3) Now we prove ← direction i.e., ( A ∩ B ) ⊂ ( A ∪ B )

c c
c

c
2.4) ∀ ω ∈ ( A ∩ B ) implies ω ∉ A or ω ∉ B or {ω ∉ A∧ω ∉ B }
c c c
2.5) Hence ω ∈ A ∨ω ∈ B ∨{ω ∈ ( A ∩ B ) }
c c
2.6) 2.5) implies ω ∈ A ∪ B (*remember the definition of Union)

 De Morgan’s law

a) Not (A and B ) < -- > not (A) or not( B)

b) Not( A or B) < -- > not(A) and not(B)

 HW_W1_P1: (page 22)

Prove

( A ∪B )c =A c ∩ B c
@@

 Definition: The null set ∅ : the set which is no elements.

1) In set theory, no definition of the null element.

2) The facts of ∅

-If the set A and B are no common elements, then A ∩ B=∅

c
- A ∩ A =∅
 c
-Ω =∅

-∅ c =Ω

-Since A ∩ Ac =∅ , this implies ∅ ⊂ A and ∅ ⊂ A c . Hence the ∅ ⊂ A ∀ A

 A set A and B is disjoint if A ∩ B=∅ .

 Question: the number of subsets given a set

1) Let Ω={ 1,2,3 } ,how many subsets of Ω

-{1},{2},{3},{1,2},{1,3},{2,3},[1,2,3] and ∅  8

 HW_W1_P2

Prove if the number of Ω = n, then number of subset of Ω is 2n

@@

1.3 Prob. Space and the Prob. Measure

 Axiom 1. Given an experiment, there exists a sample space, Ω , representing

the totality of possible outcome of the experiment and a collection, A , of
subsets, A, of Ω called events

- Sample space(Ω ¿: the set of all possible outcomes

- Event: a subset of sample space (Ω ¿

%%% Kim’s comment: What is axiom?

https://www.merriam-webster.com/dictionary/axiom

-In mathematics or logic, an axiom is an unprovable rule or first principle accepted as true
because it is self-evident or particularly useful.

https://www.merriam-webster.com/dictionary/axiom

-A statement that is taken to be true, to serve as a premise or stating point for further reasoning
and arguments.

 We do not need to prove the axiom. For example

1) It is possible to draw a straight line from any point to any other point.

2) a+ b=b+a

 Axiom 2. Each event A in A ,there can be assigned a nonnegative number

P(A) such that

P ( A )≥ 0 ,
P ( Ω )=1
 A ∩ B=∅ → P ( A ∪B )=P ( A ) + P(B)

Lemma 1.9. P ( A )=1−P( A)

c


Proof:
c c
A ∪ A =Ω, A ∩ A =∅
Implies P ( A ) + P ( A )=1 .QED
c

 Lemma 1.10. If A and B are two arbitrary events in the sample space Ω , then

P ( A ∪ B ) =P ( A )+ P ( B )−P( A ∩ B)
Proof:

A ∪B= A ∪ ( B∩ A )
c

Hence

P ( A ∪ B ) =P ( A )+ P ( B ∩ A )
c

And
P ( B )=P [ ( B ∩ A ) ∪ ( B ∩ A c ) ]=P ( B∩ A ) + P( B∩ A c ).
Hence,

P ( B ∩ A )=P ( B )−P ( B ∩ A )
c

which gives to

P ( A ∪ B ) =P ( A )+ P ( B )−P( A ∩ B)
1.4 Algebras of Sets and Probability Space

%%% Kim’s comment: measurable

- If the number of events is infinite, then something is weird…later or you may see
mathematic textbook for the measurable theory. It is to spend too much energy
to learn…!! But should be accounted for….In this course just to see a glimpse of
the measure theory….

- In order to define the probability we need the following questions be considered.

+ given a sample space Ω as, interval ( 0,1 ] , the experiment is to pick up a

number in the sample space Ω . As you know in real number it is decomposed of
rational and irrational number, Let define

R={ ω :rational number } , I ={ω :irrational number }

Then

1) For any sample point ω

ω ∈ R−→ ω ∉ I , and ω ∈ I −→ω ∉ R

 2) R ∩ I =∅
Now there are basic questions

3) How many sample points in R, i.e., how many rational number in Ω

In theoretically, it is infinite. Why?

Consider a series of rational number as

R 1= { 12 , 13 , 14 ,… … …}.
This set of R1 ⊂ R . How many points in R1 ? It is infinite. OK. Now how
many irrational numbers in I . It is infinite. Why? Given a rational number
a ∈ R , then for any irrational number b ∈ I ,
x=a+ b ( withthe constraint as x ∈Ω ) is an irrational number.
which implies the number of irrational numbers is infinite due to the fact of the
infinite of the rational number

4) Basically how many sample points in Ω ? So in theoretically it is infinite. OK..

5) In the previous chapter , it may be defined

P { Ω }=1
Now You may define a probability which is an interval as

{
P x :0< x ≤
1 1
=
2 2 }
Now it possible to define the probability P { R }∨P {I } ?

In measure theory,

P { R }=0 , but P { I }=1

It means given a interval ( 0 , 1 ] . If you pick up any number in ( 0 , 1 ] .

The number may be irrational number in probability 1. Do you agree this

fact?

- All these questions in answered in “Measure Theory”, “Number Theory”, or, in

“Real analysis” .

- We should learn “Measure Theory”, to study the probability. But as YOU EXPECT ,
let us just have a glimpse of all. WE DO NOT NEED SPEND LOTS OF ENERGY.

%%%

Def.1.12. An algebra, A , is a set of sets such that the following hold:

1) A ∈ A implies Ac ∈ A

2) A , B ∈ A implies ( A ∪ B) ∈ A
  Example (similar to Example 1.14)

Consider

- Ω={ 1,2 }, then A={ { 1,2 } , ∅ , { 1 } , {2 } } is an algebra

But set A={ { 1,2 } , ∅ , { 1 } } is not an algebra since { 1 }c = {2 } ∉ A

-Ω={ 1,2,3 } then set A={ { 1,2,3 } , ∅ , { 1 } , {2 } , { 3 } , {1,2 } , { 1,3 } } is not an

algebra since
c
{ 1 } = {2,3 } ∉ A
%%% Kim’s comment

Given a sample space Ω , there are several A .Consider Ω= { 1,2 }, then A={ { 1 } , {2 } , { 1,2 } , ∅ } is an
algebra. And A={ { 1,2 } , ∅ } is also an algebra.

%%%

Prop. 1.13. If A , B ∈ A and A is an algebra. then

1) A ∩ B∈ A
2) ( A−B ) ∈ A
3) Ω ∈ A and ∅ ∈ A

 Proof:

1) Since A is an algebra, Ac , B c ∈ A . Hence Ac ∪B c ∈ A .In addition,

c
( A c ∪ B c ) =( A ∩B ), implies ( A ∩ B ) ∈ A
c
2) By definition A−B= A ∩ B

3) A ∩ Ac =Ω and A ∩ Ac =∅ .QED
Exam 1.14 / 1.15 / Remark 1.16 (page 25)

Def. 1.17. A class of subsets of Ω is a σ −algebra , denoted A , if it is an algebra and

if it is also closed under countable unions, i.e.,
∞
A1 , A 2 ,… ∈ A → ∪ j=1 A j ∈ A

Axiom 3. Let Ω be a sample space , A a σ −algebra of subsets of Ω , and P a

probability measure defined on elements of A . Then, if Ai ∈ A is a countable of
disjoint sets, i.e., Ai ∩ A j =∅ if i≠ j , the probability of the union is found by
∞
P ( A 1 ∪ A 2 ∪… )=P ( ∪ ∞
j=1 A j ) =P ( A 1) + P ( A 2 )+ …=∑ P (A j )
j=1

 Axiom 2 is replace to Axiom 3.

Def. 1.19. If Ω is the set containing all possible outcomes of an experiment, A is a

σ −algebra of the subsets of Ω and P is a probability measure on A , then the triple
 ( Ω, A , P) is called a probability space.
 An example of uncountable operations.(skip but if you like, read it)

Let construct of space Ω :=( 0,1 ] by defining probabilities for intervals as being
the length of the intervals of which points are all equally possible. Define
probability of any interval as P ( ( a , b ] )=b−a.

For example, P ( ( 0,1 ] )=1−0=1 .

Let define an σ – algebra A as

A :={ the set of all countable operations like (a,b)}

1) The probability of any singletons {x }:

Since { x }=∩ n=1 ¿ , which is in A .

∞

Hence the probability

P ( { x } )=P( ( 0 , x ) ∪ ( x , 1 ] )¿ P ( ( 0 , x ) ) + P ( ( x ,1 ] ) ∵ disjoint¿ x−0+ (1−x )¿ 1

c

Hence P ( { x } )=1−P ( { x } ) =0
c

2) Uncountable set operation

Let forming the union of all points in ( 0,1 ] ,i . e . ,

∪ x ∈ (0,1 ] { x }=( 0,1 ] .

Thus

P ( ∪ x ∈ (0,1 ] { x }) = ∑ P ( { x } )=¿ P ( ( 0,1 ] ) .¿

x∈ ( 0,1 ]

Now the left of hand side

P ( ∪ x ∈ (0,1 ] { x }) =0 , ∵ P ( { x }=0 )

Right hand side

P ( ( 0,1 ] ) .=1

Which implies

0=1−→ contradict !
Hence uncountable set operation is not allowed in probability space.

1.5 Key Concepts in Probability Theory

Def. 1.24. A conditional probability is the probability of the occurrence of an event

subject to the hypothesis that another event has occurred
  Joint probability

The collection of events A1 , … , A n.

Joint probability: The probability of P ( A 1 ∩ A 2 ∩ , … ,∩ A n )

 Marginal probability: Let the sample space be partitioned into two

different families of disjoint sets, { A i } and { B j } , i . e . ,

Ω=∪mi=1 A i=∪nj=n B j
A p ∩ Aq =Bk ∩ Bl =∅ , p ≠ q , k ≠ l.

The marginal probability of B j

m
B j =∑ P ( A i ∩ B j )
i=1

Def. 1.24. A conditional probability is the probability of the occurrence of an

event subject to the hypothesis that another event has occurred.

 P ( B| A )
The probability of the event B given A = The conditional probability of B
given(occurred) A

 (Axiom/ definition.)
P( A ∩ B)
P ( B| A )=
P( A)
 Properties of the conditional probabilities
m
1) If Ω=∪i=1 A i, then the marginal probability of B is

m m
P ( B )=∑ P(B ∩ A i)¿ ∑ P ( B|A i ) P(A i )
i=1 i=1

2) Bayer’s rule:

P ( A|B ) P ( B )=P ( B| A ) P ( A )
 Independence (statistical independence):

Two events A and B are independent if

P ( A|B )=P( A)
Which is equivalent to

P ( A ∩ B )=P ( A ) P( B)
Ex. 1.25

Remark 1.26
 Orthogonality or mutual exclusivity or disjoint:

Two events, A and B, are orthogonal / mutually exclusive / disjoint if

P ( B| A )=0.
 Remember in linear system: Independent / orthogonal…

(in linear system:

- Two vectors are orthogonal  two vectors are independent

- Two vector are independent  not always independent.