Lecture 10: November 2, 2021: 1 Basics of Probability: The Finite Case
Lecture 10: November 2, 2021: 1 Basics of Probability: The Finite Case
∑ ν(ω ) = 1.
ω ∈Ω
We often refer to Ω as a sample space or outcome space and the function ν as a probability
distribution on this space. An event can be thought of as a subset of outcomes i.e., any
A ⊆ Ω defines an event, and we define its probability as
P [ A] = ∑ ν(ω ) .
ω∈ A
1
In a finite probability space, we define the expectation of a random variable X as:
E [ X ] := ∑ ν(ω ) · X (ω ) .
ω ∈Ω
An extremely useful fact about expectation is that is a linear transformation from the space
of random variables to R. In particular, if X and Y are random variables, then E [ X + Y ] =
E [ X ] + E [Y ], and E [c · X ] = c · E [ X ].
Proposition 1.1 (Linearity of Expectation) For any two random variables X and Y, E [ X + Y ] =
E [ X ] + E [Y ], and E [c · X ] = c · E [ X ].
E [X + Y] = ∑ ν(ω ) · ( X (ω ) + Y (ω )) = ∑ ν(ω ) · X (ω ) + ∑ ν ( ω ) · Y ( ω ) = E [ X ] + E [Y ] .
ω ∈Ω ω ∈Ω ω ∈Ω
Example: Card shuffling Suppose we unwrap a fresh deck of cards and shuffle it until
the cards are completely random. How many cards do we expect to be in the same position
as they were at the start? To solve this, let’s think formally about what we are asking. We
are looking for the expected value of a random variable X denoting the number of cards
that end in the same position as they started. We can write X as a sum of indicator random
variables Xi , one for each card, where Xi = 1 if the ith card ends in position i and Xi = 0
otherwise. These Xi are easy to analyze: P ( Xi = 1) = 1/n where n is the number of cards.
P ( Xi = 1) is also E [ Xi ]. Now we use linearity of expectation:
E [ X ] = E [ X1 + . . . + Xn ] = E [ X1 ] + . . . + E [ Xn ] = 1.
So, this is interesting: no matter how large a deck we are considering, the expected number
of cards that end in the same position as they started is 1.
1.3 Conditioning
2
Thus, one can define the conditional probability of an event B as
ν(ω ) P [ A ∧ B]
P [ B | A] = ∑ νA ( ω ) = ∑ P [ A]
=
P [ A]
.
ω∈B ω ∈ A∩ B
For a random variable X and an event A, we similarly define the conditional expectation of
X given A as
E [ X | A ] = ∑ νA ( ω ) · X ( ω ) ,
ω
Proposition 1.2 (Total Probability and Total Expectation) Let Ω be a finite “outcome space”
with probabiliy measure ν. Let A, B ⊆ Ω be events, and X : Ω → R be a random variable. Then
1. P [ B] = P [ A] · P [ B | A] + P [ Ac ] · P [ B | Ac ],
2. E [ X ] = P [ A] · E [ X | A] + P [ Ac ] · E [ X | Ac ].
1.4 Independence
Now that we have the notion of conditioning, we can define independence. Two non-zero
probability events A and B are independent if P [ A | B] = P [ A]. One can verify that this is
equivalent to P [ B | A] = P [ B]. In other words, restricting to one event does not change
the probability of the other event. Independence is a joint property of events and the
probability measure: one cannot make judgment about independence without knowing
the probability measure.
Two random variables X and Y defined on the same finite probability space are defined
to be independent if P [ X = x | Y = y] = P [ X = x ] for all non-zero probability events
{ X = x } := {ω : X (ω ) = x } and {Y = y} := {ω : Y (ω ) = y}.