ECE 603

Probability and Random

Processes
Lessons 1-3
Chapter 1
Basic Concepts

© 2020 UMass Amherst Global. All rights reserved.

Objectives
• Review set theory
• Explore random experiments and probabilities
• Examine conditional probability
• Examine independence
• Review law of total probability
• Review Bayes’ rule
• Explore conditional independence

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

2
Rationale
• An exploration of the basic concepts of probability will provide a
foundation for discussion.
• Mathematical concepts are necessary for understanding probability
theory.
• This lesson focuses on random experiments and the axioms of probability.
• You will explore discrete and continuous probability models, before
discussing conditional probability.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

3
Prior Learning
• A level of mathematical maturity consistent with the graduate engineering
level.
• Access to the online textbook: https://www.probabilitycourse.com/

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

4
Review of Set Theory
A set is an unordered collection of things (elements).

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

5
Review of Set Theory
• The set of natural numbers,

• The set of integers,

• The set of real number

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

6
Review of Set Theory
Set is a subset of set if every element of is also an element of
We write where " " indicates "subset ".

Example:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

7
Review of Set Theory
if and only if and
Example:

Universal set: The set of all things that we could possibly consider in a given
context.
Universal set
Null set ;
For any set

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

8
Venn Diagrams
In a Venn diagram any set is depicted by a closed surface.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

9
Set Operations
Union: The union of two sets and is denoted by and consist of
all objects in or

if and only if or

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

10
Set Operations
Intersection:
The intersection of two sets and is denoted by and consist of all
objects in both and

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

11
Set Operations
Complement:
The complement of a set denoted by is the set of all elements in
that are Not in

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

12
Set Operations
Difference (subtraction):
The subtraction of set from is all elements in that are not
in

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

13
Set Operations
Mutually exclusive set (disjoint):
Two sets and are mutually exclusive (or disjoint) if 

 are m.e. if

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

14
Set Operations
Partition: 
A collection of sets is a Partition of if
a) They are disjoint .
b)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

15
Set Operations
Theorem : De Morgan's law

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

16
Set Operations
Example:
Let and

a) b) c)

d) f) g)

 The sets form a partition of

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

17
Set Operations
Theorem : Distributive law

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

18
Functions

Domain
Co-domain

Range: the set of all the possible values of (Range )

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

19
Functions
Example:
Consider the function defined as

 one-to-one (invertible):

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

20
Countable and Uncountable Sets
Cardinality of a set is the number of elements in

 set is finite if

 set if countable if it is finite Or the elements of can be enumerated or

listed in a sequence that is,

Ex: is countable.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

21
Countable and Uncountable Sets
Uncountable: Not countable.
e.g.,

Equivalently: set is countably infinite if it is in one-to-one correspondence

with

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

22
Countable and Uncountable Sets
Example:
(set of integers) is countable (countably infinite).
Because

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

23
Countable and Uncountable Sets
Example:
Show that a set of the form is countable.

Example:
Show that the positive rational number form a countable set:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

24
Countable and Uncountable Sets
Example:
Show that the positive rational number form a countable set:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

25
Countable and Uncountable Sets
But is not countable.
In fact, any interval where is not countable.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

26
Orchestrated Conversation:
Review and Discussion
Review of Video and Exercises from Lesson 1 Video 1

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

27
Random experiment
Random experiment: A phenomenon whose outcome cannot be
predicted with certainty, such as
Random experiment:
• Roll a die
• Roll a die three times
• Flip a coin

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

28
Random experiment
Outcome:
An outcome is the result of a random experiment.
• Roll a die 3
• Roll a die 3 times (2, 3, 6)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

29
Random experiment

Events:
An event is collection of possible outcomes.
• Roll a die (Event=E)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

30
Random experiment
Sample Space:
The sample space is the set of all possible outcomes.
• Roll a die: random experiment

• Roll a die three times

an outcome

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

31
Random experiment
 Event Set
 Sample space Universal Set
 Outcome Element

We say that an event occurs if the outcome of the experiment is an

element of

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

32
Random experiment
Partition:
A partition is a collectively exhaustive, and mutually exclusive set of events, i.e.,

is a Partition if

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

33
Summary of Random experiment
a) Review of set theory
b) Random experiments: Roll a die, etc.
 Outcome: An outcome is a result of random experiment.
• Roll a die 3
• Roll a die three times (3,6,2)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

34
Summary of Random experiment
 Sample Space: The set of all possible outcomes
• Roll a die

 Event: An event is a collection of possible outcomes.

An event is subset of
• Roll a die :

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

35
Summary of Random experiment
 We also say that an event has occurred if the outcome of the experiment is an
element of
• Roll a die 2,

• Roll a die 3 times

elements.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

36
Probability
Event Probability of
We assign a probability to every event
The portion of times event is observed in a large number of runs of
the experiment.

events

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

37
Probability
Axioms of Probability
Definition. A probability measure is a function that maps events in the
sample space to real numbers. Such that:
1) For any event

2) Probability of the sample space is

3) For any countable collection of disjoint events

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

38
Orchestrated Conversation:
Review and Discussion
Review of Video and Exercises from Lesson 1 Video 2

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

39
Probability
Roll a fair die (fair: outcomes are equally likely).

disjoint

3rd axiom:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

40
Probability

disjoint

 Equally likely outcomes:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

41
Probability
Using the axioms:
1) what is

disjoint

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

42
Probability
2)
empty

3)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

43
Probability
4)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

44
Probability
5)

Use Venn diagram.

6)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

45
Probability
Example:
Roll a die twice and observed and

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

46
Orchestrated Conversation:
Review and Discussion
Review of Video and Exercises from Lesson 1 Video 3

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

47
Whiteboard

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

48
Summary
Probability:

 Finite Sample Space with equally likely outcomes:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

49
Sample Space
Sample Space:
a) Countable:

Discrete Probability Space

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

50
Sample Space
b) Uncountable
Continuous Probability Space

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

51
Continuous Probability Space
Example: I choose a point completely at random in

a)

b)

c)

d)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

52
Continuous Probability Space
Key point: Axioms of Probability applies to continuous probability spaces.

Example: Suppose we know that the probability that a certain machine lasts
more than or equal to years is :

Find the following sets:

a)
b)
c)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

53
Conditional Probability
Suppose that in a certain city, 0.3 percent of the days are rainy.

The probability that it rains given that it is cloudy might be:

Conditional probability: The probability given

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

54
Conditional Probability
Example. Roll a fair die, what is the probability that the outcome is an even
number given it was less than or equal to 3, i.e.,

Note:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

55
Conditional Probability
Definition: The Conditional Probability , the probability that
occurred given that has occurred is given by

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

56
Summary of Probability
Axioms of Probability:
a) For any event
b) Probability of the sample space is
c) If are disjoint events, then

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

57
Conditional Probability
If and are two events in a sample space then

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

58
Conditional Probability
Conditional probability satisfies the probability axioms :

a) For any event

b) Conditional probability of given is

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

59
Conditional Probability
c) If are disjoint events, then

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

60
Conditional Probability
Example. Roll two dice
: 3 dots are shown at least on one die

Find

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

61
Conditional Probability
Special cases:
1) and are disjoint:

2)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

62
Conditional Probability
3)

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

63
Conditional Probability
Example. Roll a die, what is the probability that it is larger than or equal to 5,
given that it is an even number ?

, even number

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

64
Conditional Probability
Definition: Two events and are independent if and only if

equivalently

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

65
Conditional Probability
Warning!
Disjoint (mutually exclusive) Independent
Disjoint:
Independent:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

66
Conditional Probability
Suppose and are disjoint:
If

If disjoint Not independent.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

67
Conditional Probability
Example:

independent

Event that there is a connection from node to node

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

68
Conditional Probability
Remark:
1)

2) If are independent, then

a) are independent.

b) are independent.

c) are independent.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

69
Conditional Probability
Example:

independent

Find
Note that are independent.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

70
Conditional Probability
If and are two events in a sample space then the conditional
probability of given is defined as

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

71
Independence
Two events and are independent if and only if

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

72
Independence
Three events and are independent if all of the following conditions
hold

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

73
Independence
Example.
Two darts players throw alternately at a board and the first to score a bull wins.
On each of their throws player has probability and player of
success; the result of different throws are independent. If starts, calculate
the probability that he/she wins.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

74
Law of Total Probability
Let be a partition of the sample space with
For any event we have

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

75
Law of Total Probability

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

76
Orchestrated Conversation:
Law of Total Probability
Example:
Three coins are in a bag:
a) Coin 1: probability of heads is 0.9.
b) Coin 2: probability of heads is 0.6.
c) Coin 3: probability of heads is 0.3.
I draw a coin at random and toss it. What is the probability of heads?

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

77
Bayes' Rule
For any two events and , where , we have

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

78
Bayes' Rule
If is a partition of the sample space , and is any event
with we have

Lesson 61
Chapter © 2020 UMass Amherst Global. All rights reserved.
79
Orchestrated Conversation:
Bayes' Rule
Example.
In the previous problem, suppose that we know the result is heads; what is the
probability that Coin 1 was chosen?

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

80
Bayes' Rule
Example. In a communication system a zero or a one is transmitted with the
probability or
respectively. Due to the noise in the channel, a zero can be received as a one,
with probability and a one can be received as a zero also with probability .
A one was observed, what is the probability that a one was transmitted?

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

81
Conditional Independence
Two events and are independent if and only if

Two events and are conditionally independent given an event if and

only if

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

82
Chain Rule for Conditional Probability

We can extend this to 3 or more events:

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

83
Summary of this Lesson
You explored the basic concepts of probability that will provide a foundation for
discussion of probability throughout this term. You also reviewed mathematical
concepts needed to understand probability theory. You had the opportunity to
examine random experiments and the axioms of probability. Additionally, you
explored discrete and continuous probability models and discussed conditional
probability.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

84
 Post-work for Lessons 1-3
• Complete homework assignment for Lessons 1-3: HW#1

Go to the online classroom for details.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

85
To Prepare for the Next Lesson
• Read Chapter 2 in your online textbook:
https://www.probabilitycourse.com/chapter2/2_1_0_counting.php
• Complete the Pre-work for Lesson 4.

Visit the online classroom for details.

Chapter 1 © 2020 UMass Amherst Global. All rights reserved.

86