Dire Dawa University

Institute of Technology

School of Electrical and Computer Engineering

Probability and Random Process

Basic concepts of probability

"March 2024"

1
Contents
2. Introduction to Probability .......................................................................................................... 3

2.1. Specifying Random Experiment ................................................................................................... 3

2.2. Mutually Exclusive Events ............................................................................................................ 7

2.3. Independent Events ........................................................................................................................ 7

2.4. Axiomatic Approachs to Probability .......................................................................................... 11

2.5. Basic Theorems of Probability .................................................................................................... 11

2.6. Conditional Probability ................................................................................................................ 12

2.7. Axioms of Conditional Probability............................................................................................. 14

2.8. Bayes’ Theorem ............................................................................................................................ 14

2.9. Law of Total Probability .............................................................................................................. 15

2
2. Introduction to Probability
What is probability?

➔ The chance or likelihood that an event will happen.

➔ It refers to how likely an event is to occur.

➔ Mathematically, probability can be defined as the ratio of number of ways an event
can occur (number of chances of occurrence of events) to the number of possible
outcomes (the set of all possible outcomes) of a random experiment.

Number of ways a certain event can occur

Probability =
Total possible outcomes of an experiment

2.1. Specifying Random Experiment

What is an experiment?

➔ Any process of observation or simply an attempt to produce a result.

➔ It need not only be a laboratory experiment.
➔ An experiment is said to be a random experiment if it is such that:
→ All the possible outcomes of the experiment are predictable (in advance).
→ The outcome of any trial of the experiment is not known (in advance).
→ It can be repeated any number of times under identical conditions.

➔ In a given random experiment, there exists an outcome, sample space, and an event.

→ Outcome: The result of observation in any random experiment.

→ Sample space: The set of all possible outcomes of a random experiment is known as
the sample space of the experiment and is usually denoted by S (or Ω). If S contains
only a finite number of elements, it is termed a finite sample space. If S contains a
countable number of elements, S may be called a countable sample space or discrete
sample space. Otherwise, S is called an uncountable sample space.

→ Event: Any subset of the sample space, and usually denoted by an upper-case English
alphabet such as A, B, C, …, Z.

3
Example 1: Tossing a coin and getting a head. In this experiment;

→ Possible outcomes: Head and Tail.

→ Sample space: S = {H, T}
→ Event: Getting a head, H.
→ If we are asked to find the probability of getting a head, it is determined as:

Number of event 1
P(getting a head) = =
Number of sample space 2

Exercise 1: In flipping two coins simultaneously, find the probability of the following.

(a) Getting at least one head, (b) Getting at most one head

Exercise 2: Find the probability of getting the sum of the faces is five in rolling two dice
simultaneously.

Example 2: Suppose a study is conducted on all families with one or two children. The possible
outcomes, in the order of births, are: boy only, girl only, boy and girl, girl and boy, both are boys
and both are girls. Then, the sample space is S = {b, g, bg, gb, bb, gg}. It is also a finite sample
space. Here, ‘b’ represents the child is a boy, and ‘g’ represents the child is a girl.

→ In this example, the event that the eldest child in the families is a girl is represented as:
A = {g, gb, gg}
→ The event that the families have one boy is represented as B = {b, bg}.

Example 3: Consider a certain bag contains 4 blue marbles, 5 red marbles, 1 green marble, and 2
black marbles inside. Suppose you select one marble at random, find the probability of getting the
following marbles.

a) Black d) Blue

b) Blue or black e) Not green

c) Not purple f) White (Answer: it is impossible to get it)

4
Directions (Examples 4 - 6): An urn contains some balls colored white, blue and green. The
probability of choosing a white ball is 4/15 and the probability of choosing a green ball is
2/5. There are 10 blue balls.

Example 4: What is the probability of choosing one blue ball?

Solution:

Probability of choosing one blue ball = 1 – (4/15 + 2/5) = 1/3

Example 5: Determine the total number of balls in the urn?

Solution

Probability of choosing one blue ball is 1/3 and total blue balls are 10. So with 10/30, we get
probability as 1/3. So total balls must be 30.

5
Example 6: If the balls are numbered 1, 2, …. up to number of balls in the urn, what is the
probability of choosing a ball containing a multiple of 2 or 3?

Solution

There are 30 balls in the urn. Multiples of 2 up to 30 = 30/2 = 15. Multiples of 3 up to 30 = 30/3 =
10 (take only whole number before the decimal part)

Multiples of 6 (2×3) up to 30 = 30/6 = 5

So total such numbers are = 15 + 10 – 5 = 20

Hence, the required probability = 20/30 = 2/3

Or,

15 10 5 20 2
P(2or3) = P(2) + P(3) − P(2 and 3) = + − = =
30 30 30 30 3
Example 8: Amen has five identical toys and one of them is underweight. His brother, Kaleb,
chooses one of these toys at random. Find the probability for Kaleb to choose an underweight toy.

Solution

It is seen from the figure above that, the sample space is given as S = {a1 , a2 , a3 , a4 , a5 }. The
events are A1, A2, A3, A4 and A5.
n(A) 1
Let A is Kaleb chooses the underweight toy, then: P(A) = =5
n(S)

1
Therefore, the probability for Kaleb to choose an underweight toy is 5.

Exercise 3: If 2 dice are thrown simultaneously, determine the probability of getting the following
events.

→ Having the same number

→ The first die is 3
→ The sum of numbers on the two dice is 7

6
 → The sum of numbers on the two dice is 13
→ The first die is 7?
→ The sum of the numbers on the two dice is an even number?

2.2. Mutually Exclusive Events

→ Two or more events are said to be mutually exclusive when the occurrence of any one
event excludes the occurrence of another event.
→ Mutually exclusive events cannot occur simultaneously.
→ In particular, events A and B are said to be mutually exclusive if they are disjoint, that
is, A ∩ B = Ø.

Example: Consider the case of rolling a die. Let A = {1, 2, 3} and B = {4, 5, 6} be two events.
Then we find A ∩ B = Ø. Hence A and B are said to be mutually exclusive events.

2.3. Independent Events

➔ Given two experiments, one following the other,

→ If the probability of each outcome of the second experiment does not depend on the
outcome of the first experiment, then we say that the two experiments are
independent.
→ If the probabilities of the outcomes of the second experiment depends on the outcome
of the first experiment, then we say that the two experiments are dependent. A sequence
of several different experiments is independent if the outcome of each experiment is
not affected by the outcomes of the preceding experiments.
➔ In general, two or more events are said to be independent, if they have no effect on the
occurrence of each other, otherwise the two events are dependent.

Example 1: If we roll a red die and then roll a white die, the outcome of the second die roll is not
influenced by the first. These two events are independent.

Example 2: If we roll a die and then flip a coin, the outcome of the coin flip is not influenced by
the die roll. These two events are independent.

Example 3: If we draw a card from a deck, and then draw a second card (without replacing the
first card), the outcome of the second draw is dependent on the first draw.

Product Rule for Independent Events

➔ For two independent experiments, each with an associated event, the probability of both
events occurring is the product of the probabilities of each of the events. For a sequence of

7
 multiple independent experiments, each with an associated event, the probability of all the
events occurring is the product of the probabilities of each of the events.

Example 4: Suppose we roll a die and flip a coin. What is the probability of rolling a 6 and flipping
heads?
1
These two experiments are independent. The probability of rolling a 6 is 6, and the
1
probability of flipping heads is 2. Hence, the probability of both happening is:
1 1 1
( )( ) =
6 2 12

Example 5: Suppose we roll a die, flip a coin, and draw a card from a deck. What is the probability
of rolling a 1, flipping tails, and drawing the number 2 of hearts?
1
These three experiments are independent. The probability of rolling a 1 is 6. The probability
1 1
of flipping tails is 2. The probability of drawing the number 2 of hearts is . Hence, the
52
probability of all three events occurring is:
1 1 1 1
( )( )( ) =
6 2 52 624

➔ In general, two events A and B are said to be independent of one another, if P(A ∩ B) =
P(A)P(B).
➔ For any two events A and B of a random experiment, if P(A/B) = P(A), then knowledge of
event B does not change the probability of the occurrence of event A. Such events are called
independent events.
From conditional probability, we have:
P(A ∩ B) = P(A)P(B)
P(A|B) = = P(A)
P(B) P(B)

=> P(A|B) = P(A) for independent events A and B

Similarly, P(B|A) = P(B) also indicates the independence of the events A and B.

8
Note: Consider the events A1, A2, …, An,
a) If they are mutually exclusive, then;
n n

P (⋃ Ai ) = ∑ P(Ai )
i=1 i=1

b) If they are independent, then;

n n

P (⋂ Ai ) = ∏ P(Ai )
i=1 i=1

Example 6: In tossing a fair coin twice, let the events A and B be defined as A: getting head on
the first toss, B: getting head on the second toss. Prove that A and B are independent events.

Solution:

The sample space of this experiment is

S = {HH, HT, TH, TT}

The unconditional probabilities of A and B are P(A) = 1/2 = P(B).

The event of getting heads in both the tosses is represented by A ∩ B. The outcome of the
experiment in favor of the occurrence of this event is HH. Hence, P(A ∩ B) = 1/4.

P(A ∩ B) = P(A)P(B) holds true. Thus, events A and B are independent events.

Example 7: In the experiment of rolling a pair of dice, the events A, B, and C are defined as A:
getting 2 on the first die, B: getting 2 on the second die, and C: sum of the faces of dice is an even
number. Prove that the events are pairwise independent but not mutually independent.

Solution

S = {(1,1), (1,2), (1,3), (1,4), (1,5), (1,6), (2,1), (2,2), (2,3), (2,4), (2,5), (2,6), (3,1), (3,2), (3,3),
(3,4), (3,5), (3,6), (4,1), (4,2), (4,3), (4,4), (4,5), (4,6), (5,1), (5,2), (5,3), (5,4), (5,5), (5,6), (6,1),
(6,2), (6,3), (6,4), (6,5), (6,6)}

n(S) = 36

The outcomes which are favorable to the occurrence of these events can be listed below:

A = {(2,1), (2,2), (2,3), (2,4), (2,5), (2,6)}; n(A) = 6

P(A) = 6/36 = 1/6

9
B = {(1,2), (2,2), (3,2), (4,2), (5,2), (6,2)}; n(B) = 6

P(B) = 6/36 = 1/6

C = {(1,1), (1,3), (1,5), (2,2), (2,4), (2,6), (3,1), (3,3), (3,5), (4,2), (4,4), (4,6), (5,1), (5,3), (5,5),
(6,2), (6,4), (6,6)}; n (C) = 18, P(C) = 18/36 = 1/2

A ∩ B = {(2,2)}; n (A ∩ B) = 1, P(A ∩ B) = 1/36

A ∩ C = {(2,2), (2,4), (2,6)}, n(A ∩ C) = 3, P(A ∩ C) = 3⁄36

B ∩ C = {(2,2), (4,2), (6,2)}, n(B ∩ C) = 3, P(B ∩ C) = 3⁄36

A ∩ B ∩ C = {(2,2)}, n(A ∩ B ∩ C) = 1, P(A ∩ B ∩ C) = 1⁄36

The following relations may be obtained from these probabilities

Hence, the above relations show that when the events A, B, and C are considered in pairs, they are
independent. But, when all three events are considered together, they are not independent.

10
2.4. Axiomatic Approachs to Probability

Let S be the sample space of a random experiment. If a number P(A) assigned to each event A ∈
S satisfies the following axioms, then P(A) is called the probability of A.

→ Axiom-1: P (A) ≥ 0
→ Axiom-2: P (S) = 1
→ Axiom-3: If {A1, A2, …} is a sequence of mutually exclusive events i.e., Ai ∩ Aj = ϕ
when i ≠ j, then:

Axiom-3 also holds for a set of a finite number of mutually exclusive events. If A1, A2 … , and An
are mutually exclusive events in S and ‘n’ is a finite positive integer, then P(A1 ∪ A2 … An ) =
P(A1 ) + P(A2 ) + … + P(An )

2.5. Basic Theorems of Probability

Theorem 1: The probability of an impossible event is 0, i.e., P(ϕ) = 0.

Proof:

Let A1 = S and A2 = ϕ. Then, A1 and A2 are mutually exclusive.

S = A1 ∪ A2 = S ∪ ϕ, Thus, by axiom-3, P(S) = P(S) + P(ϕ)

Since by axiom-2, P(S) = 1, 1 = 1 + P(ϕ)

Hence, P(ϕ) = 0

Theorem 2: If S is the sample space and A is any event of the experiment, then
̅) = 1 − P(A)
P(A

Exercise: Proof theorem 2 above.

Corollary: If A ⊂ B, then P(A) ≤ P(B).

Proof:

Since, by Axiom-1, P(B– A) ≥ 0, it follows that

P(B) – P(A) ≥ 0

P(B) ≥ P(A) => P(A) ≤ P(B)

11
2.6. Conditional Probability

➔ It is the probability of an event based on a condition.

➔ It is the probability that an event occurs given the knowledge that another event has occurred.
➔ Mathematically, for any events A and B, conditional probability of A given B is defined as:

P(A ∩ B)
P(A|B) = for P(B) > 0
P(B)

Similarly,
P(B ∩ A)
P(B|A) = for P(A) > 0
P(A)

Note: P(A ∩ B) = P(B ∩ A)

Example 1: A number is selected randomly from the digits 11 through 19. Consider the events
are given as: A = {11, 14, 16, 18, 19}, B = {12, 14, 18, 19}, C = {13, 15, 18, 19},

Find the following.

→ P(A/B)
→ P(A/C)
→ P(B/C)
→ P (B/A)

Solution

Given: A = {11, 14, 16, 18, 19}, B = {12, 14, 18, 19}, C = {13, 15, 18, 19}

A ∩ B = {14, 18, 19}, A ∩ C = B ∩ C = {18, 19}

5 4
P(A) = , P(B) =
9 9
3 2
P(A ∩ B) = , P(A ∩ C) = P(B ∩ C) =
9 9
Therefore, the probability for the occurrence of A given that B has occurred is:

P(A ∩ B) 3
P(A|B) = =
P(B) 4

The probability for the occurrence of A given that C has occurred is:

P(A ∩ C) 1
P(A|C) = =
P(C) 2

12
Similarly, the conditional probability of B given C is:

P(B ∩ C) 1
P(B|C) = =
P(C) 2

Conditional probability of B given A is given by:

P(B ∩ A) 3
P(B|A) = =
P(A) 5

Example 2: A pair of dice is rolled and the faces are noted. Let A be the sum of the faces is odd, B
be the sum of the faces exceeds 8, and C be the faces are different, then find (i) P (A/C) (ii) P (B/C).

Solution:

The outcomes favorable to the occurrence of the events are:

A = {(1,2), (1,4), (1,6), (2,1), (2,3), (2,5), (3,2), (3,4), (3,6), (4,1), (4,3), (4,5), (5,2), (5,4), (5,6),
(6,1), (6,3), (6,5)}

B = {(3,6), (4,5), (4,6), (5,4), (5,5), (5,6), (6,3), (6,4), (6,5), (6,6)}

C = {(1,2), (1,3), (1,4), (1,5), (1,6), (2,1), (2,3), (2,4), (2,5), (2,6), (3,1), (3,2), (3,4), (3,5), (3,6),
(4,1), (4,2), (4,3), (4,5), (4,6), (5,1), (5,2), (5,3), (5,4), (5,6), (6,1), (6,2), (6,3), (6,4), (6,5)}

Since A and B are proper subsets of C, A ∩ C = A and B ∩ C = B.

Hence, the probability for the sum of the faces is an odd number given that the faces are different
is given by:

13
Similarly, the probability for the sum of the faces exceeds 8 given that the faces are different is

Exercise
[1] In rolling a die, consider the events are: A = {1, 2, 3, 4, 5}, and B = {3, 4, 5, 6}, find
P(A|B).
[2] A six-sided die is rolled, find the probability of getting the number is 5 given that an odd
number has occurred.
[3] Suppose two six-sided dice are rolled. What is the probability of the event that the sum of
the two dice is 7 given that the first die is 4?

2.7. Axioms of Conditional Probability

➔ The conditional probabilities also must satisfy the same axioms introduced earlier.
➔ If S is the sample space of a random experiment and B is an event in the experiment, then
→ Axiom 1: P(A/B) ≥ 0 for any event A of S.
→ Axiom 2: P(S/B) = 1
→ Axiom 3: If A1, A2, … is a sequence of mutually exclusive events, then

2.8. Bayes’ Theorem

➔ In statistics and probability theory, the “Bayes’ theorem” which is also known as the
“Bayes’ rule” is a mathematical formula used to determine the conditional probability of
events.
➔ Essentially, Bayes’ theorem describes the probability of an event based on prior knowledge
of the conditions that might be relevant to the event. The theorem is named after the English
statistician, Thomas Baye.
➔ The conditional probability for two events A and B are given in the earlier section:

𝑃(𝐴 ∩ 𝐵)
𝑃(𝐴|𝐵) = 𝑓𝑜𝑟 𝑃(𝐵) > 0
𝑃(𝐵)
𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐵)𝑃(𝐴|𝐵) … … … … … … … … … . . … (1)

14
And similarly,

𝑃(𝐵 ∩ 𝐴)
𝑃(𝐵|𝐴) = 𝑓𝑜𝑟 𝑃(𝐴) > 0
𝑃(𝐴)
𝑃(𝐵 ∩ 𝐴) = 𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐴)𝑃(𝐵|𝐴) … … … … . . (2)

𝑃(𝐵 ∩ 𝐴) 𝑃(𝐴 ∩ 𝐵)
𝑃(𝐵|𝐴) = = … … … … … … . . (3)
𝑃(𝐴) 𝑃(𝐴)

But we have in (1) above that; 𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐵)𝑃(𝐴|𝐵),

Now (3) will become:

𝑃(𝐴 ∩ 𝐵) 𝑃(𝐵)𝑃(𝐴|𝐵)
𝑃(𝐵|𝐴) = = … … … … … (4)
𝑃(𝐴) 𝑃(𝐴)

Similarly,

𝑃(𝐴 ∩ 𝐵) 𝑃(𝐴)𝑃(𝐵|𝐴)
𝑃(𝐴|𝐵) = = … … … … . . (5)
𝑃(𝐵) 𝑃(𝐵)

Equations (4) and (5) are the “𝑩𝒂𝒚𝒆𝒔 𝒓𝒖𝒍𝒆”.

Where; 𝑃(𝐴|𝐵) – Probability of event A occurring given that event B has occurred

𝑃(𝐵|𝐴) – Probability of event B occurring given that event A has occurred

𝑃(𝐴) – Probability of event A, and 𝑃(𝐵) – Probability of event B

2.9. Law of Total Probability

In probability theory, the law of total probability is a useful way to find the probability of some
event A when we don’t directly know the probability of A but we do know that events B1, B2, B3…
form a partition of the sample space S.

➔ The law states that; If B1, B2, B3… form a partition of the sample space S, then we can
calculate the probability of event A as:

𝑷(𝑨) = ∑ 𝑷(𝑨|𝑩𝒊 )𝑷(𝑩𝒊 )

15
Or,

 𝑃(𝐴) = ∑ 𝑃(𝐴|𝐵𝑖 )𝑃(𝐵𝑖 )

Following the same principle as in equation (4) above, the Bayes theorem becomes:

Proof of Bayes Theorem:

For each event 𝐵𝑗, 𝑗 = 1, 2, … , 𝑛 by the definition of conditional probability,

Example 1: Suppose there are two bags in a box, which contain the following marbles:

→ Bag 1: 7 red marbles and 3 green marbles

→ Bag 2: 2 red marbles and 8 green marbles

If we randomly select one of the bags and then randomly select one marble from that bag, what is
the probability that it’s a green marble?

16
Solution

In this example, let P(G) = probability of choosing a green marble. This is the probability that
we’re interested in, but we can’t compute it directly.

Instead we need to use the conditional probability of G, given some events B where the Bi‘s form
a partition of the sample space S. In this example, we have the following conditional probabilities:

→ P(G|B1) = 3/10 = 0.3

→ P(G|B2) = 8/10 = 0.8

Thus, using the law of total probability we can calculate the probability of choosing a green marble
as:

→ P(G) = Σ{P(G|Bi)*P(Bi)}
→ P(G) = P(G|B1)*P(B1) + P(G|B2)*P(B2)
→ P(G) = (0.3)*(0.5) + (0.8)*(0.5) = 0.55

Hence, If we randomly select one of the bags and then randomly select one marble from that bag,
the probability we choose a green marble is 0.55.

Example 2: Company A supplies 80% of widgets for a car shop and only 1% of their widgets turn
out to be defective. Company B supplies the remaining 20% of widgets for the car shop and 3%
of their widgets turn out to be defective.

If a customer randomly purchases a widget from the car shop, what is the probability that it will
be defective?

17
Solution

If we let P(D) = the probability of a widget being defective and P(Bi) be the probability that the
widget came from one of the companies, then we can compute the probability of buying a defective
widget as:

→ P(D) = ΣP(D|Bi)*P(Bi)
→ P(D) = P(D|B1)*P(B1) + P(D|B2)*P(B2)
→ P(D) = (0.01)*(0.80) + (0.03)*(0.20)
→ P(D) = 0.014

If we randomly buy a widget from this car shop, the probability that it will be defective is 0.014.

Example 3: Suppose a computer component is given a quality score of A, B, or C after production.

On average, 70% of the components were given a quality score of A, 18% were given a quality
score of B, and 12% were given a quality score of C. In addition, it is found that 2% of the
components given a quality score of A eventually failed and the failure rate was 10% for the
components given a quality score of B, and the failure rate was 18% for the components given a
quality score C. Find the probability that it had received a quality score of B if a component is
failed.

Solution:

Let A: be a computer component that is given a quality score of A.

B: be a computer component that is given a quality score of B.

C: be a computer component that is given a quality score of C.

The corresponding probability of each computer component is given as:

𝐏(𝐀) = 𝟎. 𝟕 − − − 𝐏(𝐅|𝐀) = 𝟎. 𝟎𝟐

𝐏(𝐁) = 𝟎. 𝟏𝟖 − − − 𝐏(𝐅|𝐁) = 𝟎. 𝟏

𝐏(𝐂) = 𝟎. 𝟏𝟐 − − − 𝐏(𝐅|𝐂) = 𝟎. 𝟏𝟖

Question: 𝐏(𝐁|𝐅) = ?

Using Bayes rule, we will have:

𝐏(𝐅|𝐁)𝐏(𝐁) 𝐏(𝐅|𝐁)𝐏(𝐁)
𝐏(𝐁|𝐅) = =
𝐏(𝐅) 𝐏(𝐅|𝐀)𝐏(𝐀) + 𝐏(𝐅|𝐁)𝐏(𝐁) + 𝐏(𝐅|𝐂)𝐏(𝐂)

18
 (𝟎. 𝟏𝟖)(𝟎. 𝟏)
𝐏(𝐁|𝐅) = = 𝟎. 𝟑𝟑𝟓𝟖
(𝟎. 𝟎𝟐)(𝟎. 𝟕) + (𝟎. 𝟏)(𝟎. 𝟏𝟖) + (𝟎. 𝟏𝟐)(𝟎. 𝟏𝟖)

Exercise 1: Suppose in a certain bolt factory; machines A, B, and C manufacture total bolts of
about 25%, 35%, and 40% respectively. Of their outputs, 5%, 4%, and 2% are respectively
defective bolts. A bolt is drawn at random from the product. If the bolt drawn is found defective,
what is the probability that it is manufactured from machine A? Ans: P(A|D) = 0.36

Exercise 2: A student buys 1000 ICs from supplier A, 2000 ICs from supplier B, and 3000 ICs
from supplier C. He tested the ICs and found that the conditional probability of an IC being
defective depends on the supplier from whom bought. Specifically; given that an IC came from
supplier A, the probability that it is defective is 0.05; given that an IC came from supplier B, the
probability that it is defective is 0.1; and given that an IC came from supplier C, the probability
that it is defective is 0.1. If the ICs from the three suppliers are mixed and one is selected at random,

(a) What is the probability that it is defective? Ans: 𝐏(𝐃) = 𝟎. 𝟎𝟗𝟏𝟕

(b) What is the probability that it came from supplier A given that a randomly selected IC is
defective? Ans: 𝐏(𝐀|𝐃) = 𝟎. 𝟎𝟗𝟏

19