HoeI

Port
Stone
Introduction

to Probability
Theory

'.

-:

. ...

'

~~~~~: ~-

. .

._.-

The Houghton Mi8lin Series in Statistics

under the Editorship of Herman Chernoff

LEO BREIMAN

Probability and Stochastic Processes: With a View Toward Applications

Statistics: With a View Toward Applications
PAUL G . HOEL, SIDNEY C . PORT, AND CHARLES 1. STONE

Introduction to Probability Theory

Introduction to Statistical Theory
Introduction to Stochastic Processes
PAUL F. LAZARSFELD AND NEIL W . HENRY

Latent Structure Analysis

GO'ITFIUED E. NOETHER

Introduction to Statistics-A Fresh Approach

Y. S. CHOW, HERBERT ROBBINS, AND DAVID SIEGMUND

Great Expectations: The Theory of Optimal Stopping

I. RICHARD SAVAGE

Statistics: Uncertainty

and Behavior

Introduction to
Probability
Theory
Paul G. Hoel
Sidney C. Port
Charles J. Stone
University of California, Los Angeles

HOUGHTON MIFFLIN COMPANY

Atlanta Dallas Geneva, Illinois

Palo Alto London

BOSTON

Hopewell, New Jersey

COPYRIGHT

1971

BY HOUGHTON MIFFLIN COMPANY.

All rights reserved. No part of this work may be reproduced or transmitted in

any form or by any means, electronic or mechanical, including photocopying and
recording, or by any information storage or retrieval system, without permission
in writing from the publisher.
PRINTED IN THE U.S.A.
LIBRARY OF CONGRESS CATALOG CARD NUMBER:
ISBN:

0-39'5-04636-X

74-136173

General Preface

This three-volume series grew out of a three-quarter course in probability,

statistics, and stochastic processes taught for a number of years at UCLA. We felt
a need for a series of books that would treat these subjects in a way that is well
coordinated, but which would also give adequate emphasis to each subject as being
interesting and useful on its own merits.
The first volume, Introduction to Probability Theory, presents the fundamental
ideas of probability theory and also prepares the student both for courses in
statistics and for further study in probability theory, including stochastic processes.
The second volume, Introduction to Statistical Theory, develops the basic
theory of mathematical statistics in a systematic, unified manner. Together, the
first two volumes contain the material that is often covered in a two-semester course
in mathematical statistics.
The third volume, Introduction to Stochastic Processes, treats Markov chains,
Poisson processes, birth and death processes, Gaussian processes, Brownian
motion, and processes defined in terms of Brownian motion by means of elementary stochastic differential equations.

Preface

This volume is intended to serve as a text for a one-quarter or one-semester

course in probability theory at the junior-senior level. The material has been
designed to give the reader adequate preparation for either a course in statistics or
further study in probability theory and stochastic processes. The prerequisite for
this volume is a course in elementary calculus that includes multiple integration.
We have endeavored to present only the more important concepts of probability
theory. We have attempted to explain these concepts and indicate their usefulness
through discussion, examples, and exercises. Sufficient detail has been included in
the examples so that the student can be expected to read these on his own, thereby
leaving the instructor more time to cover the essential ideas and work a number of
exercises in class.
At the conclusion of each chapter there are a large number of exercises, arranged
according to the order in which the relevant material was introduced in the text.
Some of these exercises are of a routine nature, while others develop ideas introduced in the text a little further or in a slightly different direction. The more difficult
problems are supplied with hints. Answers, when not indicated in the problems
themselves, are given at the end of the book.
Although most of the subject matter in this volume is essential for further study
in probability and statistics, some optional material has been included to provide
for greater flexibility. These optional sections are indicated by an asterisk. The
material in Section 6.2.2 is needed only for Section 6.6; neither section is required
for this volume, but both are needed in Introduction to Statistical Theory.
The material of Section 6.7 is used only in proving Theorem 1 of Chapter 9 in this
volume and Theorem I of Chapter 5 in Introduction to Statistical Theory. The
contents of Chapters 8 and 9 are optional; Chapter 9 does not depend on Chapter 8.
We wish to thank our several colleagues who read over the original manuscript
and made suggestions that resulted in significant improvements. We also would
like to thank Neil Weiss and Luis Gorostiza for obtaining answers to all the
exercises and Mrs. Ruth Goldstein for her excellent job of typing..
vii

Table of Contents

Probability Spaces

1.1

2
6

1.2
1.3
1.4

1.5

Combinatorial Analysis
2.1

2.2
2.3
2.4
*2.5
*2.6
*2.7
*2.8

10

14
18
27

Ordered samples
Permutations
Combinations (unordered samples)
Partitions
Union of events
Matching problems
Occupancy problems
Number of empty boxes

27
30
31
34
38
40

43
44

Discrete Random Variables

49

3.1
3.2
3.3
3.4

50
57

3.5
3.6

Examples of random phenomena

Probability spaces
Properties of probabilities
Conditional probability
Independence

Definitions
Computations with densities
Discrete random vectors
Independent random variables
3.4.1 The multinomial distribution
3.4.2 Poisson approximation to the binomial distribution
Infinite sequences of Bernoulli trials
Sums of independent random variables

Expectation of Discrete Random Variables

4.1

4.2

Definition of expectation
Properties of expectation
ix

60

63
66
69

70
72
82
84
85

Contents

4.3
4.4
4.5
4.6

Moments
Variance of a sum
Correlation coefficient
Chebyshev's Inequality

92
96
99
100

Continuous Random Variables

109
110
112
115
117
123
124
124
126
128
131

5.1
5.2

5.3

*5.4

Jointly Distributed Random Variables

139

6.1
6.2

Properties of bivariate distributions

Distribution of sums and quotients
6.2.1 Distribution of sums
*6.2.2 Distribution of quotients
Conditional densities
6.3.1 Bayes' rule
Properties of multivariate distributions
Order statistics
Sampling distributions
Multidimensional changes of variables

139
145
145
150
153
155
157
160
163
166

Expectations and the Central Limit Theorem

173
173
174
177
181
183
186
190

6.3
6.4.
*6.5
*6.6
*6.7

7.1
7.2
7.3
7.4
7.5

*8

Random variables and their distribution functions

5.1.1 Properties of distribution functions
Densities of continuous random variables
5.2.1 Change of variable formulas
5.2.2 Symmetric densities
Normal, exponential, and gamma densities
5.3.1 Normal densities
5.3.2 Exponential densities
5.3.3 Gamma densities
Inverse distribution functions

Expectations of continuous random variables

A general definition of expectation
Moments of continuous random variables
Conditional expectation
The Central Limit Theorem
7.5.1 Normal approximations
7.5.2 Applications to sampling

Moment Generating Functions and Characteristic

Functions
8.1
8.2

Moment generating functions

Characteristic functions

197
197
200

Contents

xi

8.3
8.4

*9

Inversion formulas and the Continuity Theorem

The Weak Law of Large Numbers and the Central
Limit Theorem

205
209

Random Walks and Poisson Processes

216

9.1
9.2
9.3
9.4
9.5

216
219
225
228
230

Random walks
Simple random walks
Construction of a Poisson process
Distance to particles
Waiting times

Answers to Exercises
Table I
Index

239
252
255

Probability Spaces

Probability theory is the branch of mathematics that is concerned with random

(or chance) phenomena. It has attracted people to its study both because of its
intrinsic interest and its successful applications to many areas within the physical,
biological, and social sciences, in engineering, and in the business world.
Many phenomena have the property that their repeated observation under a
specified set of conditions invariably leads to the same outcome. For example, if
a ball initially at rest is dropped from a height of d feet through an evacuated
cylinder, it will invariably fall to the ground in t = J2dfg seconds, where
g = 32 ft/sec2 is the constant acceleration due to gravity. There are other
phenomena whose repeated observation under a specified set of conditions does not
always lead to the same outcome. A familiar example of this type is the tossing of a
coin. If a coin is tossed 1000 times the occurrences of heads and tails alternate in a
seemingly erratic and unpredictable manner. It is such phenomena that we think
of as being random and which are the object of our investigation.
At first glance it might seem impossible to make any worthwhile statements
about such random phenomena, but this is not so. Experience has shown that
many nondeterministic phenomena exhibit a statistical regularity that makes them
subject to study. This may be illustrated by considering coin tossing again. For
any given toss of the coin we can make no nontrivial prediction, but observations
show that for a large number of tosses the proportion of heads seems to fluctuate
around some fixed number p between 0 and l (p being very near 1/2 unless the coin
is severely unbalanced). It appears as if the proportion of heads inn tosses would
converge top if we let n approach infinity. We think of this limiting proportion p
as the "probability" that the coin will land heads up in a single toss.
More generally the statement that a certain experimental outcome has probability
p can be interpreted as meaning that if the experiment is repeated a large number of
times, that outcome would be observed "about" 1OOp percent of the time. This
interpretation of probabilities is called the relative frequency interpretation. It is
very natural in many applications of probability theory to real world problems,
especially to those involving the physical sciences, but it often seems quite
artificial. How, for example, could we give a relative frequency interpretation to
1

P1obsbHity Spaces

the probability that a given newborn baby will live at least 70 years? Various
attempts have been made, none of them totally acceptable, to give alternative
interpretations to such probability statements.
For the mathematical theory of probability the interpretation of probabilities
is irrelevant, just as in geometry the interpretation of points, lines, and planes is
irrelevant. We will use the relative frequency interpretation of probabilities only as
an intuitive motivation for the definitions and theorems we will be developing
throughout the book.
1.1.

Examples of random phenomena

In this section we will discuss two simple examples of random phenomena in order to motivate the formal structure of the theory.
A box has s balls, labeled 1, 2, . .. , s but otherwise
identical. Consider the following experiment. The balls are mixed up well
in the box and a person reaches into the box and draws a ball. The
number of the ball is noted and the ball is returned to the box. The outcome of the experiment is the number on the ball selected. About this
experiment we can make no nontrivial prediction.
Example 1.

Suppose we repeat the above experiment n times. Let N,(k) denote the
number of times the ball labeled k was drawn during these n trials of the
experiment. Assume that we had, say, s = 3 balls and n = 20 trials.
The outcomes of these 20 trials could be described by listing the numbers
which appeared in the order they were observed. A typical result might be

l, 1,3, 2,

1,~~3,~3,3,~ 1,~3,3, 1,3,~~

in which case
and
The relative frequencies (i.e., proportion of times) of the outcomes l, 2,
and 3 are then
Nzo(l) = .25,
20

N 20(2) = .40,

20

and

N zo(3)

= .35.

20

As the number of trials gets large we would expect the relative frequencies N,(1)/n, ... , N,(s)/n to settle down to some fixed numbers
PH p 2 , , Ps (which according to our intuition in this case should all
be 1/s).
By the relative frequency interpretation, the number p 1 would be called
the probability that the ith ball will be drawn when the experiment is
performed once (i = 1, 2, .. . , s).

1. 1.

Examples of random phenomena

We will now make a mathematical model of the experiment of drawing

a ball from the box. To do this, we first take a set !l having s points that
we place into one-to-one correspondence with the possible outcomes of
the experiment. In this correspondence exactly one point of n will be
associated with the outcome that the ball labeled k is selected. Call that
point w". To the point w" we associate the number p" = 1/s and call it the
probability of w". We observe at once that 0 < p" < 1 and that
Pt

+ ' + Ps =

Suppose now that in addition to being numbered from 1 to s the first r

balls are colored red and the remaining s - r are colored black. We
perform the experiment as before, but now we are only interested in the
color of the ball drawn and not its number. A moment's thought shows
that the relative frequency of red balls drawn among n repetitions of the
experiment is just the sum of the relative frequencies N,.(k)/n over those
values of k that correspond to a red ball. We would expect, and experience shows, that for large n this relative frequency should settle down to
some fixed number. Since for large n the relative frequencies N11(k)/n are
expected to be close to p" = l js, we would anticipate that the relative
frequency of red balls would be close to rjs. Again experience verifies this.
According to the relative frequency interpretation, we would then call rjs
the probability of obtaining a red ball.
Let us see how we can reflect this fact in our model. Let A be the subset
of n consisting of those points w" such that ball k is red. Then A has
exactly r points. We call A an event. More generally, in this situation we
will call any subset B of!l an event. To say the event B occurs means that
the outcome of the experiment is represented by some point in B.
Let A and B be two events. Recall that the union of A and B, A u B,
is the set of all points wen such that we A or we B. Now the points in
n are in correspondence with the outcomes of our experiment. The event
A occurs if the experiment yields an outcome that is represented by some
point in A, and similarly the event B occurs if the outcome of the experiment is represented by some point in B. The set A u B then represents
the fact that the event A occurs or the event B occurs. Similarly the intersection A f1 B of A and B consists of all points that are in both A and B.
Thus if w e A f1 B then w e A and w e B so A n B represents the fact
that both the events A and B occur. The complement Ac (or A') of A is the
set ~f points in n that are not in A . The event A does not occur if the experiment yields an outcome represented by a point in Ac.
Diagrammatically, if A and Bare represented by the indicated regions
in Figure Ia, then A u B, A n B , and Ac are represented by the shaded
regions in Figures 1b, 1c, and 1d, respectively.

P1obsbility Spaces

1a

1b

r-------------------~0

AUB
1c

Figure 1

To illustrate these concepts let A be the event "red ball selected" and
let B be the event "even-numbered ball selected." Then the union A u B
is the event that either a red ball or an even-numbered ball was selected.
The intersection A n B is the event "red even-numbered ball selected."
The event Ac occurs if a red ball was not selected.
We now would like to assign probabilities to events. Mathematically,
this just means that we associate to each set B a real number. A priori we
could do this in an arbitrary way. However, we are restricted if we want
these probabilities to reflect the experiment we are trying to model. How
should we make this assignment? We have already assigned each point
the number s- 1 Thus a one-point set {m} should be assigned the number
s- 1 Now from our discussion of the relative frequency of the event
"drawing a red ball," it seems that we should assign the event A the probability P(A) = r/s. More generally, if B is any event we will define P(B)
by P(B) = j/s if B has exactly j points. We then observe that
P(B) =

l:

p1"

CDic41B

where LQ)k e B Pt means that we sum the numbers Pt over those values of k
such that m~: e B. From our definition of P(B) it easily follows that the
following statements are true. We leave their verification to the reader.
Let 0 denote the empty set; then P(0) = 0 and P(O) = I. If A and B
are any two disjoint sets, i.e., A n B == 0 , then
P(A u B)

= P(A) + P(B).

It is known from physical experiments that an isotope of a

certain substance is unstable. In the course of time it decays by the emission of neutrons to a stable form. We are interested in the time that it takes an atom of the isotope to decay to its stable form. According to the
Example 2.

1.1.

Examples of tandom phenomena

laws of physics it is impossible to say with certainty when a specified atom

of the isotope will decay, but if we observe a large number N of atoms
initially, then we can make some accurate predictions about the number of
atoms N(t) that have not decayed by time t. In other words we can rather
accurately predict the fraction of atoms N(t)/N that have not decayed by
time t, but we cannot say which of the atoms will have done so. Since all
of the atoms are identical, observing N atoms simultaneously should be
equivalent to N repetitions of the same experiment where, in this case,
the experiment consists in observing the time that it takes an atom to decay.
Now to a first approximation (which is actually quite accurate) the rate
at which the isotope decays at time t is proportional to the number of
atoms present at timet, so N(t) is given approximately as the solution of
the differential equation
df
dt = -A.f(t),
f(O) = N ,
where A. > 0 is a fixed constant of proportionality. The unique solution
of this equation is f( t) = Ne- .tr, and thus the fraction of atoms that have
not decayed by time t is given approximately by N(t)/N = e-u. If
0 ~ 10 ~ Itt the fraction of atoms that decay in the time interval [10 , It]
is (e-.tro - e-.tra). Consequently, in accordance with the relative frequency
interpretation of probability we take (e-.tto - e-.tr,) as the probability
that an atom decays between times t 0 and t 1
To make a mathematical model of this experiment we can try to proceed
as in the previous example. First we choose a set n that can be put into a
one-to-one correspondence with the possible outcomes of the experiment.
An outcome in this case is the time that an atom takes to decay. This can
be any positive real number, so we take n to be the interval [0, oo) on the
real line. From our discussion above it seems reasonable to assign to
the interval [t0 , tt] the probability (e-.t'o - e-.tra). In particular if
10 === It = t then the interval degenerates to the set {t} and the probability assigned to this set is 0.
In our previous example n had only finitely many points ; however, here
Q has a (noncountable) infinity of points and each point has probability 0.
Once again we observe that P(Q) = 1 and P(0) = 0. Suppose A and B
are two disjoint intervals. Then the proportion of atoms that decay in the
time interval A u B is the sum of the proportion that decay in the time
interval A and the proportion that decay in the time interval B. In light
of this additivity we demand that in the mathematical model, A u B
should have probability P(A) + P(B) assigned to it. In other words, in
the mathematical model we want
P(A u B)

P(A)

whenever A and B are two disjoint intervals.

P(B)

Probability Spaces

1.2.

Probability spaces

Our purpose in this section is to develop the formal mathematical

structure, called a probability space, that forms the foundation for the
mathematical treatment of random phenomena.
Envision some real or imaginary experiment that we are trying to model.
The first thing we must do is decide on the possible outcomes of the
experiment. It is not too serious if we admit more things into our consideration than can really occur, but we want to make sure that we do not
exclude things that might occur. Once we decide on the possible outcomes, we choose a set 0 whose points w are associated with these
outcomes. From the strictly mathematical point of view, however, !l is
just an abstract set of points.
We next take a nonempty collection d of subsets of n that is to
represent the collection of "events" to which we wish to assign probabilities. By definition, now, an event means a set A in .91. The statement
the event A occurs means that the outcome of our experiment is represented
by some point we A. Again, from the strictly mathematical point of view,
d is just a specified collection of subsets of the set n. Only sets A e d,
i.e., events, will be assigned probabilities. In our model in Example I,
.91 consisted of all subsets of !l. In the general situation when !l does not
have a finite number of points, as in Example 2, it may not be possible to
choose d in this manner.
The next question is~ what should the collection d be? It is quite
reasonable to demand that d be closed under finite unions and finite
intersections of sets in .91 as well as under complementation. For example,
if A and B are two events, then A u B occurs if the outcome of our
experiment is either represented by a point in A or a point in B. Clearly,
then, if it is going to be meaningful to talk about the probabilities that A
and B occur, it should also be meaningful to talk about the probability that
either A or B occurs, i.e., that the event A u B occurs. Since only sets in
d will be assigned probabilities, we should require that A u B e d whenever A and Bare members of d. Now A r. B occurs if the outcome of
our experiment is represented by some point that is in both A and B. A
similar line of reasoning to that used for A u B convinces us that we
should have A r. Bed whenever A, Be .91. Finally, to say that the
event A does not occur is to say that the outcome of our experiment is not
represented by a point in A, so that it must be represented by some point
in Ac. It would be the height of folly to say that we could talk about the
probability of A but not of Ac. Thus we shall demand that whenever A is
in .91 so is Ac.
We have thus arrived at the conclusion that d should be a nonempty
collection of subsets of 0 having the following properties:

1.2.

Probability spaces

(i) If A is in .91 so is Ac.

(ii) If A and B are in .91 so are A u B and A n B.
An easy induction argument shows that if Ah A 2 , , A 11 are sets in .91
then so are U7=t A, and (l7=t A,. Here we use the shorthand notation
II

U A1 =

At u A 2 u u A,.

I= t

and
II

() A 1 = At n A 2 n n A11

I= t

Also, since A n Ac = 0 and A u Ac = n, we see that both the empty

set 0 and the set n must be in d.
A nonempty collection of subsets of a given set n that is closed under
finite set theoretic operations is called a field of subsets of n. It therefore
seems we should demand that .91 be a field of subsets. It turns out, however, that for certain mathematical reasons just taking .91 to be a field of
subsets of n is insufficient. What we will actually demand of the collection
.91 is more stringent. We will demand that .91 be closed not only under
finite set theoretic operations but under countably infinite set theoretic
operations as well. In other words if {A11}, n > I, is a sequence of sets in
.91, we will demand that
00

U A,. E J1
ll=t

00

and

() A,. e Jl.
ll=t

Here we are using the shorthand notation

00

U All = A

~t=t

u A2 u

to denote the union of all the sets of the sequence, and

00

() A,. = At n A 2 n

11=t

to denote the intersection of all the sets of the sequence. A collection of

subsets of a given set n that is closed under countable set theory operations
is called a a-field of subsets of n. (The u is put in to distinguish such a
collection from a field of subsets.) More formally we have the following:
_Definition 1 A nonempty collection of subsets .91 of a set n is
called au-field ofsubsets ofO provided that the following two properties
hold:

(i) If A is in .91, then Ac is also in .91.

(ii) If A 11 is in .91, n = 1, 2, . .. , then U~ 1 A 11 and
both in d.

():'=t

A 11 are

Probability Spaces

We now come to the assignment of probabilities to events. As was

made clear in the examples of the preceding section, the probability of an
event is a nonnegative real number. For an event A, let P(A) denote this
number. Then 0 s; P(A) < 1. The set n representing every possible
outcome should, of course, be assigned the number 1, so P(O) = 1. In our
discussion of Example 1 we showed that the probability of events satisfies
the property that if A and Bare any two disjoint events then P(A u B) =
P(A) + P(B). Similarly, in Example 2 we showed that if A and B are two
disjoint intervals, then we should also require that
P(A u B)

P(A)

+ P(B).

It now seems reasonable in general to demand that if A and B are

disjoint eveQts then P(A u B) = P(A) + P(B). By induction, it would
then follow that if A 1 , A 2 , , A,. are any n mutually disjoint sets (that is,
if A, f1 A1 = 0 whenever i 1: j), then
p

(Qt

Ai) =

itt

P(A,).

Actually, again for mathematical reasons, we will in fact demand that this
additivity property hold for countable collections of disjoint events.
Definition 2 A probability measure P on a a-field of subsets .91
of a set n is a real-valued function having domain .91 satisfying the
following properties:
(i) P(Q) = 1.
(ii) P(A) > 0 for all A e d .
(iii) If A,., n = 1, 2, 3, .. . , are mutually disjoint sets in .91, then
p

(,.Qt A,.) = ,.tt P(A,.).

A probability space, denoted by (0, .91, P), is a set n, a a-field of

subsets .91, and a probability measure P defined on d.

It is quite easy to find a probability space that corresponds to the

experiment of drawing a ball from a box. In essence it was already given
in our discussion of that experiment. We simply taken to be a finite set
having s points, d to be the collection of all subsets of n, and p to be the
probability measure that assigns to A the probability P(A) = jfs if AI has
exactly j points.
Let us now consider the probability space associated with the isotope
disintegration experiment (Example 2). Here it is certainly clear that
n = [0, oo), but it is not obvious what .91 and P should be. Indeed, as we
will indicate below, this is by no means a trivial problem, and one that in
all its ramifications depends on some deep properties of set theory that are
beyond the scope of this book.

1.2.

Probability spaces

One thing however is clear; whatever .91 and P are chosen to be, d must
contain all intervals, and P must assign probability (e-lto - e-At,) to the
interval [t0 , t 1 ] if we want the probability space we are constructing to
reflect the physical situation. The problem of constructing the space now
becomes the following purely mathematical one. Is there a a-field .91 that
contains all intervals as members and a probability measure P defined on
d that assigns the desired probability P(A) to the interval A? Questions
of this type are in the province of a branch of advanced mathematics
called measure theory and cannot be dealt with at the level of this book.
Results from measure theory show that the answer to this particular
question and others of a similar nature is yes, so that such constructions
are always possible.
We will not dwell on the construction of probability spaces in general.
The mathematical theory of probability begins with a~ abstract probability
space and develops the theory using the probability space as a basis of
operation. Aside from forming a foundation for precisely defining other
concepts in the theory, the probability space itself plays very little role in
the further development of the subject. Auxiliary quantities (especially
random variables, a concept taken up in Chapter 3) quickly become the
dominant theme of the theory and the probability space itself fades into
the background.
We will conclude our discussion of probability spaces by constructing
an important class of probability spaces, called uniform probability spaces.
Some of the oldest problems in probability involve the idea of picking a
point "at random" from a set S. Our intuitive ideas on this notion show
us that if A and Bare two subsets having the same "size" then the chance
of picking a point from A should be the same as from B. If S has only
finitely many points we can measure the "size" of a set by its cardinality.
Two sets are then of the same "size" if they have the same number of
points. It is quite easy to make a probability space corresponding to the
experiment of picking a point at random from a set S having a finite
numbers of points. We taken = Sand d to be all subsets of S, and
assign to the set A the probability P(A) = jfs if A is a set having exactly j
points. Such a probability space is called a symmetric probability space
because each one-point set carries the same probability s - 1 . We shall
return to the study of such spaces in Chapter 2.
Suppose now that S is the interval [a, b] on the real line where ~ oo <
a < b < + oo. It seems reasonable in this case to measure the "size" of a
subset A of [a, b] by its length. Two sets are then of the same size if they
have the same length. We will denote the length of a set A by lA I.
To construct a probability space for the experiment of "choosing a
point at random from s,~ we proceed in a manner similar to that used for
= S, and appeal to the results of
the isotope experiment. We take

ProbsbHity Spaces

10

measure theory that show that there is a a-field .91 of subsets of S, and a
probability measure P defined on .91 such that P(A) = IAI/ISI whenever A
is an interval.
More generally, let S be any subset of r-dimensional Euclidean space
having finite, nonzero r-dimensional volume. For a subset A of S denote
the volume of A by IAJ. There is then a a-field .91 of subsets of S that
contains all the subsets of S that have volume assigned to them as in
calculus, and a probability measure P defined on .91 such that P(A) =
IAI/ISI for any such set A . We will call any such probability space,
denoted by (S, .91, P), a uniform probability space.
1.3.

Properties of probabilities

In this section we will derive some additional properties of a probability

measure P that follow from the definition of a probability measure. These
properties will be used constantly throughout the remainder of the book.
We assume that we are given some probability space (Q, .91, P) and that
all sets under discussion are events, i.e., members of .91.
For any set A, A u Ac = n and thus for any two sets A and B we have
the decomposition of B:
(I)

!l n B =(Au

A~

n B = (A n B) u (Ac n B).

Since A n B and Ac n B are disjoint, we see from (iii) of Definition 2

that

By setting B

+ P(Ac n B).
and recalling that P(!l) = 1, we conclude from (2) that
P(A~ = 1 - P(A).
P(B) = P(A n B)

(2)

=n

(3)

In particular P(0) = 1 - P(!l), so that

P(0) = 0.

(4)

As a second application of (2) suppose that A c B. Then A n B :::;; A

and hence
(5)

P(B) = P(A)

Since P(Ac n B)
(6)

P(Ac n B)

0 by (ii}, we see from (5) that

P(B)

P(A)

De Morgan's laws state that if {A11} , n

(7)

if A c B.

if A c B.
~

1, is any sequence of sets, then

1.3.

11

Properties of prQbabilities

and
(8)

To see that (7) holds, observe that w e (U,. ~ 1 A,.)c if and only if m A,.
for any n; that is, we A! for all n ;;;:: 1, or equivalently, wE
A~. To
establish (8) we apply (7) to {A~}, obtaining

n,.

and by taking complements, we see that

A useful relation that follows from (7) and (3) is

(9)

Now

n,.

U,. A,. is the event that at least one of the events A,. occurs, while

A~

is the event that none of these events occur. In words, (9) asserts
that the probability that at least one of the events A 11 will occur is 1 minus
the probability that none of the events A,. will occur. The advantage of (9)
is that in some instances it is easier to compute P(n,. A~ than to compute
P(U,. A 11) . [Note that since the events A,. are not necessarily disjoint it is
not true that P(U,. A 11) = L11 P(A,.).] The use of (9) is nicely illustrated
by means of the following.
Suppose three perfectly balanced and identical coins are
tossed. Find the probability that at least one of them lands heads.
Example 3.

There are eight possible outcomes of this experiment as follows :

Coin 1

Coin 2

Coin 3

T
-

Our intuitive notions suggest that each of these eight outcomes should
have the probability 1/8. Let A 1 be the event that the first coin lands
heads, A 2 the event that the second coin lands heads, and A 3 the event
that the third coin lands heads. The problem asks us to compute
P(A 1 u A 2 u A3 ) . Now Af n A2 n Aj = {T, T, T} and thus
P(Af n

A2 n

A3) = 1/8;

12

Prob11bility $p11css

hence (9) implies that

P(A 1 u A 2 u A 3 ) = 1 - P(Af n A~ n A3) = 7/8.

Our basic postulate (iii) on probability measures tells us that for disjoint sets A and B, P(A u B) = P(A) + P(B). If A and B are not
necessarily disjoint, then
P(A u B) = P(A)

(10)

+ P(B)

- P(A n B)

and consequently
P(A u B) ~ P(A.)

(11)

+ P(B).

To see that (10) is true observe that the sets A n Be, A. n B, and Ae n B
are mutually disjoint and their union is just A u B (see Figure 2). Thus
(12)

P(A u B) = P(A n Be)

+ P(Ae n

B)

+ P(A n

B).

By (2), however,
P(A n B1

P(A) - P(A n B)

P(Ac n B)

P(B) - P(A n B).

and
By substituting these expressions into (12), we obtain (10).

Figure 2

Equations (10) and (11) extend to any finite number of sets. The
analogue of the exact formula (10) is a bit complicated and will be discussed in Chapter 2. Inequality (11), however, can easily be extended by
induction to yield
(13)

P(A. u A2 u ... u A,.)

s L"

P(A,).

I= 1

To prove this, observe that if n > 2, then by (11)

P(A 1 u u A,.)

.P((A 1 u u A.- 1 ) u A,.)

S P(A 1 u u A.- 1)

+ P(A,.).

Hence if (13) holds for n - 1 sets, it holds for n sets. Since (13) clearly
holds for n = 1, the result is proved by induction.

1.3.

13

P1operties of p1obsbilities

So far we have used only the fact that a probability measure is finitely
additive. Our next result will use the countable additivity.

Theorem 1 Let A,., n > 1, be events.

(14)

U:=t A,., then

(i) If A 1 c: A 2 c: and A

= P(A).

lim P(A,.)
n-+co

n::.

{ii) If A 1 => A 2 => and A =

A,, then (14) again holds.

Proof of (i). Suppose A 1 c A 2 c and A = U;. 1 A,.. Set

B1 = A1, and for each n ~ 2, let B11 denote those points which are in A,.
but not in A,._ 1 , i.e., B,. = A,. n A~- 1 A point ro is in B,. if and only if
ro is in A and A,. is the first set in the sequence A 1 , A 2 , containing w.
By definition, the sets B,. are disjoint,
II

UB

A,. =

1,

I= 1

and
co

U B1
I= 1

A=
Consequently,

II

P(A,.) = ~ P(B1)
I= 1

and
P(A)

co

~ P(B1).
i= 1

Now
CO

II

(15)

lim ~ P(Bc) = ~ P(B1)

n-+co I= 1

l= 1

by the definition of the sum of an infinite series. It follows from (15) that
lim P(A,.)
n-+ co

II

lim ~ P(B1)

n-+co 1=1
CIO

L P(BI) = P(A),

I= 1

so that (14) holds.

Proof of (ii). Suppose A 1 => A 2 => and A =

A f c: A 2 c and by (8)
co

A'"=

U A~.
n=1

Thus by (i) of the theorem

(16)

lim
n-+co

P(A~

= P(AC).

n:=

A,.. Then

14

Ptobsbility Spaces

Since P(A~ = 1 - P(A,.) and P(Ac) = 1 - P(A), it follows from (16)

that
lim P(A,.) = lim (1 - P(A~)
,. .... 00

..... 00

1 - lim P(A~)
n-+oo

1 -

P(A~ =

P(A),

and again (14) holds.

1.4.

Conditional probability

Suppose a box contains r red balls labeled 1, 2, . . . , r and b black balls

labeled 1, 2, . .. , b. Assume that the probability of drawing any particular
ball is (b + r)- 1 If the ball drawn from the box is known to be red, what
is the probability that it was the red ball labeled I? Another way of
stating this problem is as follows. Let A be the event that the selected
ball was red, and let B be the event that the selected ball was labeled 1.
The problem is then to determine the "conditional" probability that the
event B occurred, given that the event A occurred. This problem cannot
be solved until a precise definition of the conditional probability of one
event given another is available. This definition is as follows :
Definition 3 Let A and B be two events such that P(A) > 0.
Then the conditional probability of B given A, written P(B I A), is
defined to be
P(B I A) = P(B n A) .

(17)
If P(A)

P(A)

0 the conditional probability of B given A is undefined.

The above definition is quite easy to motivate by the relative frequency

interpretation of probabilities. Consider an experiment that is repeated
a large number of times. Let the number of times the events A, B, and
A fl B occur in n trials of the experiment be denoted by N,.(A), N 11(B), and
N,.(A n B), respectively. For n large we expect that N,.(A)/n, N,.(B)fn,
and N,.(A n B)/n should be close to P(A), P(B), and P(A n B), respectively. If now we just record those experiments in which A occurs then we
have N,.(A) trials in which the event B occurs N,.(A n B) times. Thus the
proportion of times that B occurs among these N 11(A) experiments is
N 11(A rl B)/N,.(A). But
NiA

rl

N"(if)

B)

N,(A

rl

B)/n

N"(il)jn

1.4.

Conditions/ p1obsbillty

16

and thus for large values of n this fraction should be close to

P(A

r. B)/P(A).

As a first example of the use of (17) we will solve the problem posed at
the start of this section. Since the set Q has b + r points each of which
carries the probability (b + r)- 1 , we see that P(A) = r(b + r)- 1 and
P(A n B) = (b + r)- 1 Thus
I
P(B I A)=-.
r

This should be compared with the "unconditional" probability of B,

namely P(B) = 2(b + r)- 1
Example 4.

Suppose two identical and perfectly balanced coins are

tossed once.
(a) Find the conditional probability that both coins show a head given
that the first shows a head.
(b) Find the conditional probability that both are heads given that at
least one of them is a head.
To solve these problems we let the probability space n consist of the
four points HH, HT, TH, IT, each carrying probability 1/4. Let A be
the event that the first coin results in heads and let B be the event that the
second coin results in heads. To solve (a) we compute
P(A n B

I A) =

P(A n B)/P(A) = (1/4)/(1/2)

= 1/2.

To answer (b) we compute

P(A n B

I A u B) =

P(A n B)/P(A u B)

= (1/4)/(3/4) =

1/3.

In the above two examples the probability space was specified, and we
used (17) to compute various conditional probabilities. In many problems
however, we actually proceed in the opposite direction. We are given in
advance what we want some conditional probabilities to be, and we use this
information to compute the probabiiity measure on n. A typical example
of this situation is the following.
Suppose that the population of a certain city is 40% male
and 60% female. Suppose also that 50% of the males and 30% of the
females smoke. Find the probability that a smoker is male.
Example 6.

Let M denote the event that a person selected is a male and let F denote
the event that the person selected is a female. Also let S denote the event
that the person selected smokes and let N denote the event that he does not
smoke. The given information can be expressed in the form P(S I M) = .5,

16

Probability Spaces

P(S I F) = .3, P(M)

P(M J S). By (17),

.4, and P(F)

= .6.

The problem is to compute

P(M I S) = P(M n .S) .

P(S)

Now P(M n S) = P(M)P(S I M) = (.4)(.5) = .20, so the numerator can

be computed in terms of the given probabilities. Since Sis the union of the
two disjoint sets S n M and S n F, it follows that
P(S) = P(S n M)

+ P(S n F).

Since
P(S n F) = P(F)P(S I F) = (.6)(.3) = .18,

we see that
P(S)

.20

+ .18 = .38.

Thus
20
= .38
~

P(M I S)

.53 .

The reader will notice that the probability space, as such, was never
explicitly mentioned. This problem and others of a similar type are solved
simply by using the given data and the rules of computing probabilities
given in Section 3 to compute the requested probabilities.
It is quite easy to construct a probability space for the above example.
Take the set Q to consist of the four points SM, SF, NM, and NF that are,
respectively, the unique points in the sets S n M , S r'l. F, N n M , and
N n F. The probabilities attached to these four points are not directly
specified, but are to be computed so that the events P(S I M), P(S I F),
P(M), and P(F) have the prescribed probabilities. We have already
found that P(S n M) = .20 and P(S n F) = .18. We leave it as an
exercise to compute the probabilities attached to the other t~o points.
The problem discussed in this example is a special case of the following
general situation. Suppose A1, A 2 , , A,. are n mutually disjoint events
with union n. Let B be an event such that P(B) > 0 and suppose P(B I At)
and P(At) are specified for 1 ~ k ~ n. What is P(A 1 I B)? To solve this
problem note that the A 11 are disjoint sets with union 0 and consequently

Thus

P(B) =

A:=l

But

P(B n Aa)

1.4.

Conditional probability

17

so we can write
(18)

P(A, I B) = P(A, (""\ B) =

P(Al)P(B I A,)
P(B)
L~= 1 P(AJP(B I A")

This formula, called Bayes' rule, finds frequent application. One way
oflooking at the result in (18) is as follows. Suppose we think of the events
At as being the possible "causes" of the observable event B. Then P(A 1 I B)
is the probability that the event A, was the "cause" of B given that B
occurs. Bayes' rule also forms the basis of a statistical method called
Bayesian procedures that will be discussed in Volume II, Introduction to
Statistical Theory.
As an illustration of the use of Bayes' rule we consider the following
(somewhat classical) problem.
Exa~ple 6.

Suppose there are three chests each having two drawers.

The first chest has a gold coin in each drawer, the second chest has a gold
coin in one drawer and a .silver coin in the other drawer, and the third
chest has a silver coin in each drawer. A chest is chosen at random and a
drawer opened. If the drawer contains a gold coin, what is the probability
that the other drawer also contains a gold coin? We ask the reader to
pause and guess what the answer is before reading the solution. Often in
this problem the erroneous answer of 1/2 is given.
This problem is easily and correctly solved using Bayes' rule once the
description is deciphered. We can think of a probability space being
constructed in which the events A 1 , A 2 , and A 3 correspond, respectively,
to the first, second, and third chest being selected. These events are disjoint and their union is the whole space n since exactly one chest is selected.
Moreover, it is presumably intended that the three chests are equally
likely of being chosen so that P(A 1) = 1/3, i = 1, 2, 3. Let B be the event
tha~ the coin observed was gold. Then, from the composition of the chests
it is clear that
and
The problem asks for the probability that the second drawer has a goJd
coin given that there was a gold coin in the first. This can only happen if
the chest selected was the first, so the problem is equivalent to finding
P(A 1 I B). We now can apply Bayes' rule (18) to compute the answer,
which is 2/3. We leave it to the reader as an exercise to compute the
probability that the second drawer has a silver coin given that the first
drawer had a gold coin.
For our next example we consider a simple probability scheme due to
Polya.

18

Probability Spsces

Polya's urn scheme. Suppose an urn has r red balls and

Example 7.

b black balls. A ball is drawn and its color noted. Then it together
with c > 0 balls of the same color as the drawn ball are added to the urn.
The procedure is repeated n - 1 additional times so that the total
number of drawings made from the urn is n.

Let R1, 1 ~ j < n, denote the event that thejth ball drawn is red and let
B1 , 1 ~ j < n, denote the event that the jth ball drawn is black. Of course,
for each j, R1 and B1 are disjoint. At the kth draw there are b + r +
(k - 1)c balls in the urn and we assume that the probability of drawing
any particular ball is (b + r + (k - 1)c)- 1 To compute P(R 1 (') R 2 ) we
write
Now

and thus
P(R. (') R2) = (

r ) ( r+c ) .
b+r b+r+ c

Similarly

and thus
P(R 2 ) = P(R 1 n R 2 )

P(B 1 r. R 2 )

--b + r
Consequently, P(R2 )

= P(R1). Since
P(B2) = 1 - P(R 2) =

,
r

P(B2 ) = P(B1) . Further properties of the Polya scheme will be developed

in the exercises.
1.5.

Independence

Consider a box having four distinct balls and an experiment consisting

of selecting a ball from the box. We assume that the balls are equally
likely to be drawn. Let !l = {1, 2, 3, 4} and assign probability 1/4 to
each point.

1.5. Independence

19

Let A and B be two events. For some choices of A and B, knowledge

that A occurs increases the odds that B occurs. For example, if A = {I, 2}
and B = {1}, then P(A) = 1/2, P(B) = 1/4, and P(A n B) = 1/4. Consequently, P(B I A) = 1/2, which is greater than P(B). On the other hand,
for other choices of A and B, knowledge that A occurs decreases the odds
that B will occur. For example, if A = {1, 2, 3} and B = {1, 2, 4}, then
P(A) = 3/4, P(B) = 3/4, and P(A n B) = 1/2. Hence P(BIA) = 2/3,
which is less than P(B).
A very interesting case occurs when knowledge that A occurs does not
change the odds that B occurs. As an example of this let A = {1, 2} and
B = {1, 3}; then P(A) = 1/2, P(B) = 1/2, P(A n B) = 1/4, and therefore P(B I A) = 1/2. Events such as these, for which the conditional
probability is the same as the unconditional probability, are said to be
independent.
Let A and B now be any two events in a general probability space, and
suppose that P(A) =F 0. We can then define A and B to be independent if
P(B I A) = P(B). Since P(B I A) = P(B n A)/P(A) we see that if A and
Bare independent then
(19)

P(A n B)

= P(A)P(B).
= 0 and is also symmetric in the letters

Since (19) makes sense even if P(A)

A and B, it leads to a preferred definition of independence.

Definition 4 Two events A and B are independent if and only if

P(A n B)

P(A)P(B).

We can consider a similar problem for three sets A, B, and C. Take

Q = {1, 2, 3, 4} and assign probability 1/4 to each point. Let A = {I, 2},
B = {1, 3}, and C = {1, 4}. Then we leave it as an exercise to show that
the pairs of events A and B, A and C, and B and Care independent. We
say that the events A, B, and Care pairwise independent. On the other
hand, P(C) = 1/2 and
P(C I A n B) = 1.
Thus a knowledge that the event A n B occurs increases the odds that C
occurs. In this sense the events A, B, and C fail to be mutually independent.
In general, three events A, B, and C are mutually independent if they are
pairwise independent and if

P(A n B n C)

P(A)P(B)P(C).

We leave it as an exercise to show that if A, B, and Care mutually independent and P(A n B) =F 0, then P(C I A n B) = P(C).

P1obabUhy Spaces

20

More generally we define n ~ 3 events A 1 , A 2 , , A,. to be mutually

independent if
P(A 1 n n A") = P(A 1 ) P(A")
and if any subcollection containing at least two but fewer than n events are
mutually independent.
LetS be the square 0 ~ x ~ 1, 0 ~ y ~ 1 in the plane.
Consider the uniform probability space on the square, and let A be the
event
{(x, y): 0 ~ x ~ 1/2, 0 s y ~ 1}
Example 8.

and B be the event

{(x, y): 0 < x

1, 0 < y < 1/4}.

Show that A and B are independent events.

To show this, we compute P(A), P(B), and P(A n B), and show that
P(A n B) = P(A}P(B). Now A is a subrectangle of tbe square S having
area 1/2 and B is a subrectangle of the square S having area 1/4, so
P(A) = 1/2 and P(B) = 1/4. Since
A n B = {(x, y): 0 < x

1/2, 0

1/4}

is a subrectangle of the square S having area 1/8, P(A n B) = 1/8 and

we see that A and B are independent events as was asserted.
The notion of independence is frequently used to construct probability
spaces corresponding to repetitions of the same experiment. This matter
will be dealt with more fully in Chapter 3. We will be content here to
examine the simplest situation, namely, experiments (such as tossing a
possibly biased coin) that can result in only one of two possible outcomes-success or failure.
In an experiment such as tossing a coin n times, where success and
failure at each toss occur with probabilities p and 1 - p respectively, we
intuitively believe that the outcome of the ith toss should have no influence
on the outcome of the other tosses. We now wish to construct a probability
space corresponding to the compound experiment of an n-fold repetition
of our simple given experiment that incorporates our intuitive beliefs.
Since each of the n trials can result in either success or failure, there is a
total of 2" possible outcomes to the compound experiment. These may be
represented by ann-tuple (xh ... , x")' where x 1 = 1 or 0 according as the
ith trial yields a success or failure. We take the set 0 to be the collection
of all such n-tuples. The u-fieldl JJI is taken to be all subsets of n.
We now come to the assignment of a probability measure. To do
this it is only necessary to assign probabilities to the 2" one-point sets
{(x1, .. , x.)}. Suppose then-tuple (xh ... , x,.) is such that exactly k of

1.5.

21

Independence

the x/s have the value I; for simplicity, say x 1 = x 2 = ; = X~c = I and
the other x,'s have the value 0. Then if A1 denotes the event that the ith
trial, I < i < n, is a success, we see that
{(1, 1, ... , 1, 0, .. . , 0)} = A 1 n n A~c n A~+ 1 n n A~.

n- k

According to our intuitive views, the events A1 , .. , A~c, Ak+ ~> .. . , A~ are
to be mutually independent and P(A 1) = p, 1 < i < n. Thus we should
assign P so that
P({(1, 1, .. . , 1, 0, ... , 0)}) = P(A 1 )

P(A~c)P(Ak+ 1)

P(A~)

p"'(1 - p)"- k.

By the same reasoning, we see that if then-tuple (x 1 , , x,.) is such that

exactly k of the x,'s have the value 1, then P should be such that
P({(xt> ... , x,.)}) = pk(I - p)"-k.

Let us now compute the probability that exactly k of then trials result
in a success. Note carefully that this differs from the probability that k
specified trials result in successes and the other n - k trials result in
failures. Let B" denote the event that exactly k of the n trials are successes.
Since every choice of a specified sequence having k successes has probability
pk(l - p)"-k, the event Brc has probability P(B~c) = C(k, n)pk(l - p)"-k,
where C(k, n) is the number of sequences (x 1 , .. , x,.) in which exactly k
of the x,'s have value 1. The computation of C(k, n) is a simple combinatorial problem that will be solved in Section 2.4. There it will be
shown that
(20)

C(k n)

'

n!
k!(n- k)! '

0 < k < n.

Recall that 0! - 1 and that, for any positive integer m,

m!

= m(m

1) 1.

The quantity n!/k!(n - k)! is usually written as

coefficient). Thus

(Z)

(the binomial

(21)

Various applied problems are modeled by independent success-failure

trials. Typical is the following.
Example 9. Suppose a machine produces bolts, 100/o of which are
defective. Find the probability that a box of 3 bolts contains at most one
defective bolt.

ProbsbHity Spaces

22

To solve the problem we assume that the production of bolts constitutes

repeated independent success-failure trials with a defective bolt being a
success. The probability of success in this case is then .1. Let B0 be the
event that none of the three bolts are defective and let B1 be the event that
exactly one of the three bolts is defective. Then B 0 u B 1 is the event that
at most one bolt is defective. Since the events B 0 and B 1 are clearly
disjoint, it follows that
P(B0 u B 1) = P(B0 )
=

P(B 1)

(~) (.1)0(.9)3 +

(n

(.1)1(.9)2

= (.9) 3 + 3(.1)(.9)2
= .972.

Exercises

Let (0, .!1/, P) be a probability space, where .r;1 is the a-field of all
subsets of n and P is a probability measure that assigns probability
p > 0 to each one-point set of 0.
(a) Show that 0 must have a finite number of points. Hint: show that
n can have no more than p- 1 points.
(b) Show that if n is the number of points in 0 then p must be n- 1
2 A model for a random spinner can be made by taking a uniform
probability space on the circumference of a circle of radius 1, so that the
probability that the pointer of the spinner lands in an arc of length s is
sf2n. Suppose the circle is divided into 37 zones numbered I, 2, ... , 37.
Compute the probability that the spinner stops in an even zone.

3 Let a point be picked at random in the unit square. Compute the

probability that it is in the triangle bounded by x = 0, y = 0, and

x+y= 1.
4 Let a point be picked at random in the disk of radius 1. Find the
probability that it lies in the angular sector from 0 to n/4 radians.
5 In Example 2 compute the following probabilities:
(a) No disintegration occurs before time 10.
(b) There is a disintegration before time 2 or a disintegration between
times 3 and 5.
6 A box contains 10 balls, numbered 1 through 10. A ball is drawn from
the box at random. Compute the probability that the number on the
ball was either 3, 4, or 5.
7 Suppose two dice are rolled once and that the 36 possible outcomes are
equally likely. Find the probability that the sum of the numbers on the
two faces is even.

23

Exercises

8 Suppose events A and B are such that P(A)

P(A u B) = 1/2. Find P(A n B).

2/5, P(B)

= 2/5,

and

H P(A) = 1/3, P(A u B) = 1/2, and P(A n B) = 1/4, find P(B).

1o Suppose a point is picked at random in the unit square. Let A be the
event that it is in the triangle bounded by the lines y = 0, x = I, and
x = y, and B be the event that it is in the rectangle with vertices
(0, 0), (1, 0), (1, 1/2), (0, 1/2). Compute P(A u B) and P(A n B).
11 A box has 10 balls numbered 1, 2, . . . , 10. A ball is picked at random
and then a second ball is picked at random from the remaining nine
balls. Find the probability that the numbers on the two selected balls
differ by two or more.
12 If a point selected at random in the unit square is known to be in the
triangle bounded by x = 0, y = 0, and x + y = 1,. find the probability that it is also in the triangle bounded by y = 0, x = I, and
X= y.
13 Suppose we have four chests each having two drawers. Chests I and 2
have a gold coin in one drawer and a silver coin in the other drawer.
Chest 3 has two gold coins and chest 4 has two silver coins. A chest is
selected at random and-a drawer opened. It is found to contain a gold
coin. Find the probability that the other drawer has
(a) a silver coin;
(b) a gold coin.
14 A box has 10 balls, 6 of which are black and 4 of which are white.
Three balls are removed from the box, their color unnoted. Find the
probability that a fourth ball removed from the box is white. Assume
,that the 10 balls are equally likely to be drawn from the box.
9

15 With the same box composition as in Exercise 14, find the probability

that all three of the removed balls will be black if it is known that at
least one of the removed balls is black.
16 Suppose a factory has two machines A and B that make 60% and 40%
of the total production, respectively. Of their output, machine A
produces 3% defective items, while machine B produces 5% defective
items. Find the probability that a given defective part was produced by
machine B.
17 Show by induction on n that the probability of selecting a red ball at
any trial n in Polya's scheme (Example 7) is r(b + r)- 1
18 A student is taking a multiple choice exam in which each question has
5 possible answers, exactly one of which is correct. H the student knows
the answer he selects the correct answer. Otherwise he selects one
answer at random from the 5 possible answers. Suppose that the
student knows the answer to 70% of the questions.
(a) What is the probability that on a given question the student gets
the correct answer?

24

Probebflity Spaces

(b) If the student gets the correct answer to a question, what is the
probability that he knows the answer?
19 Suppose a point is picked at random in the unit square. If it is known
that the point is in the rectangle bounded by y = 0, y = 1, x = 0, and

20

21

22

23

24
25

26

x = 1/2~ what is the probability that the point is in the triangle

bounded by y = 0, x = 1/2, and x + y = 1?
Suppose a box has r red and b black balls. A ball is chosen at random
from the box and then a second ball is drawn at random from the
remaining balls in the box. Find the probability that
(a) both balls are red;
(b) the first ball is red and the second is black;
(c) the first ball is black and the second is red ;
(d) both balls are black.
A box has 10 red balls and 5 black balls. A ball is selected from the
box. If the ball is red, it is returned to the box. If the ball is black, it
and 2 additional black balls are added to the box. Find the probability
that a second ball selected from the box is
(a) red ;
(b) black.
Two balls are drawn, with replacement of the first drawn ball, from a
box containing 3 white and 2 black balls.
(a) Construct a sample space for this experiment with equally likely
sample points.
(b) Calculate the probability that both balls drawn will be the same
color.
(c) Calculate the probability that at least one of the balls drawn will be
white.
Work Exercise 22 if the first ball is not replaced.
Work Exercise 22 by constructing a sample space based on 4 sample
points corresponding to white and black for each drawing.
Box I contains 2 white balls and 2 black balls, box II contains 2 white
balls and 1 black ball, and box III contains 1 white ball and 3 black
balls.
(a) One ball is selected from ,e ach box. Calculate the probability of
getting all white balls.
(b) One box is selected at random and one ball drawn from it. Calculate the probability that it will be white.
(c) In {b), calculate the probability that the first box was selected
given that a white ball is drawn.
A box contains 3 white balls and 2 black balls. Two balls are drawn
from it without replacement.
(a) Calculate the probability that the second ball is black given that the
first ball is black.
(b) Calculate the probability that the second ball is the same color as
the first ball.

25

Exercises

(c) Calculate the probability that the first ball is white given that the
second ball is white.

It is known that
40% of the men and 60% of the women smoke cigarettes. What is the
probability that a student observed smoking a cigarette is a man?

27 A college is composed of 70% men and 30% women.

28 Assume that cars are equally likely to be manufactured on Monday,

Tuesday, Wednesday, Thursday, or Friday. Cars made on Monday

have a 4% chance of being "lemons"; cars made on Tuesday,
Wednesday or Thursday have a 1% chance of being lemons; and
cars made on Friday have a 2% chance of being lemons. If you
bought a car and it turned out to be a lemon, what is the probability it was manufactured on Monday?
29 Suppose there were a test for cancer with the property that 90% of

those ' with cancer reacted positively whereas 5% of those without

cancer react positively. Assume that I% of the patients in a hospital
have cancer. What is the probability that a patient selected at random
who reacts positively to this test actually has cancer?
30 In the three chests problem discussed in Example 6, compute the

probability that the second drawer has a silver coin given that the
first drawer has a gold coin.
31 In Polya's urn scheme (Example 7) given that the second ball was red,

find the probability that

(a) the first ball was red;
(b) the first ball was black.
32 Suppose three identical and perfectly balanced coins are tossed once.

33

34
35

36

37

38

Let A 1 be the event that the ith coin lands heads. Show that the events
A 1, A 2 , and A 3 are mutually independent.
Suppose the six faces of a die are equally likely to occur and that the
successive die rolls are independent. Construct a probability space for
the compound experiment of rolling the die three times.
Let A and B denote two independent events. Prove that A and Be,
Ac and B, and Ac and Be are also independent.
Let !l = {1, 2, 3, 4} and assume each point has probability 1/4. Set
A = {1, 2}, B = {1, 3}, C = {1, 4}. Show that the pairs of events
A and B, A and C, and B and C are independent.
Suppose A, B, and Care mutually independent events and P(A n B) :/:
0. Show that P(C I A n B) = P(C).
Experience shows that 20% of the people reserving tables at a certain
restaurant never show up. If the restaurant has 50 tables and takes
52 reservations, what is the probability that it will be able to accommodate everyone?
A circular target of unit radius is divided into four annular zones with
outer radii 1/4, 1j2, 3/4, and 1, respectively. Suppose 10 shots are fired
independently and at random into the target.

26

Probability Spaces

(a) Compute the probability that at most three shots land in the zone
bounded by the circles of radius 1/2 and radius 1.
(b) If 5 shots land inside the disk of radius 1/2, find the probability
that at least one is in the disk of radius 1/4.
39 A machine consists of 4 components linked in parallel, so that the

40

41
42

43

44

45

46

47

machine fails only if all four components fail. Assume component

failures are independent of each other. If the components have
probabilities .1, .2, .3, and .4 of failing w~en the machine is turned on,
what is the probability that the machine will function when turned on?
A certain component in a rocket engine fails 5% of the time when the
engine is fired. To achieve greater reliability in the engine working,
this component is duplicated n times. The engine then fails only if all
of these n components fail. Assume the component failures are
independent of each other. What is the smallest value of n that can be
used to guarantee that the engine works 99% of the time?
A symmetric die is rolled 3 times. If it is known that face 1 appeared at
least once what is the probability that it appeared exactly once?
In a deck of 52 cards there are 4 kings. A card is drawn at random
from the deck and its face value noted; then the card is returned. This
procedure is followed 4 times. Compute the probability that there are
exactly 2 kings in the 4 selected cards if it is known that there is at least
one king in those selected.
Show that if A, B, and Care three events such that P(A n B n C) #: 0
and P(C I A n B) = P(C I B), the~ P(A IBn C) = P(A I B).
A man fires 12 shots independently at a target. What is the probability
that he hits the target at least once if he has probability 9/10 of hitting
the target on any given shot?
A die is rolled 12 times. Compute the probability of getting
(a) 2 sixes;
(b) at most two sixes.
Suppose the probability of hitting a target is 1/4. If eight shots are
fired at the target, what is the probability that the target is hit at least
twice?
In Exercise 44, what is the probability that the target is hit at least twice
if it is known that it is hit at least once?

Combinatorial

Analysis

Recall from Section 1.2 that a symmetric probability space having s points is the
model used for choosing a point at random from a set S having s points. Henceforth when we speak of choosing a point at random from a finite set S, we shall
mean that the probability assigned to each one-point set is s- 1 , and hence the
probability assigned to a set A havingj points isjjs.
Let N(A) denote the number of points in A. Since P(A) = N(A)fs, the problem
of computing P(A) is equivalent to that of computing N(A). The procedure for
finding P(A) is to "counf' the number of points in A and divide by the total
number of points s. However, sometimes the procedure is reversed. If by some
means we know P(A), then we can find N(A) by the formula N(A) = sP(A).
This reverse procedure will be used several times in the sequel.
The computation of N(A) is easy if A has only a few points, for in that case we
can just enumerate all the points in A. But even if A has only a moderate number
of points, the method of direct enumeration becomes intractable, and so some
simple rules for counting are desirable. Our purpose in this chapter is to present a
nontechnical systematic discussion of techniques that are elementary and of
wide applicability. This subject tends to become difficult quite rapidly, so we shall
limit our treatment to those parts of most use in probability theory. The first four
sections in this chapter contain the essential material, while the last four sections
contain optional and somewhat more difficult material.
2.1 ..

Ordered samples

Suppose we have two sets SandT. If S has m distinct points s 1 , s2 , . . ,

sm and T has n distinct points 11 , t 2 , , t,., then the number of pairs
(s" t1) that can be formed by taking one point from the setS and a second
from the set Tis mn. This is clear since any given element of the set S can
be associated with any of the n elements from the set T.
If S = {1, 2} and T = {1, 2, 3}, then there are six pairs:
(1, 1), (1, 2), (1, 3), (2, 1), (2, 2), (2, 3). Note carefully that the pair (1, 2)
is distinct from the pair (2, 1).
Example1.

27

28

Combinatorial Analysis

More generally, suppose we haven sets S 1 , S 2 , , Sn having sh s 2, . . ,

S 11 distinct points, respectively. Then the number ofn-tuples (x 1, x 2 , , X 11)
that can be formed where x 1 is an element from S 1 , x 2 an element from
S2, .. . , and X 11 an element from 8 11 is S1S2 S11 This is a quite ObViOUS
extension of the case for n = 2 discussed above. (A formal proof that the
number of n-tuples is s 1s 2 S11 could be carried out by induction on n.)
An important special case occurs when each of the sets S, 1 S i s n,
is the same set S having s distinct points. There are then s" n-tuples
(xh x 2, .. . , X 11) where each x, is one of the points of S.
S = {1, 2} and n = 3. Then there are eight n-tuples:
(1, 1, 1), (1, 1, 2), (1, 2, 1), (1, 2, 2), (2, 1, 1), (2, 1, 2), (2, 2, 1), (2, 2, 2).
Example 2.

The special case when the sets S, 1 S i < n, are the same set can be
approached from a different point of view. Suppose a box has s distinct
balls labeled 1, 2, ... , s. A ball is drawn from the box, its number noted
and the ball is returned to the box. The procedure is repeated n times.
Each of the n draws yields a number from 1 to s. The outcome of the n
draws can be recorded as an n-tuple (x1, x 2, ... , X11) , where x 1 is the
number on the 1st ball drawn, x 2 that on the 2nd, etc. In all, there ares"
possible n-tuples. This procedure is called sampling with replacement from
a population of s distinct objects. The outcome (x 1 , x 2 , , X 11) is called a
sample of size n drawn from a population of s objects with replacement.
We speak of random sampling with replacement if we assume that all of the
s" possible samples possess the same probability or, in traditional language,
are equally likely to occur.
A perfectly balanced coin is tossed n times. Find the
probability that there is at least one head.
Example 3.

Presumably the statement that the coin is perfectly balanced implies

that the probability of getting a head on a given toss is 1/2. If this is so,
and if we assume that flipping the coin n times is equivalent to drawing a
random sample of size n from a population of the two objects {H, T},
then each of the 2" possible outcomes is equally likely. Let A be the event
that there is at least one head, and let A, be the event that the ith toss
results in a head. Then A =
1 A,. But

U7=

P(A) = 1 - P(Ac)
=

1- p ( c01 A.r)
(n Ai)

= 1- p

i=l

and n;= 1 A~ occurs if and only if all of the n tosses yield tails. Thus
P(ni=t Ai) = 2- ", so P(A) = I - 2- ".

2.1.

Ordered samples

29

LetS denote a set having s distinct objects. We select an object from S

and note which object it is, but now suppose we do not return it to the set.
If we repeat this procedure we will then make a selection from the remaining (s - 1) objects. Suppose the procedure is repeated n - I additional
times, so that altogether n objects are selected. (Obviously we must have
n ~ sin this case.) Once again we may record the outcome as ann-tuple
(x 11 x 2, ... , x,.), but this time the numbers xh x 2, ... , x,. must be distinct;
there can be no duplications in our sample. The first object selected can be
any one of s objects, the second object can be any one of the remaining
s - 1 objects, the third can be any one of the remaining s - 2 objects,
etc., so in all there are (s),. = s(s - 1) (s - n + 1) different possible
outcomes to the experiment. This procedure is referred to as sampling
without replacement n times from a population of s distinct objects. We
speak of a random sample of size n drawn from a population of s objects
without replacement if we assume that each of these (s),. outcomes is equally
likely.
We have denoted the product s(s - 1) (s - n + 1) by the symbol
(s),.. In particular, (s).,
s(s - 1) I = s! Now drawing a sample of
size s from a population of s distinct objects is equivalent to writing down
the numbers, 1, 2, ... , s in some order. Thus s! represents the number of
different orderings (or permutations) of s objects.
Suppose a random sample of size n is chosen from a set of s objects with
replacement. We seek the probability of the event A that in the sample no
point appears twice. The problem is easily solved. The number of samples
of size n with replacement iss". Of theses" random samples the number
in which no point appears twice is the same as the number of samples of
size n drawn from s objects without replacement, i.e., (s),.. Thus since all the
s" samples are equally likely, we find that the required probability is

(1)

(s),.

-- =
s"

s(s- 1) ( s - n

1)

~--~--~------~

_ (1 _

s"

~)

(1 _

~) ... ( 1

_ n

1) .
5

A novel and rather surprising application of ( 1) is the

so-called birthday problem. Assume that people's birthdays are equally
likely to occur among the 365 days of the year. (Here we ignore leap years

and the fact that birth rates are not exactly uniform over the year.) Find
the probability p that no two people in a group of n people will have a
common birthday.
Example 4.

In this problems = 365, so by applying (I) we see that

Combinatorial Analysis

30

The numerical consequences are quite unexpected. Even for n as small as

23, p < 1/2, and for n = 56, p = .01. That is, in a group of 23 people the
probability that at least two people have a common birthday exceeds l/2.
In a group of 56 people, it is almost certain that two have the same
birthday.
If we have a population of s objects, there ares" samples of size n that
can be drawn with replacement and (s),. samples of size n that can be drawn
without replacement. If s is large compared to n, there is little difference
between random sampling by these two methods. Indeed, we see from (1)
that for any fixed n,
(2)

lim
s-+ao

(~~~~ =
S

lim
s-+ao

(1 - !) .. (1 - n S

1
)

= 1.

(For more precise estimates see Exercise 12.)

2.2.

Permutations

Suppose we have n distinct boxes and n distinct balls. The total number
of ways of distributing the n balls into the n boxes in such a manner that
each box has exactly one ball is n!. To say that these n balls are distributed
at random into the n boxes with one ball per box means that we assign
probability 1/n! to each of these possible ways. Suppose this is the case.
What is the probability that a specified ball, say ball i, is in a specified box,
say boxj? If ball i is in boxj, this leaves (n - 1) boxes and (n - I) balls
to be distributed into them so that exactly one ball is in each box. This
can be done in (n - 1)! ways, so the required probability is (n - 1) !/n! =
lfn.
Another way of looking at this result is as follows. If we have n distinct
objects and we randomly permute them among themselves, then the
probability that a specified object is in a specified position has probability
lfn. Indeed, here the positions can be identified with the boxes and the
objects with the balls.
The above considerations are easily extended from 1 to k > 1 objects.
If n objects are randomly permuted among themselves, the probability
that k specified objects are in k specified positions is (n - k)!/n!. We
leave the proof of this fact to the reader.
Problems involving random permutations take on a variety of forms
when stated as word problems. Here are two examples:
(a) A deck of cards labeled 1, 2, ... , n is shuftled, and the cards are
then dealt out one at a time. What is the probability that for some
specified i, the ith card dealt is the card labeled i?

2.3.

Combinations (unordered samples)

31

(b) Suppose 10 couples arrive at a party. The boys and girls are then
paired off at random. What is the probability that exactly k specified boys
end up with their own girls?
A more sophisticated problem involving random permutations is to find
the probability that there are exactly k "matches." To use our usual
picturesque example of distributing balls in boxes, the problem is to find
the probability that ball i is in box i for exactly k different values of i.
The problem of matchings can be solved in a variety of ways. We
postpone discussion of this problem until Section 2.6.
2.3.

Combinations (unordered samples)

A poker hand consists of five cards drawn from a deck of 52 cards.

From the point of view of the previous discussion there would be (52)s
such hands. However, in arriving at this count different orderings of the
same five cards are considered different hands. That is, the hand 2, 3, 4, 5, 6
of spades in that order is considered different from the hand 2, 4, 3, 5, 6 of
spades in that order. From the point of view of the card game, these hands
are the same. In fact all of the 5! permutations of the same five cards are
equivalent. Of the (52)5 possible hands, exactly 5! of them are just permutations of these same five cards. Similarly, for any given set of five
cards there are 5! different permutations. Thus the total number of poker
hands, disregarding the order in which the cards appear, is (52)5 /5!. In this
new count two hands are considered different if and only if they differ as
sets of objects, i.e., they have at least one element different. For example,
among the (52) 5 /5! poker hands, the hands (2, 3, 4, 5, 6) of spades and
(3, 2, 4, 5, 6) of spades are the same, but the hands (2, 3, 4, 5, 7) of spades
and (2, 3, 4, 5, 6) of spades are different.
More generally, suppose we have a setS of s distinct objects. Then, as
previously explained, there are (s), distinct samples of size r that can be
drawn from S without replacement. Each distinct subset {x., ... , x,} of
r objects from Scan be ordered (rearranged) in r! different ways. If we
choose to ignore the order that the objects appear in the sample, then these
r! reorderings of x., ... , x, would be considered the same. There are
therefore (s),fr! different samples of size r that can be drawn without
replacement and without regard to order from a set of s distinct objects.
The quantity (s),jr! is usually written by means of the binomial coefficient symbol

(s),. = (s).
r!

Combinatorial Analysis

32

Observe that for r = 0, 1, 2, . . . , s

(s)
r

s!

(s),.

= r! =r!(s-r)!.

We point out here for future use that ( ~) is well defined for any real
number a and nonnegative integer r by
(3)

a)
(r

= (a),. =

1) (a - r

a(a -

r!

1) ,

r!

where 0! and (a)0 are both defined to be 1.

Example 5.

(-1t)
3

= ( -1r)( -~

tX -n

3!
=-

1t(7t

1)(1t

2)

3!

Observe that if a is a positive integer then ( ~)

the convention that ( ~)

- 2)

= 0 if r >

= 0 if r is a negative integer.

a. We adopt

Then ( ~) is defined

for all real numbers a and all integers r.

As previously observed, when s is a positive integer and r is a nonnegative integer, it is useful to think of(;) as the number of ways we can
draw a sample of size r from a population of s distinct elements without
replacement and without regard to the order in which these r objects were
chosen.
Example 6.

Consider the set of numbers {1, 2, .. . , n}. Then if

1 < r < n, there are exactly(~) choices of numbers i 1 , i 2 ,

i,. such that

1 < i 1 < i2 < < i,. < n. Indeed, each of the (n),. choices of r distinct
numbers from 1 to n has r! reorderings exactly one of which satisfies the
requirement. Thus the number of distinct choices of numbers satisfying
the requirement is the same as the number of distinct subsets of size r
that can be drawn from the set { 1, 2, ... , n}.
The mathematics department
consists of 25 full professors, 15 associate professors, and 35 assistant
professors. A committee of 6 is selected at random from the faculty of
the department. Find the probability that all the members of the committee are assistant professors.
Example 7.

Committee membership.

2.3.

Combinations (unordeted samples)

33

In all, there are 75 faculty members. The committee of 6 can be chosen

7
ways. There are 35 assistant professors, and the 6 that
from the 75 in (
3
are on the committee can be chosen from the 35 in ( :) ways. Thus the
1
3
required probability is (
Calculations yield the approximate

i)

i)I(i).

value of .01; therefore the tenure staff (associate and full professors) need
not worry unduly about having no representation.
Consider a poker hand of five cards. Find the probability
of getting four of a kind (i.e., four cards of the same face value) assuming
the five cards are chosen at random.
Example 8.

We may solve the problem as follows.

5
There are ( ;) different hands, which are to be equally likely. Thus !l
5
will have ( ;) points. For the desired event to occur we must have four
cards of the same face value. There are 13 different choices for the value
that the four of a kind is to have, namely 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A.
For each such choice (which determines four of the five cards in the desired
hand) there are 48 other cards from which to choose the 5th card of the
hand. Since any of the 13 choices of the four of a kind can be paired with
any of the 48 choices remaining for the 5th card, in all there are (13)(48)
possible ways of getting a poker hand with four of the five cards equal.
The desired probability is therefore
(1 3)(48) ~ 2.40 x

(s;)

to- 4

Suppose n balls are distributed into n boxes so that all of

the Tf possible arrangements are equally likely. Compute the probability
that only box I is empty.
Example 9.

The probability space in this case consists of n" equally likely points.
Let A be the event that only box 1 is empty. This can happen only if the
n balls are in the remaining n - I boxes in such a manner that no box is
empty. Thus, exactly one of these (n - 1) boxes must have two balls, and
the remaining (n - 2) boxes must have exactly one ball each. Let B1 be the
event that box j, j = 2, 3, ... , n, has two balls, box 1 has no balls, and
the remaining (n - 2) boxes have exactly one ball each. Then the B1 are
disjoint and A = Uj= 2 B1 To compute P(B1) observe that the two balls
put in box j can be chosen from the n balls in (;) ways. The (n - 2)
balls in the remaining (n - 2) boxes can be rearranged in (n - 2)! ways.

34

Combinatorial Analysis

Thus the number of distinct ways we can put two balls into boxj, no ball in

(~) (n

box 1, and exactly one ball in each of the remaining boxes is

Hence

- 2)!.

(~) (n

- 2}!
P(B} = -'-''--n-,.- -

and consequently
(n -

P(A) =

1)

(n)2
n"

2.4.

(n - 2)!

(") (n - 1)!
= _,_2--'-----

n"

Partitions

A large variety of combinatorial problems involving unordered samples

are of the following type. A box has r red balls and b black balls. A
random sample of size n is drawn from the box without replacement.
What is the probability that this sample contains exactly k red balls (and
hence n - k black balls)?
To solve the problem we argue as follows. We are interested only in the
total number of red balls and black balls in the sample and not in the order
in which these balls were drawn. That is, we are dealing with sampling
without replacement and without regard to order. We can, therefore, take
our probability space for this problem to be the collection of all

(b ~ ')

samples of size n that can be drawn in this manner from the b

in the population.
probability

E!~h of these

r balls

(b : ') samples is assigned the same

(b ; ') . We must now compute the number of ways in

which a sample of size n can be drawn so as to have exactly k red balls.

The k red balls can be chosen from the r red balls in

(~)

ways without

regard to order, and then - k black balls can be chosen from the b black
balls without regard to order in

(n ~ k) ways. Since ea,ch choice of k r~d

balls could be paired with each choice of n - k black balls there are,
therefore, a total of
probability is

(~)

(n bk) possible choices.

Thus the desired

2.4.

35

PtJrtitions

The essence of this type of problem is that the population (in this case
the balls) is partitioned into two classes (red and black balls). A random
sample of a certain size is taken and we require the probability that the
sample will contain a specified number of items in each of the two classes.
In some problems of this type the two classes are not explicitly specified,
but they can be recognized when the language of the problem is analyzed.
A poker hand has five cards drawn from an ordinary deck
of 52 cards. Find the probability that the poker hand has exactly 2 kings.
Example 10.

To solve the problem note that there are (

ff) poker hands. In the

deck there are 4 kings and 48 other cards. This partitions the cards into
two classes, kings and non-kings, having respectively 4 and 48 objects
each. The poker hand is a sample of size 5 drawn without replacement
and without order from the 52 cards. The problem thus is to find the
probability that the sample has 2 members of the first class and 3 members
of the second class. Hence the required probability is

A deck of playing cards has 4 suits of 13 cards each,

namely clubs, diamonds, hearts, and spades.
Example 11 .

(a) What is the probability that in a hand of 5 cards exactly 3 are clubs?
(b) What is the probability that in a hand of 5 cards exactly 3 are of the
sam~ suit?
To solve problem (a) we note that the conditions of the problem partition the deck of 52 cards into 2 classes. Class one is the "clubs" having
13 members, and class two is "other than clubs" having 39 members. The
5 cards constitute a sample of size 5 from the population of 52 cards, and
the problem demands that 3 of the 5 be from class one. Thus the required
probability is
p=

8.15

10- 2

To solve (b) let A 1 be the event that exactly 3 cards are clubs, A 2 the
event that exactly 3 cards are diamonds, A 3 the event that exactly 3 cards
are hearts, and A 4 the event that exactly 3 cards are spades. Then since
there are only 5 cards in the hand, the events A 1, A 2 , A 3 , A4 are mutually

36

Combinatorial Analysis

disjoint. Their union, A 1 u A 2 u A 3 u A4 , is just the event that of the 5

cards exactly 3 are of the same suit. Thus the required probability is 4p.
Consider again a poker hand of 5 cards. What is the
probability that it is a full house (i.e., one pair of cards with equal face
value and one triple of cards with equal face value), assuming that the cards
are drawn at random from the deck?
5
To solve the problem we again note that there are (
poker hands
Example 12.

ff)

each of which is equally likely. Of these we must now compute the number
of ways in which we can have one pair and one triple. Consider the number
of ways we can choose a particular triple, say 3 aces, and a particular pair,
say 2 kings. The triple has 3 cards that are to be chosen without regard to
order from the four aces and this can be done in

(~) ways.

The pair has

two cards to be drawn without regard to order from the four kings. This
can be done in

(~)

ways. The total number of ways then of drawing a

hand having a triple of aces and a pair of kings is

(~) (~)

Thus the

probability of getting a poker hand that has a triple of aces and a pair of
5
kings is
= p. Of course, this probability would be the

(j) (i) j(ff)

same for any specified pair and any specified triple. Now the face value
of the cards on the triple can be any of the possible 13, and the face value
of the cards in the pair can be any of the 12 remaining possible face values.
Since each of the 13 values for the triple can be associated with each of the
12 values for the pair, there are (13)(12) such choices. Each of these
choices constitutes a disjoint event having probability p, so the required
probability is
(13)(12)p = (13)(12)(4)(6) ~ 1.44

10-3.

(sff)
In a poker hand what is the probability of getting exactly
two pairs? Here, a hand such as (2, 2, 2, 2, x) does not count as two pairs
but as a 4-of-a-kind.
Example 13.

To solve the problem we note that if the hand has two pairs, then two of
the cards have the same face value x, two of the cards have the same face
value y # x, and the fifth card has a different face value from x or y.
Now there are 13 different face values. The face values of the two pairs
1
can be chosen from them in (
ways. The other card can be any one of

i)

11 face values. The two cards of value x can be chosen from the four of

2.4.

Partitions

that value in

37

(~) ways and likewise for the 2 of value y.

card of value z can be chosen from the four of that value in

1
Thus t~~ ~umber of choices is ( ]) (11)
probabthty ts

The remaining

(i)

= 4 ways.

(i) (i) (4) so the desired

In some problems involving partitioning, the classes are imagined as in

the following.
Suppose we have a box containing r balls numbered
1, 2, ... , r. A random sample of size n is drawn without replacement and
the numbers on the balls noted. These balls are then returned to the box,
and a second random sample of size m is then drawn without replacement.
Find the probability that the two samples had exactly k balls in common.
Example 14.

To solve this problem we can argue as follows. The effect of the first
sample is to partition the balls into two classes, viz., those n selected and
those r - n not selected: (We can imagine that then balls selected in the
first sample are painted red before being tossed back). The problem is
then of finding the probability that the sample of size m contains exactly
k balls from the first class, so the desired probability is

If the argument were done in reverse, and we thought of the second

sample as marking the balls, then we would find that the probability is

We leave it as an exercise to show that these two are equal.

We can easily extend our consideration of partitioning a population into
two classes to partitioning it into m > 2 classes. Suppose We have a set of
r objects such that each object is one of m possible types. The population
consists of r 1 objects of type 1, r2 objects of type 2, . . . , r'" objects of type
m, where r1 + r2 + + r.,. = r. If a random sample of size n is drawn

38

Combinatorial Analysis

without replacement from the population of these r objects, what is the

probability that the sample contains exactly k 1 objects of type 1, ... , km
objects of type m, where k 1 + + km = n?
Once again the probability space is the collection of all (:) equally
likely samples of size n that can be drawn without replacement and without
regard to order from the r objects in the population. The k 1 objects of
type i in the sample can be chosen from the r 1 objects of that type without
regard to order in ( ~:) ways. Thus the probability of choosing the
sample with the specified composition is

In a hand of 13 cards chosen from an ordinary deck,

find the probability that it is composed of exactly 3 clubs, 4 diamonds,
4 hearts, and 2 spades.
Example 15.

In this problem r = 52, n = 13. Let class 1 be clubs, class 2 diamonds,

class 3 hearts, and class 4 spades. Then m = 4, k 1 = 3, k 2 = 4, k 3 = 4,
and k 4 = 2, so the desired probability is

In the committee problem discussed earlier, find the probability that the committee of 6 is composed of
2 full professors, 3 associate professors, and 1 assistant professor.
Example 16.

Committee problem.

Using the same method as above, we find the answer to be

2.5.

Union of events

Consider again the random permutation of n distinct objects. We say

a match occurs at the ith position if the ith object is in the ith position. Let
A, be the event that there is a match at position i. Then A =
1 A, is

Ui=

2.5.

39

Union of events

the event that there is at least one match. We can compute

for n = 2 by Equation (10) of Chapter 1 which states that

P(A 1 v A 2 )

P(A 1 )

P(A 2 )

P(A 1

r1

P(Ui'= 1 A 1)

A 2 ).

It is possible to use this formula to find a similar formula for n

A 1 , A 2 , and A 3 be three events and set B = A 1 u A 2 Then

3. Let

Now

(4)

P(B)

= P(A 1 u

A 2 ) = P(A 1)

Since B n A 3 = (A 1 v A2 ) n A 3
that

(5) P(B () A3 )

= P(A 1 fl

A3 )

P(A 2 )

P(A 1 n A 2 ).

(A 1 n A3 ) v (A 2 n A 3 ), it follows

P(A 2 n A 3 )

P(A 1 n A 2

r1

A3 ) .

Substituting (4) and (5) into the expression for P(A 1 u A 2 u A 3 ), we see
that
P(A 1 u A 2 u A 3 ) = [P(A 1)

+ P(Az)

- [P(A 1

r1

A3 )

- P(A 1 n A 2 )]

+ P(A 2 n

A3 )

+ P(A 3 )

+ P(A 2 ) + P(A 3 )]
[P(A 1 n A 2 ) + P(A 1 n A 3 ) +

P(A 1 n A 2 n A 3)]

= [P(A 1 )

+ P(A 1 n

P(A 2 n A 3 )]

A 2 n A 3 ).

In order to express this formula more conveniently, we set

St = P(At)

+ P(A2) + P(A3),

S 2 = P(A 1 n A2 )
and

S3

= P(A 1 n

P(A 1 n A3 )

P(A 2

()

A 3 ),

A 2 n A 3 ).

Then
(6)
There is a generalization of (6) that is valid for all positive integers n.
Let A 1 , . , A, be events. Define n numbers S,., 1 < r :::;;; n, by
S,. =

1:

1 ~it< ...

<i,.~ll

P(A 11

r1 r1

Then in particular

S 1 = P(A 1 )
11-1

S2

= I:

io;J

+ .-. + P(A,.),
II

L
j=i+l

P(A; n A),

A;J.

40

Combinatorial Analysis

and
S,.

P(A 1 n n A,.).

The desired formula for P(Ui= 1 A,) is given by:

P(,Q Al) = ,.t1 (-1Y- S,.

1

(7)

= S1

S2

+ + (- 1)"- 1S,..

The reader can easily check that this formula agrees with (6) if n = 3 and
with Equation (10) of Chapter 1 if n = 2. The proof of (7) proceeds by
induction, but is otherwise similar to that of (6). We will omit the details
of the proof.
The sum S 1 has n terms, the sum S 2 has

(~) terms, and in general the

sum S,. has ( ~) terms. To see this, note that the rth sum is just the sum
of the numbers P(A 11 n n A,r) over aU the values of the indices
i 1 ,. i 2 , , i,. such that i1 < i 2 < < i,.. The indices take values between
1 and n. Thus the number of different values that these indices can take is
the same as the number of ways we can draw r distinct numbers from n
numbers without replacement and without regard to order.
2.6.

Matching problems*

We now may easily solve the problem of the number of matches. Let
A, denote the event that a match occurs at the ith position and let p,.
denote the probability that there are no matches. To compute 1 - p,. =
P(U1= 1 A,), we need to compute P(A 11 n A12 n n A,,.) where i 1 ,
i 2 , , i,. are r distinct numbers from {I, 2, ... , n}. But this probability
is just the probability of a match at positions i 1, i 2 , , i,., and we have
already found that the probability of this happening is (n - r) !/n!. Since
the rth sum

sr has exactly ( ~) terms we see that

P(A 1

u.. uA,.)= f (n) (n-n! r)! (-1)"r= 1

f (-1)'-

r=1
"

( -l}r-1

r=1

r!

=L

n!
(n - r)!
r!(n - r)!
n!

that is,
(8)

(1 -

p,J

=1-

2!

1
+ -I - ... + ( -1)"- .

3!

n!

2.6.

Matching p1oblems

41

Using (8) we see that the probability, Pm that there are no matches is
(9)

Pn

1 - 1

1
2!

1
3!

I)"
n!

+ - - - + ''' + -

" ( 1)k
~
k=O
k!

= L

Now the right-hand side of (9) is just the first n + 1 terms of the Taylor
expansion of e- 1 Therefore, we can approximate Pn by e- 1 and get
1 - e- 1 = .6321 ... as an approximation to (I - Pn) It turns out that
this approximation is remarkably good even for small values of n. In the
table below we compute the values of(I - Pn) for various values ofn.

I - Pn

.6667

.6250

.6333

.6320

We thus have the remarkable result that the probability of at least one
match among n randomly permuted objects is practically independent of n.
The problem of matches can be recast into a variety of different forms.
One of the most famous of these is the following.
Two equivalent decks of cards are well shuffled and matched against
each other. What is the probability of at least one match?
To solve the problem we need only observe that the first deck can be
used to determine positions (boxes). With no loss of generality then we can
assume the cards in the first deck are arranged in the order 1, 2, . . . , n.
The cards in the second deck (the balls) are then matched against the
positions determined by the first deck. A match occurs at position i if and
only if the ith card drawn from the second deck is card number i.
Now that we know how to compute the probability Pn of no matches, we
can easily find the probability fJn(r) that there are exactly r matches. To
solve the problem we first compute the probability that there are exactly
r matches and that these occur at the first r places. This can happen only
if there are no matches in the remaining (n - r) places. The probability
that there are no matches among j randomly permuted objects is p1
Hence j! p1 is the number of ways that j objects can be permuted among
themselves so that there are no matches. (Why?) Since there is only pne
way of having r matches at the first r positions, the number of ways we
can have exactly r matches at the first r positions and no matches at
the remaining (n - r) positions is (n - r)! Pn-r Thus the required
probability is
(n - r)!

a., =

n!

Pn-r

The probability that there are exactly r matches and that these occur at
any specified r positions is the same for all specifications, namely, a.,.

Combinatolial Analysis

42

To solve the problem that there are exactly r matches, all that is now
necessary is to realize that the events "exactly r matches occurring at
positions i., i 2 , , i," are disjoint events for the various choices of

i 1 , i 2 , , i,.. The number of such choices is

probability is

(~) rx,..

(~)

Thus the required

Hence, if fJn(r) is the probability of exactly r matches

among n randomly permuted objects, we find that

(10)

()=
P,.r

n! rx,.
r I(
. n - )r I.
n!
(n- r)!p,._,
r!(n-r)!
n!

1 [
1-1+ -1 ++ ( l)n-r] .
r!
2!
(n - r)!

=-

Using the approximation that p,. _, is approximately e- 1 (which is very

good even for n - r moderately large) we find that
(11)

fJ,.{r)

e-t

~ -

r!

As a final illustration of these ideas, we compute the probability that

there is a match in the jth place given that there are exactly r matches.
To solve this problem let A 1 be the event that a match occurs at thejth
place and let B, be the event that there are exactly r matches. The desired
probability is P(A 1 I B,.). From (10), P(B,) = p,._,fr!, so we need to
compute P(A1 n B,). Now the event A 1 n B, occurs if and only if there is
a match in thejth place and exactly (r - 1) matches among the remaining
(n - 1) places. The number of ways in which we can have exactly (r - 1)
matches in the remaining (n - 1) places is (n - 1)! p,._ 1(r - 1). Thus
P(AJ 11 B,) = (n - 1)! P,. - 1(r- 1)

n.1

__ Pn-r

..::....::.........:...__

(r - 1)! n

Hence
P(Ai I B,) =

p,._, ' ~

n(r-1).p,._,

!_

2. 7.

Occupancy ptoblems

2.7.

43

Occupancy problems

A large variety of combinatorial probability problems are equivalent to

the problem of distributing n distinct balls into r distinct boxes. Since
each of the n balls can go into any one of the r boxes, in all there are r"
different ways of distributing the balls into the boxes. Assuming the balls
are distributed at random into the boxes, each of these r" ways has probability r -n. The underlying probability space Cl therefore has r" equally
likely points. In the problems involving this distribution of balls, we
impose various conditions on the occupancy of the boxes and ask for
the probability that our stipulated situation occurs. As a first example
consider the following problem.
If n balls are distributed at random into r boxes, what is the probability
that no box has more than one ball?
To solve the problem note first of all that the required probability is 0 if
n > r, so assume n < r. Then (thinking of distributing the balls one by
one) the first ball can go in any one of r boxes, the second into any one of
the remaining (r - 1) boxes, etc., so in all there are (r),. different ways.
The required probability is then (r),/r".
This probability is exactly the same as that of drawing a sample of size
n with replacement from a population of r objects and having all elements
in the sample distinct. Also note that r" is the number of samples of size n
from a population of r distinct objects. This is no accident. Random
sampling n times with replacement is formally the same as the random
distribution of n balls into r boxes. To see this, just think of distributing
the balls into the boxes as follows. We first draw a random sample of size
n from a set of r objects, and if the ith element in the sample was the jth
object we place ball i in boxj. It is sometimes useful to think of random
sampling with replacement in this manner, i.e., as the random distribution
of balls into boxes (see the coupon problem at the end of the chapter).
Consider the random distribution of n balls into r boxes. What is the
probability that a specified ball, say ball j, is in a specified box, say box i1
If ballj is in box i, then we have (n - 1) more balls to distribute in the r
boxes with no restrictions on where they go. Ball j can be placed in box i
in only one way, and the (n - 1) remaining balls can be placed into the
r boxes in r"- 1 ways. Thus the required probability is r"- 1/r" = lfr.
Translated into the language of random sampling we see that in a random
sample of size n drawn with replacement from a population of r objects,
it is equally likely that the jth element in the sample is any one of the
r objects.
The above considerations extend easily from one specified box to k
boxes, 1 ~ k ~ r. We leave it as an exercise to show that the probability

Combinatorial Analysis

44

that k specified balls are ink specified boxes is just r -k. In the language of
random sampling this says that if a sample of size n is drawn with replacement from a population of r objects, then the probability that the .i1 th,
j 2 th, . .. ,Ath elements in the sample are any k prescribed objects is ,-k.
Let A 1(i) be the event that the jth element in the sample is the ith object.
Then we have just said that for any choicej1 < j 2 < <A, 1 < k < n,
of elements in the sample (i.e., balls) and any choice it> i2, . .. , ik of objects
(i.e., boxes),
P(Ah(i 1) n Ah(i 2 ) n n A1"(ik)) = ,-k.

Since P(AJ.i)) = ,- 1 for any j and i, we see that

(12)

P(Ah(i 1 ) n n Ai"(ik)) = P(Ait(i 1)) P(Ai"(ik)).

Since this is true for all k and all choices of j 1 , . . ,jh we see that for any
ih i 2 , , i,. the events A 1(i1), . , A,.(i,.) are mutually independent.
If we think of drawing a random sample of size n from a set of r distinct
objects as ann-fold repetition of the experiment of choosing one object at
random from that set of r distinct objects, then we see that the statement
taat the events A 1(i1 ), . . , A 11{i11) are independent says that the outcome of
one experiment has no influence on the outcome of the other experiments.
This, of course, is in good accord with our intuitive notion of random
sampling.
Suppose n balls are distributed at random into r boxes.
Find the probability that there are exactly k balls in the first r 1 boxes.
Example 17.

To solve the problem observe that the probability that a given ball is in
one of the first r 1 boxes is r 1fr. Think of the distribution of then balls as
an n-fold repetition of the experiment of placing a ball into one of the
r boxes. Consider the experiment a success if the ball is placed in one of the
first r 1 boxes, and otherwise call it a failure. Then from our results in
Section 1.5, we see that the probability that the first r1 boxes have exactly
k balls is

2.8.

Number of empty boxes

We return again to consider the random distribution of n balls into r

boxes and seek the probability p,c(r, n) that exactly k boxes are empty.
To begin solving the problem, we let A 1 be the event that the ith box is
empty. For this event to occur, all of then balls must be in the remaining
(r - 1) boxes, and this can happen in (r - It ways. Thus P(A 1) =
(r - 1t/r 11 = (1 - 1/r)".

2.8.

45

Number of empty boxes

Similarly, if 1 < i 1 < i 2 < < i" < r, then the event A, 1 fl A, 2 n
fl A1k occurs if and only if all of the balls are in the remaining r - k
boxes. Consequently, P(A 11 fl fl A 1k) = (r - k)"/r" = (1 - k/r)".
We can now apply (7) to compute the probability of A 1 u u A,,
which is just the event that at least one box is empty. In this situation

S"

= (~) (I

- k/r }", so using (7) we find that

r

Ik=l (-1)"- 1S"

P(A 1 u u Ar) =

(-

1)"- 1

( ') (

k=l

~)
".
r

1 -

Thus the probability p 0 (r, n) that all boxes are occupied is

(13)

p0(r, n)

= 1 - P(A 1 u u Ar)

.(

-1)j-l

= 1 -

J= t

(~)
1

jtO (-l)i (~) ( 1 -

(1 -

;r.

1)"
r

As a next step, let us compute the probability rx~c(r, n) that exactly k

specified boxes (say the first k) are empty. This event can occur only if the
n balls are all in the remaining r - k boxes and if none of these r - k
boxes are empty. The number of ways we can distribute n balls into
r - k boxes in such a manner that no box is empty is (r - k)"p0 (r - k , n).
Thus the required probability is

ex"( r, n) =

(14)

(r -

k)"p 0 (r - k, n)
r"

(1 - ;k)" p (r 0

k, n).

We may now easily compute the probabilities P~c(r, n). For each choice
of k distinct numbers i., i2 , , i~c from the set of numbers {1, 2, . . . , n},
the event {exactly k boxes i 1 , i 2 , , i" empty} has probability cx~c(r, n) and
these events are mutually disjoint. There are ( ~) such events and their
union is just the event {exactly k boxes empty}. Thus

P~c(r, n) = (~) ( 1 -

(15)

;r

p 0 (r - k, n).

Using the expression for p 0 (r, n) given in (13) we see that

(16)

P~c(r,

n). =

10

~) r-k
~ (-1)1 ( r j

k) (1- 1- +, - k)".

46

Combinstorisl Analysis

As does the problem of matches, occupancy problems have various

reformulations. We mention below one of the more famous of these.
Coupons or, in the present day, toys are placed in
cereal boxes to entice young purchasers. Suppose that there are r different
types of coupons or toys, and that a given package is equally likely to
contain any one of them. If n boxes are purchased, find the probability of
Coupon problem.

(a) having collected at least one of each type,

(b) of missing exactly k of the n types.
Exercises
1

The genetic code specifies an amino acid by a sequence of three

nucleotides. Each nucleotide can be one of four kinds T, A, C, or G,
with repetitions permitted. How many amino acids can be coded in
this manner?
The Morse code consists of a sequence of dots and dashes with repetitions permitted.
(a) How many letters can be coded for using exactly n symbols?
(b) What is the number of letters that can be coded for using n or
fewer symbols?
A man has n keys exactly one of which fits the lock.. He tries the keys
one at a time, at each trial choosing at random from the keys that were
not tried earlier. Find the probability that the rth key tried is the
correct key.
A bus starts with 6 people and stops at 10 different stops. Assuming
that passengers are equally likely to depart at any stop, find the
probability that no two passengers leave at the same bus stop.
Suppose we have r boxes. Balls are placed at random one at a time
into the boxes until, for the first time, some box has two balls. Find
the probability that this occurs with the nth ball.
A box has r balls labeled 1, 2, .. . , r. N balls (where N ~ r) are
selected at random from the box, their numbers noted, and the N balls
are then returned to the box. If this procedure is done r times, what is
the probability that none of the original N balls are duplicated?
If Sam and Peter are among n men who are arranged at random in a
line, what is the probability that exactly k men stand between them?
A domino is a rectangular block divided into two equal subrectangles
as illustrated below. Each subrectangle has a number on it; let these
X

be x and y (not necessarily distinct). Since the block is symmetric,

domino (x, y) is the same as (y, x). How many different domino
blocks can be made using n different numbers?

Exercises

47

Consider the problem of matching n objects, and let i and r denote

distinct specified positions.
(a) What is the probability that a match occurs at position i and no
match occurs at position r?
(b) Given that there is no match at position r what is the probability
of a match in position i?

1 o Suppose n balls are distributed in n boxes.

(a) What is the probability that exactly one box is empty? Hint: use
the result of Example 9.
(b) Given that box 1 is empty, what is the probability that only one
box is empty?
(c) Given that only one box is empty, what is the probability that box 1
is empty?
11

If n balls are distributed at random into r boxes, what is the probability

that box 1 has exactly j balls, 0 ~ j ::s; n?

12 Show that

1)"s

n( 1-

(s),.

~ - ~

1)"-t
.
s

( 1- -

13 A box has b black balls and r red balls. Balls are drawn from the

box one at a time without replacement. Find the probability that

the first black ball selected is drawn at the nth trial.
The following problem pertains to poker hands. A deck has 52
cards. These cards consist of 4 suits called clubs, diamonds, hearts,
and spades. Each suit has 13 cards labeled 2, 3, ... , 10, J, Q, K, A.
A poker hand consists of 5 cards drawn without replacement and
without regard to order from the deck. Poker hands of the following
types are considered to be in sequence: A, 2, 3, 4, 5; 2, 3, 4, 5, 6; .... ;
10, J, Q, K, A.
14 Compute the probability of each of the following poker hands

occurrmg:
(a) Royal flush ((10, J, Q, K, A) of the same suit);
(b) Straight flush (five cards of the same suit in a sequence);
(c) Four of a kind (face values of the form (x, x, x, x, y) where x andy
are distinct);
(d) Full house (face values of the form (x, x, x, y, y) where x andy are
distinct);
(e) Flush (five cards of the same suit);
(f) Straight (five cards in a sequence, regardless of suit);
(g) Three of a kind (face values of the form (x, x, x, y, z) where x, y,
and z are distinct);
(h) Two pairs (face values of the form (x, x, y, y, z) where x, y, and z
are distinct);
(i) One pair (face values of the form (w, w, x, y, z) where w, x, y, and z
are distinct).

48

Combin11toriBI An11/ysis

15 A box has 10 balls labeled 1, 2, ... , 10. Suppose a random sample of

16

17

18

19

20

size 3 is selected. Find the probability that balls 1 and 6 are among
the three selected balls.
Cards are dealt from an ordinary deck of playing cards one at a time
until the first king appears. Find the probability that this occurs with
the nth card dealt.
Suppose in a population of r elements a random sample of size n is
taken. Find the probability that none of k prescribed elements is in the
sample if the method used is
(a) sampling without replacement;
(b) sampling with replacement.
Suppose a random sample .of size n is drawn from a population of r
objects without replacement. Find the probability that k given objects
are included in the sample.
Suppose n objects are permuted at random among themselves. Prove
that the probability that k specified objects occupy k specified positions
is (n - k)!fn!.
With reference to Example 14, show that

box contains 40 good and 10 defective fuses. If 10 fuses are selected,

what is the probability they will all be good?

21 A

What is the probability that the bridge hands of north and south
together (a total of 26 cards) contain exactly 3 aces?
23 What is the probability that if 4 cards are drawn from a deck, 2 will be
black and 2 will be red?
22

24 Find the probability that a poker hand of 5 cards will contain no card

smaller than 7, given that it contains at least 1 card over 10, where aces
are treated as high cards.
25 If you hold 3 tickets to a lottery for which n tickets were sold and

5 prizes are to be given, what is the probability that you will win at
least 1 prize?
26 A box of 100 washers contains 5 defective ones. What is the probability that two washers selected at random (without replacement)
from the box are both good?
27

Two boxes each haver balls labeled 1, 2, . .. , r. A random sample of

size n < r is drawn without replacement from each box. Find the
probability that the samples contain exactly k balls having the same
numbers in common.

Discrete Random

Variables

Consider the experiment of tossing a coin three times where the probability of a
head on an individual toss is p. Suppose that for each toss that comes up heads we
win $1, but for each toss that comes up tails we lose $1. Clearly, a quantity of
interest in this situation is our total winnings. Let X denote this quantity. It is
clear that X can only be one of the values $3, $1, -$1, and -$3. We cannot with
certainty say which of these values X will be, since that value depends on the outcome of our random experiment. If for example the outcome is HHH, then X will
be $3; while for the outcome HTH, X will be $1. In the following table we list the
values of X (in dollars) corresponding to each of the eight possible outcomes.
(J)

P{ro}

X(w)

HHH

p3

HHT

p2(l - p)

HTH

p2(1 - p)

THH

p2(l - p)

HIT

-1

p(l - p)2

THT

-1

p(l - p)2

TTH

-1

p(l - p)2

TIT

-3

(1 - p)J

We can think of X as a real-valued function on the probability space corresponding to the experiment. For each w e 0, X(m) is then one of the values, 3, 1, -1, -3.
Consider, for example, the event {w: X(w) = 1}. This set contains the three points
w 2 , w 3 , and w 4 corresponding to the outcomes HHT, HTH, and THH, respectively.
The last column in the table gives the probabilities associated with the eight
possible outcomes of our experiment. From that table we see that the event
{w: X(w) = I} has probability 3p2 (1 - p). We usually abbreviate this by saying
49

Discrete Random Variables

50

that {X = 1} has probability 3p2 (1

the other values that X assumes.
of X, there is a precisely defined
shall see in the next section, the
discrete random variable.
3.1.

- p). Similar considerations, of course, apply to

We see, therefore, that for each possible value

probability for X assuming that value. As we
quantity X is an example of what is called a

Definitions

Let (!l, .91, P) be an arbitrary -probability space, and let X be a realvalued function on n taking only a finite or countably infinite number of
values x 1, x 2 , As in the example just given, we would certainly like to
be able to talk about the probability that X assumes the value x, for each i.
For this to be the case we need to know that for each i, {w e n: X (ro) = x 1}
is an event, i.e., is a member of d . If, as in the previous example, dis the
u-field of all subsets of n then this is certainly the case. For in that case,
no matter what xi might be, {ro: X(ro) = x 1} is a subset of!l and hence a
member of d, since .91 contains every possible subset of n. However, as
was indicated in Section 1.2, in general .91 does not consist of all subsets of
n, so we have no guarantee that {roE n: X(ro) = x,} is in .91. The only
reasonable way out is to explicitly assume that X is a function on n such
that this desired property holds. This leads us to the following.

Definition 1 A discrete real-valued random variable X on a

probability space (!l, .91, P) is a function X with domain !l and range a
finite or countably infinite subset {xtt x2 ~ } of the real numbers R
such that {ro: X( co) = x 1} is an event for all i.
By definition, then, {w : X(w) = x 1} is an event so we can talk about its
probability. For brevity we usually write the event {ro: X(ro) = x 1} as
{X = x 1} and denote the probability of this event as P(X = x 1) rather than
P({ro: X(ro) = x 1}).
Let X be a discrete real-valued random variable. Then for any real
number x, {w: X (ro) = x} is an event. Indeed, if x 1 , x2 ~ are the values
that X can assume, then {ro: X(ro) = x,} is an event by the definition of a
discrete real-valued random variable. If x is not one of these numbers,
then {w : X(w) = x} = 0, which is also an event.
If the possible values of a disc.r ete random variable X consist only of
integers or of nonnegative integers, we say that X is respectively an
integer-valued random variable or a nonnegative integer-valued random
variable. Most of the discrete random variabJes that arise in applications
are nonnegative integer-valued.

Definition 2 The real-valued function f defined on R by f(x) =

P(X = x) is called the discrete density function of X. A number xis
called a possible value of X iff(x) > 0.

3.1.

Definitions

Whenever necessary, we will denote the density function of X by fx to

show that it is the density function for the random variable X.
Let X be the random variable introduced at the beginning
of this chapter in our discussion of tossing a coin three times with, say,
p = .4. Then X has the discrete density f given by
Example 1.

f(- 3)

= .216,

= .432,

f( -1)

/(1)

.288,

f(3) = .064,

andf(x) = 0 if x =F -3, -1, 1, 3. This density can be represented in

terms of a diagram as illustrated in Figure 1.
.432

.288
.216

.064

-3

-2

- 1

Figure 1

Binomial distribution. Consider n independent repetitions of the simple success-failure experiment discussed in Section 1.5.
Let S,. denote the number of successes in the n trials. Then S,. is a random
variable that can only assume the values 0, 1, 2, . .. , n. In Chapter 1 we
showed that for the integer k, 0 < k ~ n,
Example 2.

P(S,. = k) =

(~) y(l -

p)"-";

hence the density f of S,. is given by

f(x) =

{(~) r(l
0,

- p)"-",

x = 0, 1, 2, . . . , n,
elsewhere.

This density, which is among the most important densities that occur in
probability theory, is called the binomial density with parameters nand p.
The density from Example 1 is a binomial density with parameters n = 3
andp = .4.
One often refers to a random variable X having a binomial density by
saying that X has a binomial distribution (with parameters n and p if one
wants to be more precise). Similar phraseology is also used for other
random variables having a named density.
As expJained in Chapter 2, the binomial distribution arises in random
sampling with replacement. For random sampling without replacement
we have the following.

Discrete Random Variables

62

Hypergeometric distribution. Consider a population of

r objects, of which r 1 are of one type and r2 = r - r 1 are of a second
type. Suppose a random sample of size n ~ r is drawn from the population. Let X be the number of objects of the first type in the sample.
Then X is a random variable whose possible values are 0, 1, 2, ... , n.
From the results in Section 2.4 we know that
Example 3.

x = 0, 1, 2, ... , n.

P(X = x) =

Now we can write

(rl}x(r - rl)n ~x n!
x!(n - x)! (r),.

_(n)x (r1)x(r -

r1)n-x.

(r),.

Thus the density f of X can be written in the two forms

f(x) =

0,

0, 1, 2, .. . , n,

elsewhere

or

x = 0, 1, 2, . .. , n,
elsewhere.
This density is called the hypergeometric density.
Here are a few more examples of random variables.
Constant random variable. Let c be a real number. Then
the function X defined by X(w) = c for all ro is a discrete random
variable, since the set {ro: X(ro) = c} is the entire set nand n is an event.
Clearly, P(X = c) = 1, so the density f of X is simply f(c) = 1 and
f(x) = 0, x :1: c. Such a random variable is called a constant random
variable. It is from this point of view that a numerical constant is considered a random variable.
Example 4 .

Indicator random variable. Let A be an event. Set

X(ro) = 1 if we A and X(w) = 0 if w A. Then the event A occurs if
and only if X = 1. This random variable is called the indicator random
variable of A because the value of X tells whether or not the event A occurs.
Example 5.

3. 1.

53

Definitions

Conversely, if X is a random variable on a probability space (Q, .91, P)

taking the values 1 or 0, then X is the indicator random variable of the
event
A = {w: X(w) = 1}.
Let p

= P(X =

1). The density f of X is then given by

f(O) = I - p,

and

f(I) = p,

x =F 0 or I.

f(x) = 0,

Consider the following game of chance. A circular target

of radius I is zoned into n concentric disks of radius 1/n, 2/n, .. . , n/n = 1,
as illustrated in Figure 2 for the case n = 5. A dart is tossed at random
onto the circle, and if it lands in the annula~ zone between the circles with
radii i/n and (i + 1)/n, n - i dollars are won, i = 0, 1, 2, ... , n - 1. Let
X denote the amount of money won. Find the density of X.
Example 6.

Figure 2

The probability space for this experiment will be chosen to be the

uniform probability space on the disk of radius 1. Clearly X is a discrete
random variable on this space with the possible values 1, 2, . . . , n. The
event A = {X= n - i} occurs if and only if the dart lands in the region
bounded by the circles of radii i/n and (i + 1)/n. According to our
discussion in Section 1.2 the probability of A is the area of A divided by
the area of the unit disk. Thus fori = 0, 1, 2, . . . , n - 1
P(X = n - i)

P(A)

e
:
1)
2]
-------------- -

n[

2 -

(;)

2i

----

n2

Setting n - i = x we see that the density of X is

2(n - x)
f(x) =

n2

0,

1
,

x = 1, 2, ... , n,

elsewhere.

The density f of a discrete random variable X has the following three

important properties:

54

Disctets Random Variables

(i) f(x)

0,

X E

R.

(ii) {x: f(x) #= 0} is a finite or countably infinite subset of .R. Let

{x1 , Xz, ... } denote this set. Then
(iii)

Li f(x,)

= I.

Properties (i) and (ii) are immediate from the definition of the discrete
density function of X. To see that (iii) holds, observe that the events
{co: X(w) = xi} are mutually disjoint and their union is n. Thus
~ f(xi)
i

= ~ P(X = xi)
i

=P(

y {X = xi}) = P(O) = 1.

Definition 3 A real-valued function f defined on R is called a

discrete density function provided that it satisfies properties (i), (ii),
and (iii) stated above.
It is easy to see that any discrete density function/is the density function
of some random variable X. In other words, given f we can construct a
probability space (0, .91, P) and a random variable X defined on n whose
discrete density isf Indeed, let/be given and suppose {x11 x 2 , } is the
set of values where f(x) #= 0. Take n = {x1 , x 2 , }, .91 all subsets
of n, and P the probability measure defined on .91 by P({w}) = f(x 1) if
co = xi. The random variable X defined by X(ro) = x 1 if ro = xi is then
such a random variable. To see this note that {co: X( co) = xi} = {x1} and
thus
P(X = x 1) = P({x1}) = f(x 1).

The above result assures us that statements like "Let X be a random

variable with discrete density /" always make sense, even if we do not
specify directly a probability space upon which X is defined. To save
writing we will henceforth use the term density instead of discrete density
throughout the remainder of this chapter.
The notion of a discrete random variable forms a convenient way of
describing a random experiment that has a finite or countably infinite
number of possible outcomes. We need not bother to set up a probability
space for the experiment. Instead we can simply introduce a random
variable X taking values x 1, x 2 , such that X= x 1 if and only if the
experiment results in the ith outcome. Thus, for example, in drawing a
card at random from a deck of n cards, we can let X = i if the ith card was
drawn. Then P(X = i) = n -l, so we could describe the experiment by
saying we observe a random variable X taking integer values 1, 2, .. . , n
and having f(x) = n- 1 for x = 1~ 2, ... n, andf(x) = 0 elsewhere for its
density function.

3 . 1.

55

Definitions

In general, conducting an experiment which has a finite or countably

infinite number of possible outcomes can be described as observing the
value of a discrete random variable X. Many times, in fact, this is how
the experiment already appears to us, and often it is easier to think of the
experiment in these terms rather than in terms of a probability space.
As an illustration of this idea consider the experiment of picking a point
at random from the finite subset S of R consisting of the distinct points
x 1 , x 2 , . , Xs. Then the function f defined by
f( )

s- 1 ,

x = ( 0,

x = x1 , x 2 , ,

Xs,

elsewhere

is clearly a discrete density function. A random variable X having this

density is said to be uniformly distributed on S. Observing a value of X
corresponds to our intuitive notion of choosing a point at random from S.
We will now introduce two more discrete densities that are very useful
for solving certain classes of problems whose importance will become
apparent later.
Geometric densities.

Example 7.

Let 0 < p < 1. Then the real-

valued function f defined on R by

f(x) =

{p(l
- pt,
0,

X = 0, 1, 2, ... ,
elsewhere

is a discrete density function called the geometric density with parameter p.

To see thatfis a density, all that needs to be checked is that condition
(iii) holds, for here conditions (i) and (ii) are obviously satisfied. But (iii)
follows from the familiar fact that the sum of the geometric series
I:.:=o(l -p)"isjustp- 1
Let a be any positive real
number and let 0 < p < 1. A density closely related to the geometric
is the negative binomial density with parameters a and p defined by
Example 8.

(1)

f(x)

Negative binomial densities.

(p" (-;) (-lY(l -

p)",

0,

= 0, 1, 2, . . . ,

elsewhere.

To show that this .is a density we must verify that properties (i)-(iii)
hold. Here property (ii) is obviously true. That (i) holds may be seen as
follows. For x a nonnegative integer,

= (-a)~
x!
= (-a)( -a -

1) .. (-a - x

,.
X.

1)

Disc1ete Random Variables

56

= (- 1t(ClXCl +

1) ... (Cl

X -

1)

x!

= ( -lY (a + X

n~

x!

Thus
(2)

p(~a)(-lY(l- PY

pa(a + ~

-I)ct- py.

Since the right-hand side of (2) is clearly nonnegative we see that (i) holds.
To verify (iii), recall that the Taylor series of (1 - t) -cz for - I < t < I is

f (-Cl) (-t)x.

(1 - t)-cz =

(3)

x=O

From (3) with t = 1 - p, we see that

p- =

(-a) (-lY(l -

x=O

p)x

and hence that Lx f(x) = I.

From (2) we see that we can write the negative binomial density in the
alternate form
(4)

f(x) =

p
{

cz

(a + x

X -

1) (l

)x
- p '

0,

= 0, 1, 2, ... ,

elsewhere.

For some purposes this form is more useful than that given in (1). Observe
that the geometric density with parameter p is a negative binomial density
with parameters a = I and p.
Let A. be a positive number.
Poisson density with parameter A. is defined as
Example 9.

Poisson densities.

The

= 0, 1, 2, . . . ,

elsewhere.
It is obvious that this function satisfies properties (i) and (ii) in the definition of a discrete density function. Property (iii) follows immediately from
the Taylor series expansion of the exponential function, namely,
00

A_X

e"=~-.
x=O X!
Many counting type random phenomena are known from experience
to be approximately Poisson distributed. Some examples of such phenom-

3.2.

Computations with densities

57

ena are the number of atoms of a radioactive substance that disintegrate in

a unit time interval, the number of calls that come into a telephone
exchange in a unit time interval, the number of misprints on a page of a
book, and the number of bacterial colonies that grow on a petri dish that
has been smeared with a bacterial suspension. A full treatment of these
models requires the notion of a Poisson process, which will be discussed in
Chapter 9.
3.2.

Computations with densities

So far we have restricted our attention to computing P(X = x). Often

we are interested in computing the probability of {ro: X(w) e A} where A
is some subset of R other than a one-point set.
Let A be any subset of R and let X be a discrete random variable having
distinct possible values x 1, x 2 , Then {ro : X(w) e A} is an event. To
see this, observe that
(5)

{ro I X(ro) e A}

U {ro I X(ro) =

x,eA

x1},

where by Ux,eA we mean the union over all i such that xi eA. Usually
the event {ro: X(ro) e A} is abbreviated to {X e A}, and its probability is
denoted by P(X e A). If - oo ~ a < b < oo and A is an interval with
end points a and b, say A = (a, b], then we usually write P(a < X S b)
instead of P(X e (a, b]). Similar notation is used for the other intervals
with these endpoints.
An abbreviated notation is also used for conditional probabilities. Thus,
for example, if A and B are two subsets of R we write P(X E A I X e B)
for the conditional probability of the event {X e A} given the event
{X eB}.
Let/be the density of X. We can compute P(X e A) directly from the
density /by means of the formula
(6)

P(X e A)

x,eA

f(x 1),

where by Lx,e A we mean the sum over all i such that x 1 eA. This formula
follows immediately from (5) since the events {ro I X(ro) = x 1}, i =
1, 2, .. . , are disjoint. The right side of (6) is usually abbreviated as
LxeA f(x). In terms of this notation (6) becomes
(7)

P(X e A) =

f(x).

xeA

The function F(t), - oo < t < oo, defined by

F(t) = _P(X S t) =

L
x~t

f(x),

-00

< t < oo,

Discrete Random Variables

58

is called the distribution function of the random variable X or of the

density f It follows immediately from the definition of the distribution
function that
P(a < X < b) = P(X < b) - P(X < a)

F(b) - F(a).

If X is an integer-valued random variable, then

[t]

F(t)

= l:

f(x),

x= -co

where [t] denotes the greatest integer less than or equal to t (e.g.,
[2.6] = [2] = 2). We see that F is a nondecreasing function and that,
for any integ,e r x, F has a jump of magnitude f(x) at x and F is constant on
the interval [x, x + 1). Further properties of distribution functions wi1l be
obtained, from a more general viewpoint, in Chapter 5.
SetS = {1, 2, ... , 10} and let X be uniformly distributed
on S. Then f(x) = 1/10 for x = 1, 2, . . . , 10 and f(x) = 0, elsewhere.
The distribution function of X is given by F(t) = 0 fort < I, F(t) = I for
t > 10 and
Example 10.

[t]

[t]

l:

F(t) =

f(x) = -

10

x= l

1 <X< 10.

A graph of this distribution function is given in Figure 3. The probability

P(3 < X < 5) can be calculated either as
P(3

= /(4) + /(5) = 2/10

< X < 5)

or as
P(3 < X < 5)

Similarly P(3

P(3

= F(5)

- F(3)

5jl0 - 3/10 = 2/10.

5) is obtained as

< X< 5)

= /(3) + /(4) + /(5)

= 3/10

or as
P(3 < X ~ 5)
Figure 3

= P(2 <

X < 5)

= F(5) - F(2) = 5/10 - 2/10 = 3/10.

P (X :5 t)

10

3.2.

Computations with densities

59

Let X have a geometric distribution with parameter p.

Find the distribution function of X and find P(X ~ x) for the nonnegative
integer x.
Example 11.

The density of X, according to Example 7, is

f(x) = (
Thus F(t)

= 0 fort

r,1 - PY',

= 0, 1, 2, . .. ,

elsewhere.

< 0 and
[t]

F(t) = ~ p(l - p)x,

t ~ 0.

x=O

Using the formula for the sum of a finite geometric progression we

conclude that
F(t) = 1 - (1 - p)ltl+ 1,
t ~ 0.
In particular, for x a nonnegative integer, F(x - l) = l - (l - p)x and
hence
P(X > x) = 1 - P(X < x) = 1 - P(X < x - 1)
=

1 - F(x - 1)

(1 - p)x.

Geometrically distributed random variables arise naturally in applications. Suppose we have a piece of equipment, such as an electrical fuse,
that neither deteriorates nor improves in the course of time but can fail
due to sporadic chance happenings that occur homogeneously in time. Let
the object be observed at fixed time periods such as hours or days, and let
X be the number oftime units up to and including the first failure, assuming
that the object is new at time 0. Clearly X is a discrete random variable
whose possible values are found among the integers 1, 2, 3, . . . . The
event {X = n} occurs if and only if the object first fails at the nth time
period. Our intuitive notion that the object neither deteriorates nor improves with time can be precisely formulated as follows. If we know that
the object has not failed by time n, i.e., the first failure is after time n so
X > n, then the probability that it does not fail until after time n + m, i.e.,
P(X > n + mIX > n), should be the same as the probability of starting
with an object which is new at time n and having it not fail until after time
n + m. The fact that the failure causes occur homogeneously in time can
be taken to mean that this probability depends only on the number of time
periods that elapse between n and n + m, namely m, but not on n. Thus
P(X > n) should satisfy the equation
(8)

P(X > n

+ m IX >

n) = P(X > m).

Since
P(X .> n

m I X > n) = P(X > n

P(X > n)

m),

60

Disc1ete Random Valiables

we can rewrite (8) as

+ m) = P(X > n)P(X > m), n, m = 0, 1, 2, . .. .

Setting n = m = 0 we see that P(X > 0) = P(X > 0) 2 , so P(X > 0)
equals 1 or 0. If P(X > 0) = 0, then P(X = 0) = 1, which is impossible
(9)

P(X > n

in our case since X can assume only values that are positive integers.
Therefore P(X > 0) = 1.
Set p = P(X = 1). Then P(X > 1) = 1 - p and from (9) we see that
P(X

> n + 1) = (1 - p)P(X > n).

By iteration on n it follows that P(X > n)

n = 1, 2, . . . ,
(10)

P(X = n) = P(X > n -

(1 - p)". Thus for

1) - P(X > n)

(1 _ p)"-1 _ (1 _ p)"

= p( 1

_ p)"- 1.

Ifp = 0 then P(X = n) = 0 for all n = 1, 2, ... and thus P(X = + oo)
= 1, i.e., the object never fails. We exclude this case from consideration.
Likewise p = 1 is excluded because then P(X = 1) = 1, so the object
always fails.
Let Y = X - 1. Then Y assumes the values 0, 1, 2, . . . with probabilities P(Y = n) = p(l - p)". We see therefore that Yhas the geometric
distribution with parameter p.
As we have just shown, the random variable Y = X - 1 is geometrically
distributed. This example is typical in the sense that geometrically
distributed random variables usually arise in connection with the waiting
time for some event to occur. We shall discuss this in more detail after we
treat independent trials in Section 3.4.
3.3.

Discrete random vectors

It often happens that we are interested in studying the relationship

between two or more random variables. Thus, for example, in drawing a
random sample of size n from a box of r balls labeled 1, 2, ... , r, we might
want to know the largest number Y on the balls selected as well as the
smallest number Z on the selected balls.
Let (0, d, P) be a probability space and let X 1 , X2 , . . , X, be r discrete
random variables defined on this space. Then for each point (J) e n each
of the random variables X 1 , , X, takes on one of its possible values,
which will be indicated by writing

X 1((J))

= xh X 2(w) = x 2 , ,

X,((J))

= x,.

Instead of thinking of observing r real numbers x 1 , x 2 , , x, we can

think of observing one r-tuple x = (x 1 , x 2 , , x,), where for each index

3.3.

Discrete random vectors

61

i, x 1 is one of the finite or countably infinite number of values that the

random variable Xi can assume.
Let K denote the collection of all r-tuples of real numbers. A point
x = (x., ... , x,}of K is usually called an r-dimensional vector. Thus for
each roe n, the r values X1 (ro), ... , X,(ro) define a point

X(ro)

(X1 (w), X2 (ro), ... , X,(ro))

of R'. This defines an r-dimensional vector-valued function on 0,

X: n --+ K, which is usually written X = (X1 , X 2 , . , X,). The function
X is called a discrete r-dimensional random vector.
We have just defined an r-dimensional random vector in terms of r realvalued random variables. Alternatively, an r-dimensional random vector
can be defined ,directly as a function X : n --+ K by extending the definition
of a real-valued random variable almost verbatim.

Definition 4 A discrete r-dimensional random vector X is a

function X from n toR' taking on a finite or countably infinite number
of values x 1 , x 2 , . . such that
is an event for all i.
The discrete density function/for the random vector X is defined by

f(x 1 ,

x,) = P(X1

= x 1 , . , X,

= x,)

or equivalently

f(x) = P(X = x),

xeK.

The probability that X belongs to the subset A of R' can be found by using
the analog of (7), namely,
P(X e A) =

f(x).

xe...t

As in the one-dimensional case, this function f has the following three

properties :
(i) f(x)

0, x e K.

(ii) {x: f(x) #: 0} is a finite or countably infinite subset of R', which

will be denoted by {x 1 , x 2 , . }.
(iii)

Lt f(x 1)

= 1.

Any real-valued function f defined on R' having these three properties

will be called a discrete r-dimensional density function. The argument
given in the one-dimensional case applies verbatim to show that any rdimensional discrete density function is the density function of some
r-dimensional random vector.

82

Discrete Random Variables

There is a certain amount of traditional terminology that goes along

with random vectors and their density functions. Let X = (X1, X 2 , , X,)
be an r-dimensional random vector with density f. Then the function f is
usually called the joint density of the random variables X 1, X 2 , , X,.
The density function of the random variable X 1 is then called the ith
marginal density of X or off.
Let X and Y be two discrete random variables. For any real numbers x
and y the set {ro I X(ro) = x and Y(m) = y} is an event that we will
usually denote by {X= x, Y = y}. Suppose that the distinct possible
values of X are x 1 , x 2 , , and that the distinct possible values of Yare
y 1 , y 2 , . For each x,. the events- {X =-x, Y = y 1}, j = l, 2, . .. , are
disjoint and their union is the event {X = x}. Thus
P(X = x) = P

(y {X= x,

= ~j P(X

=X,

Y =

Y))

= YJ) = L, P(X

=X,

y = y).

This last expression results from using the same notational convention
that was introduced for random variables in Section 3.2. Similarly,
P(Y = y) = P (

{ X= xi, Y = y})

= ~ P(X

= y) = 1: P(X = X ,

= x, y

= y).

In other words, if we know the joint density of two discrete random

variables X and Y then we can compute the density fx of X by summing
over y and the density fr of Y by summing -over x. Thus, in terms of
densities, iff is the joint density of X and Y, then
(11)

fx(x)

~ f(x, y)
y

and
fr(Y) = ~ f(x, y).

(12)

Suppose two cards are drawn at random without replacement from a deck of 3 cards numbered l, 2, 3. Let X be the number on the
first card and let Y be the number on the second card. Then the joint
density f of X and Y is given by f(l, 2) = f(l , 3) = /(2, 1) = /(2, 3) =
/(3, 1) = /(3, 2) = 1/6 and f(x, y) = 0 elsewhere. The first marginal
density, that is, the density of X is given by
Example 12.

fx(I)

= f(l,

1}

+ /(1, 2) + /(1, 3)

+ 1/6 + 1/6 = 2/6 = l /3

Therefore fx(x) = 1/3, x = 1, 2, 3, and

and similarly for x = 2 and 3.

fx(x) = 0 elsewhere, as it should be.

3.4.

63

Independent random variables

Suppose X and Y are random variables that assume the

values x and y, where x = 1 or 2 andy = I, 2, 3, 4, with probabilities
given by the following table.
Example 13.

I/4

I/8

I/I6

I/16

l/16

1/16

1/4

1/8

Then /x{l) = L:=l /(1, y) = l/4 + I/8 + 1/16 + 1/16 = 1/2, and
fx(2) = 1 - fx(l) = 1/2 so X has the uniform distribution on 1, 2.
Similarly

/r(l) = 1/4
3.4.

+ 1/16 = 5/16, /r(2) = 3/16, fr(3) = 5/16, /r(4) = 3/16.

Independent random variables

Consider the experiment of tossing a coin and rolling a die. Intuitively,

we believe that whatever the outcome of the coin toss is, it should have no
influence on the outcome of the die roll, and vice-versa. We now wish to
construct a pr.obability model that reflects these views. Let X be a random
variable that is 1 or 0 according as the coin lands heads or tails, i.e., such
that the event {X = 1} represents the outcome that the coin lands heads
and the event {X = 0} represents the outcome that the coin lands tails. In
a similar way we represent the outcome of the die roll by a random variable
Y that takes the value I, 2, ... , or 6 according as the die roll results in the
face number 1, 2, ... , or 6. The outcome of the combined experiment can
then be given by the random vector (X, Y). Our intuitive notion that the
outcome of the coin toss and die roll have no influence on each other can
be stated precisely by saying that if x is one of the numbers 1 or 0 and y is
one of the numbers 1, 2, ... , 6, then the events {X= x} and {Y = y}
should be independent. Thus, the random vector (X, Y) should have the
joint density f(x, y) given by
f(x, y) = {P(X = x)P(Y
0,

y),

0, 1, y = 1, 2, .. . , 6,
elsewhere.
X =

In other words the joint density f of X and Y should be given by

f(x, y)

= fx(x)fy(y).

Definition 5 Let X 1 , X2 ~ , Xr be r discrete random variables

having densities f 1, f 2 , . , /, respectively. These random variables

Discrete Random Variables

64

are said to be mutually independent if their joint density function f is

given by
f(xh x2, ... , x,.) = ft (xt)/2(x2) /,{x,.).

(13)

The random variables are said to be dependent if they are not independent.
As in the case of the combined experiment of tossing a coin and rolling a
die, the notion of independent random variables forms a convenient way
to precisely formulate our intuitive notions that experiments are independent of each other.
Consider two independent discrete random variables having densities
fx andfy, respectively. Then for any two subsets A and B of R
P(X E A, y E B)

(14)

P(X E A)P(Y E B).

To see this, note that

P(X e A, Y e B)

I:

fx.r(x, y)

xeA yeB

~ fx(x)fr(Y)

xeA )IEB

= [ .xeA
~ fx(x)]
=

[I: fr(Y)]
,eB

P(X E A)P(Y E B).

Formula (14) above extends easily from 2 to r independent random

variables. Thus if At, A 2 , , A,. are any r subsets of R then
(15)

P(X t EAt, ... ' X,. E A,.)

P(X t E Al) .. P(X,. E A,.).

Let X and Y be independent random variables each

geometrically distributed with parameter p.
Example 14.

(a) Find the distribution of min (X, Y).

= X) =

(b) Find P(min (X, Y)

(c) Find the distribution of X

(d) Find P(Y = y I X

+Y

P(Y ~ X).
Y.

:::::: z) for y

= 0, 1, . . . , z.

To solve (a) we observe that for z a nonnegative integer

P(min (X, Y) > z)

P(X > z, Y > z) = P(X > z) P(Y > z),

so by Example 11
P(min (X, Y) ~ z)

= (1

- py(I - p)z

= (1

_ p)2z.

It follows from Example 11 that min (X, Y) has a geometric distribution

with parameter

3.4.

65

Independent 'andom va,iables

To solve (b) we observe that

00

P(Y ~ X) =

P(X = x, Y ~ X)

x=O
00

= l:

P(X

.x=O

= x,

Y > x)

00

~ P(X = x)P(Y ~ x)
.x=O
00

l: p(l
x=O

- Pt(l - PY

00

= p ~ (1 - p)2%
x=O

To solve (c) we let z be a nonnegative integer. Then

%

P(X

z) =

P(X = x, X+ Y = z)

x=O

=
=

..

l: P(X =
x=O
L"

x, Y = z - x)

P(X = x)P(Y =

z- x)

x=O
%

l:

p(t - pyp(t - p):-x

.x=O

The solution to (d) is given by

P(Y

= y IX + y =

z)

= P(Y =

y' X + y

P(X

P(X =

P(X

z - y, Y

P(X

+ Y

=zP(X

= z)

Y = z)
= y)

= z)

y)P(Y = y)
Y = z)

- p(l - p)"-1 p(l - p)>'

(z + l)p2 (1 - p)z
1

z+1

66

Discrete Random Variables

Consider some experiment (such as rolling a die) that has only a finite or
countably infinite number of possible outcomes. Then, as already explained, we can think of this experiment as that of observing the value of a
discrete random variable X. Suppose thC experiment is repeated n times.
The combined experiment can be described as that of observing the values
of the random variables Xh X 2 , , X,., where X, is the outcome of the
ith experiment. If the experiments are repeated under identical conditions,
presumably the chance mechanism remains the same, so we should require
that these n random variables all have the same density. The intuitive
notion that the repeated experiments have no influence on each other can
now be formulated by demanding that the random variables xh x2, ... '
X,. be mutually independent. Thus, in summary, n independent random
variables X 1 , , X,. having a common discrete density f can be used to
represent an n-fold independent repetition of an experiment having a finite
or countably infinite number of outcomes.
The simplest random experiments are those that have only two possible
outcomes, which we may label as success and failure. In tossing a coin, for
example, we may think of getting a head as a success, while in drawing a
card from a deck of r cards we may consider getting an ace as a success.
Suppose we make n independent repetitions of our simple experiment. We
can then describe the situation by letting Xh X 2 , , X,. ben independent
indicator random variables such that Xi = I or 0 according as the ith trial
of the experiment results in a success or failure. In the literature, trials that
can result in either success or failure are called Bernoulli trials, and the
above situation is described by saying we perform n Bernoulli trials with
common probability p = P(X1 = I) for success. In this context a random
variable that takes on the values I and 0 with probabilities p and 1 - p
respectively is said to have a Bernoulli density with parameter p.
The outcome of performing n Bernoulli trials can be given by the
random vector X = (X1 , X 2 , , X,.). The information conveyed in this
vector tells exactly which trials were a success and which were a failure.
Often, such precise information is not required, and all we want to know is
the number S,. of trials that yielded a success among the n trials. In
Example 2 we showed that S,. was binomially distributed with parameters
n and p. Observe that S,. = X 1 + + X,.. Any random variable Y
that is binomially distributed with these same parameters can be thought of
as the sum of n independent Bernoulli random variables Xh . .. , X,. each
having parameter p.
Let us now consider independent repetitions of an experiment that has a
finite number r ~ 2 of possible outcomes.
Consider an experiment, such as
rolling a die, that can result in only a finite number r of distinct possible

3.4.1.

The multinomial distribution.

3.4.

Independent random variables

87

outcomes. We may represent this experiment by saying we observe a

random variable Y that assumes the values 1, 2, .. . , r, so that the event
{Y = i} represents the fact that the experiment yielded the ith outcome.
Let p 1 = P( Y = i). If we perform an n-fold independent repetition of the
experiment; we can represent the outcome of these n trials as an n-dimensional random vector (Y1, . . , YJ, where the random variable Y1 corresponds to the jth trial. Here the random variables Y1 , , Y,. are mutually
independent and P(Y1 = i) = p 1
The random vector (Y1, . . , Y,.) tells us the outcomes of these n trials.
As in the case of r = 2 outcomes we often are not interested in such a
detailed account, but only would like to know how many of the n trials
resulted in each of the various possible outcomes. Let Xi, i = 1, 2, . . . , r,
denote the number of trials that yield the ith outcome. Then X 1 = x 1 if
and only if exactly x 1 of the n random variables Y1 , . . . , Y11 assume the
value i, i.e., exactly x 1 of then trials yield the ith outcome.
For example, for r = 3, n = 5, if
Y1 = 2, Y2 = 3, Y3 = 3, Y4 = 2, and Y5 = 2,

then

x1

= 0,

x2 = 3, and x3

= 2.

We will now compute the joint density of X 1, , X,. To this end let
xh x 2 , , x, be r nonnegative integers with sum x 1 + + x, = n. A
moment's thought shows that since the random variables Y1 , Y2 , , Y,.
are independent with a common density, every specified choice of x 1 of
them having value 1, x 2 of them having value 2, ... , Xr of them having
valuer, has the same probability, namely
. px,.
1 2
,
Pxpx2
Thus letting C(n; x 1,
see that
P(X 1

x,) denote the number of possible choices, we

x., ... , X, == x,) = C(n; x., ... , x,.)p~ 1

p;".

The computation of C(n ; x 1 , , x,) is a problem in combinatorial

analysis that can easily be solved by the methods of Chapter 2. The
simplest way of doing this is to think of the r values 1, 2, ... , r as r boxes
and the n trials as n balls. Then C(n; x 1 , , x,) is the number of ways we
can place the n balls into the r boxes in such a manner as to have exactly
x 1 balls in box 1, ... , exactly x,. balls in box r. If this is so, then box 1 has
x 1 balls. These x 1 balls can be chosen from then balls in

(:J

ways. The

remaining n - x 1 balls must be placed into the r - 1 boxes 2, ... , r in

such a manner as to have x 2 balls in box 2, ... , x, balls in box r. Thus
(16)

C(n; x., ... , x,) =

(:J

C(n- x 1 ; x 2 ,

x,).

68

Discrete Random Variables

It now follows by induction on r that

(17)

C(n; x" ... , x,.)

n!

= (x ')(
') ... (x,.')
x
1

Indeed, for r = 1 there is nothing to prove. Assume that (17) holds for
r - 1 boxes. Then from (16) we see that
C(n; x 1 ,

... ,

n!
(n - x )!
x,) = - - -- - ___,___ _ 1
(x1 !)(n- x 1)! (x2!)" (x,!)
n!
-=..:...___

as desired.
The joint density f of X1 ,
(18) f(xl, ... ' x,) =

. ,

-----

X, is therefore given by

n!
p"t ... p""
(x1 !) . .. (x,!) 1
.r '
x 1 integers ;;:: 0 such that x 1
(

+ + x, = n,

0, elsewhere.

Th!s density is called the multinomz'al density with parameters n and

Ph ,p,.
We observe at once that the r random variables X1 , . . . , X, are not
independent. In fact, since X1 + + X, = n, any r - I of them determine the rth. This, plus the fact that p 1 + + p, = 1, is sometimes
used to express the multinomial distribution in a different form. Let
x1, x 2 , , x,_ 1 be r - 1 nonnegative integers such that x1 + +
x,_ 1 < n. Then
n!
- - - -- -- -- - -- -- ----(x1!) .. (x,- 1 !)(n - x 1 - .. - x,- 1)!
X

p"t
.. . p.r-1
" "- (1 _ p _ .. _ p
)n-xt- -x,.-t
1
1
r-1

This form is convenient when we are interested in the first r - 1 outcomes

and think of the rth outcome as the outcome which is "not one of the
r - 1 outcomes." Thus in rolling a die we might be interested in only
knowing if a 2, 4, or 6 appeared. The experiment would then have four
possible outcomes "2," "4," "6," and "not (2, 4, 6)."
Let k be a nonnegative integer, k < r . A simple probability argument
shows that for x 1 , x 2 , . . , xk nonnegative integers such that x 1 +

xk

<

n,

n!

-------- - - ------- --(x1!) .. (xlt!)[n - (x 1 + .. + xJt)]!

X Pi
Pt"(l - (Pt + ... + Plt))"-("t + ... +:x~c>.
1

3.4.

Independent random variables

69

To see this, observe that in performing then trials we are now only interested in the k + 1 outcomes "1," "2," . . . , "k," and "not (I, 2, ... , k)."
Thus in essence we haven repeated trials of an experiment having k + 1
outcomes, with Xi being the number of times that the ith outcome occurs,
i = I, 2, ... , k. Equation (20) now follows from (19) with r - I = k.
There is
an important connection between the binomial distribution and the
Poisson distribution. Suppose, for example, that we perform n Bernoulli
trials with success probability p,. = ).jn at each trial. Then the probability
of having S,. = k successes in the n trials is given by

3.4.2.

Poisson approximation to the binomial distribution.

P(S,. = k) =

=
Now as n -+ oo, (n)kfnk
Consequently,
(21)

.
Itm

-+

(~)

(p,.)k(l - p,.)"-k

).k (n)k

k! nk

(t - ~)" (t - ~)
n

1, (l - ).jn)"

(11)
k (p,.)k(l

rt-+CXl

-+

-k.

e-J., and (I - ).jn)-k -+ 1.

- Pn)"-k = ).k e -). .

k!

In the derivation of (2I) we had np,. = ).. Actually (21) holds whenever
np,. -+ ). as n -+ oo.
Equation (21) is used in applications to approximate the binomial
distribution by the Poisson distribution when the success probability p is
smalJ and n is large. This is done by approximating the binomial probability P(S,. = x) by means of f(x) where f is the Poisson density with
parameter ). = np. The approximation is quite good if np2 is small.
The following example illustrates the use of this technique.
A machine produces screws, 1% of which are defective.
Find the probability that in a box of 200 screws there are no defectives.
Example 15.

Here we have n = 200 Bernoulli trials, with success probability

p = .01. The probability that there are no defective screws is
(1 - .01)200 = (.99)200 = .1340.

The Poisson approximation to this is given by

e-2oo<.oo

= e- 2 = .1353.

The fact that the Poisson distribution can arise as a limit of binomial
distributions has important theoretical consequences. It is one justification
for developing models based on Poisson processes, which will be discussed
in Chapter 9. The use of the Poisson approximation as a labor-saving

Discrete Random Variables

70

device in computing binomial probabilities is of secondary importance,

since the binomial probabilities themselves are readily computed.
3.5.

Infinite sequences of Bernoulli trials

Consider repeatedly performing a success-failure e:tperiment having

probability p of success until the first success appears. For any prescribed
number n of trials, there is the nonzero probability (1 - p)" that no success
occurs. Thus, in considering the number of trials until the first success, we
cannot limit ourselves to any prescribed number of trials, but instead must
consider an unending sequence of trials.
A given finite number n of trials constitute n Bernoulli trials, represented
by n independent Bernoulli random variables, X 1 , . . , X,.. To represent
an infinite sequence of Bernoulli trials we consider an infinite sequence
{X,.}, n > 1, of independent Bernoulli random variables having the same
parameter p.
In general, random variables Xh X 2 , . are said to be independent if
for any positive integer n, the random variables X 1 , , X,. are mutually
independent. It can be shown that, given any discrete density f, there is a
probability space (!l, .91, P) upon which are defined mutually independent
random variables X 1 , X 2 , . each having the density f
As our model for performing an unlimited sequence of Bernoulli trials,
we therefore take an infinite sequence {X,.}, n ~ 1, of mutually independent Bernoulli random variables such that P(X,. = 1) = p, n > 1. We
interpret X,. = 1 as meaning that the nth trial results in success, and
X,. = 0 as meaning that it results in failure.
Consider the number of trials W 1 until the first success. The random
variable W 1 can assume only the integer values 1, 2,.... The event
{ W1 = n} occurs if and only if the first n - 1 trials are failures and the
nth trial is a success. Therefore
{W1 = n}

{X1

= 0, ... , X,._ 1 = 0, X,.= 1}.

It follows that
P(W1 = n)

P(X1 = 0, . . . , X,._ 1

P(X1

0) P(X,._ 1

= 0, X,.

= 1)

= O)P(X,. =

= (1 - p)"-p.

Consequently

(22)

P(W1

1 = n) = p(1 - p)".

1 is geometrically distributed with parameter p.

1)

3.5.

Infinite sequences of Bemou/li t1ia/s

71

Let r > I be an integer and let T, denote the number of trials until the
rth success (so that the rth success occurs at trial T,.). Then T,. is a random
variable that can assume only the integer values r, r + 1, . ... The event
{T,. = n} occurs if and only if there is a success at the nth trial and during
the first n - I trials there are exactly r - 1 successes. Thus
{ T,

= n} =

{X1

+ .. +

=r-

XII_ 1

I} n {X 11

I}.

Since the two events on the right are independent and X 1 + + X11 _ 1
has a binomial distribution with parameters n - 1 and p, we see that for
n = r, r + 1, ...
P(T,

n)

= P(X 1 + . . +

X 11 -

= r - 1)P(X = 1)

1) ,_ 1(1_
- (~ =Dp'(l - p)~~-r.
Consequently
+ n - 1) p'(1 (23)
P(T, 1
n -

) 11 - r

- ( r-1 P

r = n) = (

11

r _

P)11

We see from Equations (4) and (23) that T, - r has the negative binomial
distribution with parameters a = r and p.
Let T0 = 0 and for any integer r > 1 let T,. be as above. Define
W1 = T 1 - T 1_ 1 , i = 1, 2, . . . . Then Wi is the number of trials after the
(i - 1)st success until the ith success. We will now show that for any integer
r > 1 the random variables W1 - I, W2 - 1, . .. , W,. - 1 are mutually
independent and have the same geometric density with parameter p.
To see this let n 1 , n 2 , . , n, be any r positive integers. Then the
event {W1 = n 1 , , W, = n,} occurs if and only if among the first
n1 + + n, trials all are failures except for trials

which are successes. Since the trials are mutually independent with
success probability p we see that
P(W1

"~> W,. = n,)

= (1-

p)"' - 1p(1- p)112 - 1p(l- p)11,.- 1p

n (p{l -

p)"'-IJ.

1=1

Thus the random variables W1 - 1, . .. , W, - 1 are independent,

geometrically distributed with parameter p.
Now clearly T,. - r = (W1 - 1) + .. + ( W,. - 1), so we see that
T, - r is the sum of r independent, geometrically distributed random
variables. We have previously found that T, - r is negative binomially

Discrete Random Variables

72

distributed with parameters rand p. We have thus established the interesting and important fact that the distribution of the sum of r independent,
identically distributed geometric random variables with parameter p is
negative binomially distributed with parameters r and p.
Further properties of infinite sequence~ of independent Bernoulli trials
will be treated in the exercises.
3.6.

Sums of independent random variables

In this section we discuss methods for finding the distribution of the sum
of a finite number of independent discrete random variables. Let us start
by considering two such variables X and Y.
We assume, then, that X and Y are independent discrete random
variables. Let x 1 , x 2 , . denote the distinct possible values of X. For any
z, the event {X + Y = z} is the same as the event

U {X = x1, Y = z
i

Since the events {X

it follows that
P(X

= xi,

Y = z) =

- x 1} .

z - x 1} are disjoint for distinct values of i,

L P(X = x1,
i

Y = z - x 1)

L P(X = x1)P(Y = z

- x1)

Li fx(xi)fy(z

- x,).

L fx(x)fr(z

- x).

In other words
(24)

fx+r(z) =

If X and Y are integer-valued random variables, then X + Y is also

integer-valued. In this case we can interpret (24) as being valid when z is
an integer and the variable x in the right-hand side of (24) ranges over the
integers. One further specialization is useful Suppose that X and Y
assume only nonnegative integer values. Then X + Y also assumes only
nonnegative integer values. If z is a nonnegative integer, then
fx(x)fy(z - x) = 0 unless xis one of the values 0, 1, . .. , z. Thus under
these assumptions (24) can be written as
%

(25)

fx+ r(z)

= L

fx(x)fr(z - x).

.x=O

Although Equation (25) is useful for computing the density of X + Y,

it is usually simpler to use probability generating functions . We will next
describe such functions and then give several important applications of

3.6.

73

Sums of independent random variables

their use in computing the density of the sum of independent random

variables.
Definition 6 Let X be a nonnegative integer-valued random
variable. The probability generating function tbx of X is defined as
CIO

CIO

<l>x(t} = ~ P(X
x=O

x)r = ~ fx(x}r,

-l=:;;;t<l.

x=O

We will now calculate <l>x(t) in three specific cases.

Binomial distribution.
bution with parameters n and p. Then
Example 16.

P(X

= x) =

Let X have a binomial distri-

(j p"(l -

p)"-x

From the binomial expansion formula

we conclude that

(26)

<l>x(t) == (pt

+ 1 - p)".

Negative binomial distribution. Let X have a negative

binomial distribution with parameters a. and p. Then
Example 17.

and hence
<l>x(t) =

f i' (-a.) (-1)"(1 -

p)"r

.r=O

ao

= p ~

(_ex)
'
( -t(l -

x=O

p))".

From the Taylor series expansion

with s = -t(I - p), it follows that

(27)

p
)llo.
c)x(t) = (
1 - t(1 - p)

74

Discrete Random Variables

Poisson distribution.
with parameter A.. Then

Let X have a Poisson distribution

Example 18.

A.xe-;.

P(X = x) = - -

x!

and hence
(llx(t)

e-;.

x=O

By settings

(A.t)x .
X!

= ).tin the Taylor series expansion

we see that
(28)

Let X and Y be independent, nonnegative integer-valued random

variables. Then
(29)
To see this, note that by (25)
00

(llx+r(t) =

fz(z)tz

z=O
00

= L

tz

z=O

fx(x)fr(z - x)

x=O

00

00

fx(x)tx

.x=O

z=x

00

00

I:

.x=O

fy(Z - x)tz-x

/x(x)r ~ fr(Y)t'
y= O

which is the desired result.

It follows easily from (29) by induction that if x., ... , xr are independent, nonnegative integer-valued random variables, then
(30)

The conclusions of the next theorem can be proven most easily by the
"generating function technique/' which is based upon the fact that if
00

!:

.x=O

00

axr =

!:

b.xtx,

-1 < t <

.1,

.x=O

then we may equate the coefficients of tx in the two power series and
conclude that ax = hx, x = 0, 1, 2, . . . . This shows that if two nonnegative integer-valued random variables have the same probability

3.6.

76

Sums of independent random variables

generating function, they must have the same distribution. In other words,
the probability generating function of a nonnegative integer-valued
random variable uniqueJy determines the distribution of that random
variable.
Theorem 1 Let X1 ,

X, be independent random variables.

(i) If X 1 has the binomial distribution with parameters n1 and p,

then X1 + + X, has the binomial distribution with

parameters n 1 + + n, and p.
(ii)

If X 1 has the negative binomial distribution with parameters a 1

and p, then X1 + + X, has the negative binomial distribution with parameters a 1 + + a, and p.

(iii) If Xi has the Poisson distribution with parameter A., then

X1 + + X, has the Poisson distribution with parameter

At + .. . + A.,.
Proofof(i). If the X 1's are as in {i), then by Example 16
4>x 1 + ... +x,.(t) = 4>x1 (t) 4>x,.(t)

+
(pt +

= (pt

1 - p)"1 (pt

1 - p)"1 + .. . +"...

1 - p)""

Thus the probability generating function of X1 + + X, is the same as

that of a random variable having a binomial distribution with parameters
n1 + + n, and p. This implies that X1 + + X, must have that
binomial distribution. For let
ax=

(n1 + + n,) P"(l X

P)"t + +,.-x

denote the corresponding binomial probabilities. Then

00

x=O

P(X 1

+ + X, = x)t"

= 4>x 1 + ... +x,.(t)

= (pt

1 - p)" 1 + ... +rt,.

Thus by equating coefficients we see that

P(X 1

and hence that X1

+ +

Proof of (ii). If the

x,~s

+ +

X,

= x) = ax

X, is binomially distributed as stated in {i).

are as in (ii), then by Example 17

4>x, + . . +x,(t) = 4>x,(t) 4>x,(t)

Discrete Random Variables

76

)CIIl (

1 - t(1 --. p)

(1 - tft - p)r

p
)CX,.
1 - t(1 - p)

+ +ex...

Thus the probability generating function of X 1 + + X, is the same as

that of a random variable having a negative binomial distribution with
parameters cx 1 + + ex, and p. It now follows by the same argument
used in proving (i) that X 1 + + X, has that negative binomial
distribution.
The proof of (iii) is similar to that of (i) and (ii) and is left as an exercise
for the reader.
I
Suppose cx 1 = = a, = 1 in statement (ii) of Theorem 1. Then
X 1 , . . . , X, are each geometrically distributed with parameter p, and (ii)
states that X 1 + + X, has a negative binomial distribution with
parameters r and p. This provides an alternative proof to the result
obtained in Section 3.5.
The next example illustrates the use of conditional probabilities.
Let Xb X 2 , . . . be independent nonnegative integervalued random variables having a common density. Set S 0 = 0 and
S.,. = X 1 + + X.,., n ~ 1. Let N be a nonnegative integer-valued
random variable and suppose that N, X 1 , X 2 , are independent. Then
SN = X 1 + + XN is the sum of a random number of random variables.
For an interpretation of SN suppose that at time 0 a random number N
of bacteria enters a system and that by time 1 the colony started by the
ith bacterium contains X1 members. Then SN is the total number of
bacteria present at time 1. Show that the probability generating function
of SN is given by
Example 19

(31)

-1<t<l.

To verify (31) we observe first that

00

P(SN

= x) = :!:

P(SN

x, N

n)

~ P(Sn

x, N

n)

n)P(S,

n== 0
00

r~=O

00

= :!:

P(N

= xI N =

n).

r~=O

Since N is independent of X 1, X 2 , , X.,., it is independent of S.,., and

hence P(S,. = x I N = n) = P(S11 = x). Thus
00

(32)

P(SN

x)

~ P(N
n=O

n)P(S,

x).

77

Exercises

Consequently for - 1 < t < 1

00

4>sN(t)

= L

rP(SN

x=O

co

x=O

co

tx ~ P(N = n)P(S,. = x)
n=O

00

= x)

~ P(N

00

n=O

n) ~ rP(S,.

x)

x=O

co

~ P(N = n) 4>s"(t)
n=O
00

L P(N =

n)(4>x,(t))"

= 4>N(4>x,(t)).

rt=O

Exercises

1 Any point in the interval [0, 1) can be represented by its decimal

expansion .x1 x 2 . ... Suppose a point is chosen at random from the

interval [0, 1). Let X be the first digit in the decimal expansion
representing the point. Compute the density of X.
2 Let X have the negative binomial density with parameters rx. = r
(r an integer) and p. Compute the density of X + r.
3 Suppose a box has 6 red balls and 4 black balls. A random sample of
size n is selected. Let X denote the number of red balls selected.
Compute the density of X if the sampling is (a) without replacement,
(b) with replacement.
4 Let N be a positive integer and let
f(x) = (c2x,
0,

x = 1, 2, ... , N,
elsewhere.

Find the value of c such that/is a probability density.

5 Suppose X is a random variable having density f given by

.l

.05

Compute the following probabilities:

(a) X is negative;
(b) X is even;
(c) X takes a value between 1 and 8 inclusive;
(d) P(X = - 3 I X ~ 0);
(e) P(X ;::: 3 I X > 0).
6 Suppose X has a geometric distribution with p = .8. Compute the

probabilities of the following events:

(a) X> 3;
(b) 4 s; X s; 7 or X > 9;
(c) 3 ~ X ~ 5 or 7 s; X ~ 10.

78

Discrete Random Variables

7 Let

9
10

11

12

13

X be uniformly distributed on 0, I, . .. , 99. Calculate

(a) P(X > 25);

(b) P(2.6 < X< I2.2);
(c) P(8 < X < I0 or 30 < X < 32);
(d) P(25 < X< 30).
Suppose a box has 12 balls labeled 1, 2, .. . , 12. Two independent
repetitions are made of the experiment of selecting a ball at random
from the box. Let X denote the larger of the two numbers on the balls
selected. Compute the density of X.
Suppose the situation is as in Exercise 8, except now the two balls are
selected without replacement. Compute the density of X.
Let X be a geometrically distributed random variable having parameter
p. Let Y = X if X < M and let Y = M if X ~ M; that is, Y =
Min (X, M). Compute the density of Y.
Let X be geometrically distributed with parameter p . Compute the
density of
(a) X 2 ;
(b) X+ 3.
Suppose a box has r balls numbered 1, 2, . .. , r. A random sample of
size n is selected without replacement. Let Y denote the largest of the
numbers drawn and let Z denote the smallest.
(a) Compute the probability P(Y < y).
(b) Compute the probability P(Z > z).
Let X and Y be two random variables having the joint density given
by the following table.

-I

-2

1/9

1/27

1/ 27

1/9

2/9

1/9

1/9

1/9

4/27

Compute the probability of the following events:

(a) Yis even;
(b) XYis odd;
(c) X > 0 and Y > 0.
14 Let

X and Ybe independent random variables each having the uniform

density on {0, 1, . .. ~ N}. Find

(a) P(X > Y);
(b) P(X = Y).
15 Let X and Y be as in Exercise 14. Find the densities of
(a) min (X, Y);

(b) max (X, Y);

(c)

IY- XI.

79

Exercises

16 Let X and Y be independent random variables having geometric

densities with parameters p 1 and p 2 respectively. Find

(a) P(X ~ Y);

(b) P(X = Y).
17 Let X and Y be as in Exercise 16. Find the density of
(a) min (X, Y);
(b) X+ Y.
18 Let X and Y be discrete random variables and let g and h be functions
such that the following identity holds:
P(X = x, Y = y) = g(x)h(y).
(a)
(b)
(c)
(d)

Express P(X = x) in terms of g and h.

Express P( Y = y) in terms of g and h.
Show that <Lx g(x))(L, h(y)) = 1.
Show that X and Y are independent.

19 Let X and Y be independent random variables each having a geometric

density with parameter p . Set Z = Y- X and M

(a) Show that for integers z and m > 0

P(M = m
'

= min (X,

= z) = {P(X = m - z)P(Y = m),

. P(X

m)P(Y

+ z),

Y).

z < 0,
z > 0.

(b) Conclude from (a) that for integers z and m > 0

P(M = m, Z = z) = p 2(1 - p)2m(l - p)lzl.
(c) Use (b) and Exercise 18 to show that M and Z are independent.
20 Suppose a circular target is divided into three zones bounded by con-

centric circles of radius 1/3, 1/2, and 1, as illustrated in the following

diagram.

Figure 4

If three shots are fired at random at the target, what is the probability
that exactly one shot lands in each zone?
21 Suppose 2r balls are distributed at random into r boxes. Let X 1 denote
the number of balls in box i.
(a) Find the joint density of Xtt ... , X,.
(b) Find the probability that each box contains exactly 2 balls.
22 Consider an experiment having three possible outcomes that occur with
probabilities p 1 , p 2 , and p 3 , respectively. Suppose n independent

80

Discrete Random VsTisb/es

repetitions of the experiment are made and let X 1 denote the number of
times the ith outcome occurs.
(a) What is the density of X 1 + X 2 ?
(b) Find P(X2 = y I X 1 + X 2 = z), y = 0, 1, ... , z.
23

Use the Poisson approximation to calculate the probability that at

most 2 out of 50 given people will have invalid driver's licenses if
normally 5% of the people do.

24 Use the Poisson approximation to calculate the probability that a box

of 100 fuses has at most 2 defective fuses if 3% of the fuses made are
defective.
25

A die is rolled until a 6 appears.

(a) What is the probability that at most six rolls are needed?
(b) How many rolls are required so that the probability of getting 6 is
at least I/2?
Exercises 26--30 are related problems concerning an infinite sequence
of Bernoulli trials as discussed in Section 3.5.

26 Let Ti be the number of trials up to and including the ith success. Let

0 < x 1 < < x, be integers.. Compute the probability

P(T1 = x 1 , T2 = x 2 ,

T, = x,).

Hint: Let W,. = T, - T,_ 1 , r > 2, and let W1 = T1 ; then

P(T1 = x 1 ,

T, = x,)

= P(W1 = xh w2 = x2- x1, ... ,

w,

= x,- x,-1).

Now use the fact that the random variables W1 - 1, . .. , W,. - 1 are
mutually independent random variables, each having a geometric
distribution with parameter p .
27

Let N" be the number of successes in the first n trials. Show that
P(T1 =

28

I N,.

1
= 1) = - '

x = 1, 2, .. . , n.

More generally, show that

P(T1 = Xtt T 2 = x 2 ,

7;. = x, 1 N,. = r) =

(~) -t,

0 < x 1 < x 2 < < x, :S n.

This shows that given there are r successes during the first n trials, the
trials at which these successes occur constitute a random sample of size
r (without replacement) from the "population" of possible positions.
29 Let k be a positive integer, k ~ r. From Exercise 28 we may readily
compute that
P('Jl

=X

IN,.= r) =

1) (nX)
r-k
.
(;)

X( k-1

81

Exercises

Indeed, if ~ ~ x then the kth success is at position x. In the first

x - 1 positions there must be exactly k - 1 successes, and in the last
n - x positions there must be exactly r - k successes. Since, given
Nn = r, the positions of the r successes are a random sample of size r
from the "population" of n positions, the result follows. Verify that
this is in fact so, by computing directly P(7k = x 1 N,. = r ).
30 Let 1 ~ i

<j

~ r

be nonnegative integers. Compute

P(Ti =

31
32

33

34

35

36

X,

1j = Y I Nn = r)

for 0 < x < y ~ n.

Suppose X and Yare independent random variables having the uniform
density on 1, 2, ... , N. Comput,e the density of X + Y.
Let X be uniformly distributed on {0, 1, 2, ... , N}. Find ~x(t).
Let X be a nonnegative integer-valued random variable whose
probability generating function is given by ~x(t) = e.t<rl-1), where
A. > 0. Find fx
Prove (iii) of Theorem 1.
Let X and Y be independent random variables having Poisson densities
with parameters A. 1 and A.2 respectively. Find P(Y = y I X+ Y = z)
for y = 0, ... , z. Hint: Use (iii) of Theorem l.
Let X, Y, and Z be independent random variables having Poisson
densities with parameters .l1 , A.2 , and .l3 respectively. Find
P(X

= x,

= y, Z = z I X +

=x +y +

z)

for nonnegative integers x, y, and z. Hint: Use (iii) of Theorem 1.

37 In Example 19 suppose that X 1 takes on the values 1 and 0 with
respective prol?abilities p and 1 - p, where 0 < p < l. Suppose also
that N has a Poisson density with parameter A..
(a) Use Equation (31) to find the probability generating function of SN.
(b) Use (a) to find the density of SN.
For an interpretation of SN suppose a random number N of cancer
cells is introduced at time 0 and that each cell, independently of the other
cells and independently of N, has probability p of surviving a treatment
of radiation. Let X 1 = 1 if the ith cell survives and let X 1 = 0 otherwise. Then SN is the number of cells that survive the treatment.
38 Solve (b) of Exercise 37 without using probability generating functions,
but using instead Equation (32) and the fact that X 1 + + X,. has a
binomial density.

Expectation of
Discrete Random
Variables

Let us consider playing a certain game of chance. In order to play the game, we
must pay a fee of a dollars. As a result of playing the game we receive X dollars,
where X is a random variable having possible values x 1 , x 2 , , x,. The question
is, should we play the game? If the game is to be played only once, then this
question is quite difficult. However, suppose we play the game a large number of
times. After n plays we would pay na dollars and receive X 1 + + X11 dollars.
If we assume that the successive plays of the game constitute independent repetitions of the same experiment (observing a value of X), then we can take the random
variables X 1 , X 2 , , X 11 as mutually independent and having the common density
f of X. Let N 11(x1) denote the number of games that yielded the value x, i.e., the
number of X,'s that assume the value xi. Then we can write
r

X 1 + + X 11

= 1=1
I: x 1Nix1).

The average amount received is then

X 1 +- - +- XII
---=. . :.: -_
n

t=l

x, [Nn{x1)] .
n

According to the relative frequency interpretation of probabilities, if n is large,

the numbers N 11(x1)/n should be approximately equal to f(x 1}, and thus the sum on
the right should be approximately equai to Jl = :Lr= 1 x,f(x1). Thus it seems reasonable to anticipate a net gain in playing the game if Jl > a and to expect a net
loss if Jl < a. If Jl = a we would anticipate just about breaking even.
The quantity :Lr= 1 x,f(x,) is called the expectation of the random variable X.
More generally, let X be any discrete random variable that assumes a finite number
of values x 1 , . , x,. Then the expected value of X, denoted by EX or p,, is the
number
r

(1)

EX =

I= 1

xJ(x1),

where f is the density of X.

Suppose X has the uniform distribution on the set {xh ... , x,}. Then f(x 1) =
P(X = x,) = ,-1, and from (1) we see that EX= (x 1 + + x,)r- 1 , so in this
82

lnt1oduction

83

case EX is just the arithmetic average of the possible values of X. In general,

(1) shows that EX is a weighted average of the possible values of X; the weight
attached to the ith value x 1 is its probability f(x 1}.
The expected value EX is also called the mean of X (or of the density f of X)
and iscustomarily denoted by p.. The mean is one way of trying to summarize a
probability distribution by a single number that is supposed to represent a "typical
value" of X. How good this is depends on how closely the values of X are clustered
about the value J.L We will examine this question in more detail when we discuss
the variance of X in Section 4.3.
Example 1. Binomial distribution.
parameters n and p. Find EX.

Let X have the binomial distribution with

For n = I, X assumes the two values 0 and 1 with probabilities (I - p) and p

respectively. Hence

EX = 0 P(X = 0)

I P(X = I) = p.

Since a random variable having a binomial density with parameters 1 and p is just
an indicator random variable, we see that we can find the probability of the event A
that X = 1 by computing the expectation of its indicator.
We now compute EX for any n ~ 1. In this case X assumes the values 0, 1, 2, . .. ,
n,and
EX =

(~) pi(1

j=O

- p)"- i ,

To calculate this quantity we observe that

(J~)

- j!(n-j)!
jn!

1)!
( j - 1}![(n- 1)n(n -

--------~--~-------

U-

1)]!

=n(;=D

Thus

EX= n

)=1

(~
}

1
1

y(l -

p)"-1.

Making the change of variable i = j - 1 we see that

Ex

= np "~1
"- (n -. 1) P'(1 1
1=0

P)n-1-1

By the binomial theorem

"i:t (" ~ 1) p'(l l=O

p)"-t-1

= [p + (I

so we see that

EX= np.

- p)]"-1

= I,

84

Expectation of Discrete Random Variables

4.1.

Definition of expectation

Suppose now that X is any discrete random variaMe having possible

values x 1 , x 2 , We would like to define the expectation of X as
00

(2)

I:

EX =

x1J(x1).

j=l

If X has only a finite number of possible values x 1, . , x, then (2) is

just our previous definition. In the general discrete case, this definition is
valid provided that the sum Li x 1f(x1) is well defined. For this to be the
case we require that LJ lx11f(x1) < oo. This leads to the following.

Definition 1 Let X be a discrete random variable having density

f. If Lj lxilf(xi) < oo, then we say that X has .finite expectation and
we define its expectation by (2). On the other hand if Lj= 1 lx11f(x1) =
oo, then we say X does not have finite expectation and EX is undefined.
If X is a nonnegative random variable, the fact that X has finite
expectation is usually denoted by EX < oo.
Poisson distribution. Let X be Poisson distributed with
parameter .A.. Then
Example 2.

00

j=l

--

~~.e

-A.

~
'--

J!

A
-).
e
(J - 1)!
.

J=l

~ )._i

'-- -

}=0

00

~ . A -A 1-J}-:-e
-

Ex --

j!

,_-A. A.
= ~~.c::
e

Geometric distribution.
ution with parameter p. Find EX.
Example 3.

--

,
11..

Let X have a geometric distrib-

Now
00

EX=

l: jp{l
}=0

= p(l

- p)i
00

- p)

l: j(1

- p)l-1

j=O
00

j=O

dp

L-

= -p(l - p)

(t - p)l.

Since a power series can be differentiated term by term, it follows that

d
EX = - p{1 - p)dp

00

l: (1
}=0

- p)l.

Using the formula for the sum of a geometric progression, we see that

(1)

EX= - p{l- p) !!__

dp p

1
-p(l- p) (- ).

p2

4.2.

Ptoperties of expectation

85

Consequently

EX= 1 - p .
p

We will next consider an example of a density that does not have finite
mean.
Letfbe the function defined on R by

Example 4.

= {x(x ~

f(x)

1) '

= 1, 2, . .. ,

elsewhere.

0,

The function/ obviously satisfies properties (i) and (ii) in the definition of
density functions given in Chapter 3. To see that f satisfies property (iii)
we note that
1
1
1
- --= - - -x(x + 1)
x
x + 1
and hence
00

00

f(x) =

x=l

= (1

[1

1]

- --

x=l

- 1/2)

+ (1 /2 - 1/3) + .. . =

1.

Thus (iii) holds andfis a density. Nowfdoes not have finite mean because
1
tX)

tX)

L lxlf(x) = x=1
LX+1

x=l

and it is well known that the harmonic series L:= 1 x- 1 does not converge.
4.2.

Propenies of expectation

Often we want to compute the expectation of a random variable such as

z = x1 + x2 or z = X 2 that is itself a function cp(X) of the random
vector X. Of course, if we know the density fz of Z, this can be done by
using (2). Quite frequently, however, the density of Z may not be known,
or the computation of EZ from a known density of Z may be quite
difficult. Our next result will give us a way of deciding if Z has finite expectation and, if it does, of computing EZ directly in terms of the density
fx and the function cp.
Before stating this result we introduce a notational convention. Let X
be a discrete r-dimensional random vector having possible values
x 1, x 2 , and density f, and let <p be any real-valued function defined on
K . Then Ls <p(x)f(x) is defined as
(3)

~ ({J(x)f(x)
](

= L qJ(x1)f(x1) .
J

86

Expectation of Discrete Random Variables

Theorem 1 Let X be a discrete random vector having density f,

and let q> be a real-valued function defined on R'. Then the random
variable Z = q>(X} has finite expectation if and only if
~ lq>(x)lf(x)

(4)

< oo

ll:

and, when ( 4) holds,

(5)

EZ

~ q>(x)f(x).
ll:

Proof Let z 1, z 2 , denote the distinct possible values of Z and let

x 1 , x 2 , . denote the distinct possible values of X. For any z1 there is at
least one x 1 such that z1 = q>(xi), but there may be more than one such xi.
Let A 1 denote the collection of such x,'s, that is,
A 1 = {x1 I q>{x1)
Then {X e A 1} and {Z

P(Z

= z1}
=

= z1}.

denote exactly the same events. Thus

z1) = P(X e A1)

~ fx(x).
x e.AJ

Consequently,
~ lzJI/z(zj) = ~ lz11P(Z = z1)
j

L lzil

Since q>(x)

~ fx(x)
XEAJ

z1 for x in A1, it follows that

Lj lzilfz(zi) = ~ L
j

lq>(x)l/x(x).

li:EAJ

By their definition, the sets A 1 are disjoint for distinct values of j, and their
union is the set of all possible values of X. Therefore

This shows that Z has finite expectation if and only if (4) holds.
If Z does have finite expectation, then by repeating the above argument
with the absolute signs eliminated, we conclude that (5) holds.
I

Let X be a random variable having density f and let <p(x)

= lxl .

Then
by Theorem I, lXI has finite expectation if and only if L x lxlf(x) < oo.
But, according to our definition of expectation, X has finite expectation if
and only if the same series converges. We see therefore that X has finite
expectation if and only if EIXI < oo .
We shall now use Theorem 1 to establish the following important
properties of expectation.

4.2.

87

Properties of expectstion

Theorem 2 Let X and Y be two random variables having finite

expectation.
(i) If cis a constant and P(X = c) = 1, then EX= c.

(ii)

If c is a constant,

then eX has finite expectation and E(cX) =

cEX.
(iii) X

Y has finite expectation and

E(X

Y) = EX

EY.

(iv) Suppose P(X > Y) = l. Then EX~ EY; moreover, EX=

EY if and only if P(X = Y) = I.
(v)

IEXI < EIXI.

Proof. The proof of (i) is quite simple. If P(X = c) = 1, then X has

density fx(x) = 0 for x ::!= c andfx(c) = 1. Thus by (2)
EX =

L xfx(x) = cfx(c) = c.
JC

To prove (ii) let q>(x)

ex and observe that

LX lcxlfx(x) = lei LX lxlfx(x)

< oo,

so eX has finite expectation. Thus by (5)

E(cX) =

L (cx)fx(x)

= c

L xfx(x) =

To establish (iii) set q>(x, y) = x

of X and Y. Then
~ lx
x,y

cEX.

JC

+y

and letf denote the joint density

Ylf(x, y) < ~ lxlf(x, y)

x,y

+ L IYlf(x,

y)

= L lxl ~ f(x, y) + L IYI L f(x, y)

X

)/

)/

= LX lxlfx(x) +

L IYl/r(Y) <
y

oo

and hence X+ Yhas finite expectation. Applying (5) we see that

E(X

Y) =

L
(x + y)f(x, y)
x,y

=~
xf(x, y) +
x,y

~ yf(x, y)
JC,)I

=EX+ EY.

To prove (iv) observe that Z

(iii) we see that
EX- EY

= X- Y

= E(X-

Y)

X+ (- Y), and by (ii) and

= EZ

= 1:zfz(z).

..

88

Expectation of Discrete Random Variables

Since P(Z ~ 0) = P(X ~ Y) = 1, the values z 1 that Z = X- Y

assumes must all be nonnegative. Thus Lz zfz(z) ~ 0 and hence
EX - EY > 0. This yields the first part of (iv). If EX = EY, then
EZ = 0. But then
0 = EZ = L zJz(z1) .
I

Now the sum of nonnegative terms can equal zero only if all the individual
terms equal zero. Since fz(z 1) > 0 it must be that z 1 = 0. Thus the only
possible value of Z is 0 and consequently P(Z = 0) = 1.
Finally, (v) follows from (iv) and (ii) because -lXI ~ X :S lXI and
hence -.EIXI < EX < EIXl . This completes the proof of the theorem. I
It easily follows from (ii) and (iiii.) that if X 1 , . . . , Xn are any n random
variables having finite expectation, and c 1, , en are any n constants, then

(6)

E(c 1 X 1

+ + cnXn)

= c 1EX1

+ +

cnEXn.

It is useful to know that a bounded random variable always has finite

expectation. More precisely,
Theorem 3 Let X be a random variable such that for some
constant M, P(IXI < M) = 1. Then X has finite expectation and
IEXI ~ M .
Proof Let x 11 x 2 , be the possible values of X. Then
all i. Indeed, if lx11 > M for some possible value x, then

P(IXI > M) > P(IXI

which contradicts the fact that P(IXI

L lxllf(x1)

< M

= lx11)
M)

L f(x1)

lx11<

M for

> 0,

1. Consequently
~ M,

so X has finite expectation. Moreover by (v) of Theorem 2,

IEXI ~ EIXI =

L lx1lf(xl) ~

M.

This completes the proof.

It follows easily from Theorem 3 and (iii) of Theorem 2 that if X and Y

are two random variables such that Y has finite expectation and for some
constant M, P(IX - Yl < M) = 1, then X also has finite expectation
and lEX - EYl ~ M . We leave the proof of this fact for the reader.
Since the expectation of the sum of two random variables is the sum of
their expectations, one might suppose that the expectation of a product is
the product of the expectations. That this is not true in general can be

4.2.

Properties of expectation

89

seen by considering the random variable X taking values I and - I each

with probability 1/2 and setting Y = X. Then EX = EY = 0 but
EXY = EX 2 = 1.
There is an important case when this product rule is valid, namely,
when X and Y are independent random variables. We state this more
formally as follows.

TMorem 4 Let X and Y be two independent random variables

having finite expectations. Then XY has finite expectation and
E(XY) = (EX)(EY).

(7)

Proof. Observe that since X and Y are independent, the joint density
of X and Y isfx(x)fy{y). Thus
~ lxylf(x, y) = ~ lxiiYifx(x)fr(Y)
x,y

X,)l

so XY has finite expectation. Using Theorem 1, we conclude that

E(XY} =

L (xy)fx(x}fr(Y)

= [~ xfx(x)]

[ ~ Yfr(Y)]

= (EXXEY).

The converse of this property does not hold; two random variables X
and Y may be such that E(XY) = (EX)(EY) even though X and Yare not
independent.
Let (X, Y) assume the values (1, 0), (0, 1), ( -1, 0), and
(0, -1) with equal probabilities. Then EX = EY = 0. Since XY = 0,
it follows that E(XY) = 0 and hence E(XY) = (EX)(EY). To see that
X and Yare not independent observe, for example, that P(X = O) =
P(Y = 0) = 1/2 whereas P(X = 0, Y = 0) = 0. Thus
Example 5.

P(X

= 0,

= 0)

:#= P(X = O)P(Y = 0).

It is often easier to compute expectations by using the properties given

in Theorem 2 than by using the definition directly. We now illustrate this
technique with several examples.
Binomial distribution. We already know from Example
1 that the mean of the binomial distribution with parameters n and p
is np. We can also derive this fact in a very simp1e manner by using the
property that the expectation of a sum is the sum of the expectations ((iii)
Example 6.

90

Expectation of Discrete Random Vsriables

of Theorem 2). To this end, let X 1, ... , X, be n independent Bernoulli

random variables having parameter p and setS, = X 1 + + X,. Then
S,. has the binomial distribution with parameters nand p. By the first part
of Example 1, EXi = p, I < i < n, and hence
II

ES,.

= E(X1 + +

=L

X,.)

EXi = np.

'"" 1

Hypergeometric distribution. Suppose we have a population of r objects, r1 of which are of type one and r - r1 of type two.
A sample of size n is drawn without replacement from this population. Let
S, denote the number of objects of type one that are obtained. Compute
Example 7.

ESII.

We know that S, has the hypergeometric distribution, so we could

compute ES, by using (2). It is far simpler, however, to proceed by
introducing indicator random variables X 1 , . , X, as follows. The
random variable Xi = 1 if and only if the ith element in the sample is of
type one. Then
EXi

But sft

= x1 + .. . +

= P(X1 =

I)= r 1
r

XII, so using (iii) of Theorem 2 we see that

ES,.

= L"

EXi

r
= n _J

i= 1

Note that the random variables X, 1 < i

n, are not independent.

Suppose we have a population of r distinct objects labeled

1, 2, ... , r. Objects are drawn with replacement until exactly k ~ r
distinct objects have been obtained. Let S" denote the size of the sample
required. Compute ESk.
Example 8.

It is clear that S 1 = 1 and hence ES 1 = I . Assume k ~ 2 and let

Xi = Si+ 1 - S;, i = 1, 2, ... , k - 1. Then clearly S" = 1 + X 1 +
+ X~c_ 1 Now Xi is the number of objects that must be drawn after the
ith new object enters the sample and until the (i + 1)st new object enters
the sample. A moment's thought shows that the event {Xi = n} occurs if
and only if the first n - I objects drawn after the ith neyt object enters the
sample duplicate one of the previous i objects, and the nth object drawn
after the ith new object enters is different from the previous i objects.
Thus, as the trials are independent,
P(Xi = n) =

(;')11-1 (1 - ;') ,

n = 1, 2, .. . .

This shows that the random variable Xi - 1 is geometric with parameter

Pi = 1 - (i/r). Hence by Example 3, E(X1 - 1) = pi" 1 (1 -Pi), and
EXi

P;- 1(1 -Pi)+ 1

p,- 1

:....

(1 - i/r)- 1

= r(r-

i}- 1 .

4.2.

91

Properties of expectation

Consequently,
(8)

ESk = 1

+ k-1
~ ( - ' - .)
i= 1

k-1 (

=~

i=o

r -

-.
r -

r(lr + r -

_1_
1

1) .

+ ... + __1_ _
r-k+

We point out for later use that it is clear from the construction of the X 1
that they are mutually independent random variables.
In the previous chapter we have seen that nonnegative integer-valued
random variables X play a prominent role. For these random variables
the following theorem can frequently be applied both to decide if X has
finite expectation and to compute the expectation of X.

Theorem 5 Let X be a nonnegative integer-valued random

variable. Then X has finite expectation if and only if the series
:L:= 1 P(X ~ x) converges. If the series does converge, then
OC)

EX = ~ P(X > x).

(9)

x=t

Proof We will show that

OC)

OC)

~ xP(X

(10)

x=l

= x) = L

P(X > x),

x=l

from which the theorem follows immediately. To this end we first write
the left side of (1 0) as
X

00

}: P(X = x) }: 1.
x=l

y=l

It is permissible to interchange the order of summation and rewrite this

expression as
OC)

OC)

OC)

P(X = x) = ~ P(X > y).

y=l x=y

y=l

Replacing the dummy variable y by the dummy variable x in the right side
of this equality, we obtain the right side of (10). This shows that (10)
holds, as desired.
I
For an elementary application of this theorem, suppose that X is a
geometrically distributed random variable having parameter p. Then
P(X > x) = (I - p)" and thus by the theorem
OC)

EX =

(1 - p)" = (1 - p)

(1 - p)2 = = p- 1(1 - p).

x=1

This agrees with the result found in Example 3.

92

Expectation of Discrete Random Variables

4.3.

Moments

Let X be a discrete random variable, and let r ~ 0 be an integer. We

say that X has a moment of order r if X' has finite expectation. In that
case we define the rth moment of X as EX'. If X has a moment of order r
then the rth moment of X - p,, where p, is the mean of X, is called the rth
central moment (or the rth moment about the mean) of X . By Theorem 1
we can compute the rth moment and the rth central moment of X directly
from the density f by the formulas
EX"=~ x'f(x)

(11)

JC

and
(12)

E(X - J.L)'

= ~ (x

- tt)'f(x).

In view of (11) and (12), the rth moment and rth central moment are
determined by the density f, and it therefore makes perfectly good sense
to speak of them as the rth moment and rth central moment of this
density.
Suppose X has a moment of order r; then X has a moment of order k
for all k :;:;; r . To see this, observe that if lxl < 1, then

lx"l = lxlk <

while for

lxl > 1,

Thus in either case it is always true that

lxlk < lxl" +

1.

Thus, by the comparison theorem for the convergence of series, we see

that
~ lxlkf(x) < ~ [lxl" + 1]/(x) = E(IXIJ + 1 < oo,
X

so

xk has finite expectation.

On the other hand, as was shown in Example 4, a random variable X

may not have even a first moment. A simple modification of this example
shows that a random variable may have a moment of order r but possess
no higher order moment. (See Exercise 9.)
The first moment (r = 1) is just the mean of X . In general, the more
moments of X that we know the more information we have gained about
the distribution of X; however, in applications it is the first two moments
that are usually of major interest.
By property (iii) of Theorem 2 we know that if X and Y both have a
finite first moment, then so does X + Y. We will now show that this
desirable property holds true for moments of order r as well.

4.3.

Moments

93

Theorem 6 If the random variables X and Y have moments of

order r, then X + Y also has a moment of order r.
Proof This theorem rests on the following simple inequality. For
any nonnegative integer j < r,
(13)

X,

R.

To see this, observe that if lxl < lyl, then lxl 1lyl'- 1 S: lyl 1lyl'- 1 =
IYI' S: lxl' + lyl'; while if lxl ~ lyl, then lxl1lyl'- 1 < lxl' s; lxl' + IYI'.
Thus (13) holds. Using (13) and the binomial theorem we now see that

lx + yl' < (lxl + IYIY

(~) lxl 1yl'-i

1

j=O

S:

But

(;)

(lxl' + Iyl').

(~) =
J

j=O

because
2'

(1

1)'

(~)

j=O

2'

1j1r-j

(~).

j=O

Consequently

lx + Yl' < 2'(1xl' + IYl').

Let f be the joint density of X and Y. Then

L
lx + Yl'i(x, y) ~ 2' ~ (lxl' + I yl')f(x, y)
x,y
x,y
= 2'E(IXI' + IYl')
= 2'"(EIXI' + EIYI') < oo.
Hence by Theorem I, (X+ Y)' has finite expectation.

It follows easily by induction that if X 1 , X 2 , , X, all have a moment

of order r, then so does X 1 + + X,.
Let X be a random variable having a finite second moment. Then the
variance of X, denoted by Var X or V(X), is defined by
Var X = E[(X - EX) 2 ] .

By expanding the right-hand side we see that

Var X= E[X 2
= EX 2

(2X)(EX)
2(EX) 2

(EX) 2 ]

(EX) 2

94

Expectation of Discrete Random Variables

In other words
(14)

One often denotes EX by 11 and Var X by u 2 The nonnegative number

u = .JVar X is called the standard deviation of X or of fx
According to our previous discussion, the mean 11 is the average value
of the random variable X . One use of the variance is as a measure of the
spread of the distribution of X about the mean. The more X tends to
deviate from its mean value 11, the larger (X - 11) 2 tends to be, and hence
the larger the variance becomes.
On the other hand, Var X = 0 if and only if X is a constant. To see
this, observe that if P(X = c) = 1 for some constant c, then EX= c and
Var X = 0. Conversely, if Var X = 0, then E[(X - EX) 2 ] = 0 and
hence P((X - EX) 2 = 0) = 1. Consequently P(X = EX) = 1.
An alternative use of the mean and variance is provided by the following
problem, which is of interest in statistics. Let X be a random variable
having a finite second moment, and suppose we want to choose the value
of a that minimizes E(X - a) 2 Such a value would provide the best fit to
X by a constant if the error were measured by the mean square deviation.
One way of solving this problem is to use calculus. Note that

E(X- a)2 = EX 2

2aEX

+ a2

If we differentiate with respect to a and set the derivative equal to zero,

we see that a = EX. Since the second derivative is positive (in fact, it
equals 2), the point corresponds to a minimum, and the minimum value is
VarX.
There is a second way of solving this problem that is also important to
understand. Note that
(X - a) 2 = [(X - 11)
= (X - 11)

+ (p - a)] 2
+ 2(X- p)(l1

- a)

+ (p

- a)2

Since E(X - 11) = 0, it follows that the cross-product term has zero
expectation and hence
(15)

E(X - a) 2 = E(X - 11) 2

= Var X

+ (Jl

(Jl - a)2

- a)2

It is now clear from (15) that E(X - a)2 is at a minimum when 11 = a,

and this minimum value is Var X.
We can often find the moments of a nonnegative integer-valued random
variable X most simply by differentiating its probability generating function ~x Suppose for simplicity that
co

.x=O

fx(x)ro < oo

4.3.

95

Moments

for some t0 > 1. Then we can regard (l>x as being defined on - t0 < t < t 0
by
IX)

(l>x(t) = ~ fx(x)~,
x=O

-t0 < t < t 0

We can differentiate (l>x(t) any number of times by differentiating the

corresponding power series term by term. In particular
IX)

(l>x(t) = ~ xfx(x)~- 1,
x=l

and
IX)

(l>x(t) = ~ x(x - 1)/x(x)tx- 2 ,

-t0 < t < t 0

x=2

By our assumptions on t 0 , we can lett = 1 in these formulas, obtaining

00

(l>x(1)

~ xfx(x)

EX

x=l

and
00

(l>i(l) = ~ x(x - 1)/x(x) = EX(X - 1).

.x=2

Thus the mean and variance of X can be obtained from (l>x by means of
the formulas
EX = (l>X(l)
and
Similar formulas, in terms of the higher derivatives of (l>x(t) at t = 1,
can be developed for the other moments of X.
We now illustrate the use of these formulas with the following examples.
Negative binomial distribution. Let X be a random
variable having a negative binomial distribution with parameters ex and p.
Find the mean and variance of X.
Example 9.

From Example 17 of Chapter 3, we know that the generating function of

Xis given by >x(t) = i'[1 - t(l - p)]- 11 Consequently,
(l>X(t) = C(p11[1 - t(l - p)]-(+ 1)(1 - p)
and
(l>.X(t)

= (a + l)atft

~ t(l - p)]-<+ 2 >(1 -

Thus

~X(l) =
and

"'c ~ p).

p) 2

96

Expectation of Discrete Random Variables

Hence, EX= ap- 1 (1 - p) and

Var X = (+ !)a
=IX

1- p

p2

e~ pr "e ~ p) - c~ pr
+

"z

In particular, if X has a geometric distribution with parameter p, then

EX= p- 1(1 - p) (as we have already seen) and Var X= p- 2 (1 - p).
Let X have a Poisson distribution
with parameter A.. Find the mean and variance of X.
Example 10.

Poisson distribution.

In Example 18 of Chapter 3 we found that ~x(t) = e;.<r-o. Thus

4>~(t)

= A.e-4<r-t>

and
4>x(t)

Consequently 4>~(1)

= ..PeJ.<t-t>.

A. and 4>i(1)

.A.

It follows immediately that

EX= ;.,
which agrees with the answer found in Example 2, and
Var X = .A.l

+ A. -

A- 2 =

A..

This shows that if X has a Poisson distribution with parameter A., then the
mean and variance of X both equal A..
4.4.

Variance of a sum

Let X and Ybe two random variables each having finite second moment.
Then X + Y has finite second moment and hence finite variance. Now
Var (X

+ Y)

= E[(X

Y) - E(X

+ Y)] 2

= E[(X - EX) + (Y - EY)] 2

= E(X - EX) 2 + E(Y - EY) 2
+ 2E[(X - EX)(Y - EY)]
= Var X+ VarY+ 2E[(X- EX)(Y- EY)].

Thus, unlike the mean, the variance of a sum of two random variables
is, in general, not the sum of the variances. The quantity
E[(X - EX)(Y- EY)]

is called the covariance of X and Y and written Cov (X, Y). Thus we have
the important formula
(16)

Var (X

+ Y)

= Var X

+ Var

+ 2 Cov (X,

Y).

4.4.

97

Variance of a sum

Now
(X- EX)(Y- EY) = XY- (Y)(EX)- X(EY)

(EX)(EY),

and hence taking expectations we see that

(17)

Cov (X, Y) = E[(X - EX)(Y - EY)] = E(XY) - (EX)(EY).

From this form, it is clear that Cov (X, Y) = 0 whenever X and Y are
independent. (Example 5 shows that the converse is false.) We see from
(16) that if X and Yare independent random variables having finite second
moments, then Var (X+ Y) = Var X+ VarY.
In particular if P(Y = c) = 1 for some constant c, then X and Yare
independent and the variance of Y equals zero ; consequently
Var (X+ c)

(18)

More generally, if Xh X2 ,
finite second moment, then
(19)

Var

= Var X+
. ,

and, in particular, if X 1 ,
(20)

Var

Var (

Var X.

X,. are n random variables each having a

Ct1 xi) ,t1 x, +

=

Var (c)

:t: j=tt

Cov (Xb Xj),

X,. are mutually independent, then

t x,) t
=

I= 1

Var X 1

I= 1

These formulas can be derived by a direct computation similar to (but

more complicated than) that used for the case n = 2, or they can be
established from the case n = 2 by induction on n.
In particular, if X 1 , . , X,. are independent random variables having a
common variance u 2 (for example, if they each had the same density), then
(21)

Var (X1

+ +

X,.)

= n Var X 1 = nu2

Another elementary but quite useful fact is that Var (aX)

We leave the verification of this fact to the reader.

= a 2 Var X.

Let X., ... , X,. ben independent

Bernoulli random variables each having the same probability p of assuming the value 1. Then (see Example 6) the sumS,. = X 1 + + X,.
is binomially distributed with parameters n and p. We have previously
shown that ES,. = np. Using (21) we find at once that
Example 11.

Binomial distribution.

Var S,.
Now X~
hence

n Var X1

= X 1 because X1 is either 0 or 1.

Thus EX~

EX1 = p and

Var X 1 = EX~ - (EX1 ) 2 = p - p 2 = p(l - p).

Consequently Var S,. = np(1 - p).

98

Expectation of Discrete Random Variables

In summary then, the mean of a binomially distributed variable is np

and its variance is np(I - p).
Hypergeometric distribution. Consider the situation in
Example 7. We now want to compute Var S,., so as to obtain the variance
Example 12.

of a hypergeometric distribution. To this end we will use Equation (19).

For the dependent indicators, Xh . . . , X,., we previously found that
P(X1 = 1)

= EX1 = ' 1
r

Since Xf = X 1 we see that

Var Xi

EXl - (EXi) = (';)

= ( ';)

(': )

(I - ';) .

Next we must compute the covariances. Assume that l ::s; i < j < n.
Now X 1Xi = 0 unless both Xi and X1 are l, so
EXiXJ

= P(X1 =

1,X1 = 1)

= (':) (': ~ :) .

Thus

(';)2
- ( ':) ( ';

~ 11 -

',)

and hence

"it
i=1

Cov (Xi, Xj)

J=i+t

't -

n(n - l) (')
r
2
r r(r - l)

It now follows from (19) that

V ar S = n r 1(r -2 r 1) - n( n- 1) r 12(r - r 1 )

"

n ( ';) ( 1 - ':) ( 1 - :

r (r - 1)

=~) .

It is interesting to compare the mean and variance for the hypergeometric distribution with those of the binomial distribution having the
same success probability p = (r 1 /r). Suppose we have a population of r
objects of which r 1 are of type one and r - r 1 are of type two. A random

99

4.5. Correlation coefficient

sample of size n is drawn from the population. Let Y denote the number
of objects of type one in the sample.
If the sampling is done with replacement then Y is binomially distributed
with parameters nand p = (r 1 /r), and hence
and
On the other hand, if the sampling is done without replacement, then Y
has a hypergeometric distribution,
and

Var Y = n ( ', )

1 - ':) ( 1 - ;

=~) .

The mean is the same in the two cases, but in sampling without replacement the variance is smaller. Intuitively, the closer n is to r the more
deterministic Y becomes when we sample without replacement. Indeed, if
n = r then the variance is zero and P(Y = r 1) = 1. But if r is large
compared to n, so that (n/r) is close to zero, the ratio of the variances
obtained in sampling with and without replacements is close to one. This
is as it should be, since for fixed n and large r there is little difference
between sampling with replacement and sampling without replacement.
4.5.

Correlation coefficient

Let X and Y be two random variables having finite nonzero variances.

One measure of the degree of dependence between the two random
variables is the correlation coefficient p(X, Y) defined by

(22)

_ (X Y) _
p - p '
-

Cov (X, Y)
(Var X) (VarY)

-r==========~

.J

These random variables are said to be uncorrelated if p = 0. Since

Cov (X, Y) = 0 if X and Yare independent, we see at once that independent random variables are uncorrelated. It is also possible for dependent
random variables to be uncorrelated, as can be seen from Example 5.
It is important for applications in statistics to know that the correlation
coefficient p is always between - I and 1, and that IPI = 1 if and only if
P(X = aY) = 1 for some constant a. These facts are easy consequences
of the following basic inequality called the Schwarz inequality.

Theorem 7 The Schwarz Inequality.

second moments. Then

Let X andY have finite

(23)

Furthermore, equality holds in (23) if and only if either P(Y = 0) = 1

or P(X = a Y) = I for some constant a.

Expectation of Discrete Random Variables

100

Proof. If P(Y = 0) = I, then P(XY = 0) = I, EXY = 0, and

EY 2 = 0; thus in this case (23) holds with equality. Also, if P(X = aY) =
1, then a simple computation will show that both sides of (23) are equal
to (a2 EY2 ) 2

We now show that (23) always holds. From the above discussion we can
assume that P( Y = 0) < I and hence EY 2 > 0. The proof is based on a
simple but clever trick. Observe that for any real number A.
0 < E(X- lY) 2

A.2 EY 2

2A.EXY

+ EX 2

This is a quadratic function of l. Since the coefficient EY 2 of l 2 is positive,

the minimum is achieved for some value of A., say A. = a, that can be found
by the usual calculus method of setting the derivative equal to zero and
solving. The answe~ is a = [E(XY)][ EY 2 ]- 1 Since the corresponding
function value is
(24)

E(X - aY)2 = EX 2

2
-

[E(XY)]

EY 2

it follows that (23) holds. If equality holds in the Schwarz inequality

(23), then from (24) we see that E(X- aY) 2 = 0, so that
P[(X- aY)

=OJ =

1.

This completes the proof.

Applying the Schwarz inequality to the random variables (X - EX)

and ( Y - EY) we see that
(E[(X- EX)(Y- EY)]) 2 < [E(X- EX) 2 ][E(Y- EY) 2 ];

that is,
[Cov (X, Y)r < (Var X)(Var Y).
Thus by the definition of p
lp(X, Y)l ~ 1.

We also see from Theorem 7 that IPI = 1 if and only if P(X = aY) = I
for some constant a.
The correlation coefficient is of limited use in probability theory. It
arises mainly in statistics and further discussion of it will be postponed to
Volume II.
4.6.

Chebyshev's inequality

Let X be a nonnegative random variable having finite expectation, and

let t be a positive real number. Define the random variable Y by setting
Y = 0 if X < t and Y = t if X > t. Then Y is a discrete random variable

4.6.

Chebyshev's inequality

101

having the two possible values 0 and t which it assumes with probabilities
P(Y = 0) = P(X < t) and P(Y = t) = P(X ~ t) respectively. Thus
EY

= tP(Y =

t)

0 P(Y

0)

= tP(Y =

t)

= tP(X ~

t).

Now clearly X ;;::: Y and hence EX > EY. Thus

EX ~ E Y = tP(X ~ t)

or
P(X

(25)

t)

~EX.
t

Quite a variety of useful inequalities can be deduced from (25). The

most important of these is the Chebyshev inequality.
Let X be a random variable with mean
Then for any real number t > 0

Chebyshev's Inequality.
Jl and finite variance a

(26)

P(IX -

Ill

(]2

t) < 2 .
t

To prove (26), we apply (25) to the nonnegative random variable

(X- Jl) 2 and the number t 2 We conclude that

Since (X - p) 2 :2:: t 2 if and only if IX - PI > t we see that (26) holds.

Chebyshev's inequality gives an upper bound in terms of Var X and~
for the probability that X deviates from its mean by more than t units.
Its virtue lies in its great generality. No assumption on the distribution of
X is made other than that it has finite variance. This inequality is the
starting point for several theoretical developments. For most distributions
that arise in practice, there are far sharper bounds for P(IX- pi > t) than
that given by Chebyshev's inequality; however, examples show that in
general the bound given by Chebyshev's inequality cannot be improved
upon (see Exercise 26).
Let X 1 , , X,. be n independent random variables having the same
distribution. These random variables may be thought of as n independent
measurements of some quantity that is distributed according to their
common distribution. In this sense we sometimes speak of the random
variables X 1 , , X,. as constituting a random sample of size n from this
distribution.
Suppose that the common distribution of these random variables has
finite mean Jl Then for n sufficiently large we would expect that their
arithmetic mean S,Jn = (X1 + + X,.)/n should be close to Jl If the
X 1 also have finite variance, then

Expectation of Discrete Random VBTiables

102

Var

(~n)

and thus Var (Sn/n) -+ 0 as n -+ oo. As discussed in Section 4.3, this

implies that as n gets large the distribution of Snfn becomes more concentrated about its mean p.. More precisely, by applying Chebyshev's
inequality to SJ n we obtain the inequality
(27)

(Isnn - I > b)
P. -

=s;; Var (S,./n) =

b2

~2 .
nb

In particular, it follows from (27) that for any b > 0

!~~ p (I ~" - p.l >

(28)

b) = o.

We may interpret (28) in the following way. The number b can be

thought of as the desired accuracy in the approximation of p. by S,jn.
Equation (28) assures us that no matter how small b may be chosen the
probability that S,/n approximates p. to within this accuracy, that is,
P(I(S,/n) - JLI < b), converges to 1 as the number of observations gets
large. This fact is called the Weak Law of Large Numbers. We have proven
the weak law only under the assumption that the common variance of the
X 1 is finite. Actually this is not necessary; all that is needed is that the X 1
have a finite mean. We state this more general result in the following
theorem. The proof will be given in Chapter 8.

Theorem 8 Weak Law of Large Numbers. Let X 1, X 2 , . , X,.

be independent random variables having a common distribution with
finite mean p. and setS, = X 1 + + Xn. Then for any b > 0
lim p (
n-+00

sn
n

Jl. l

b) = 0.

Whenever the X 1 have finite mean, the weak law holds. However, when
they also have finite variance, then (27) holds. This is a more precise
statement since it gives us an upper bound for P

(I~n p,l ~ b) in terms

-

of n. We now illustrate the use of (27) by applying it to binomially distributed random variables.
Let X 1 , X 2 , , Xn be n independent Bernoulli random variables assuming the value 1 with common probability p . Then Jl = p and u 2 =
p(l - p). Thus (27) shows that
(29)

(l ~n- PI ~ <5)

<

p(ln~ p) .

4.6.

1t3

Chebyshev's inequality

Since p(1 - p) < 1/4 ifO < p < 1 (because by the usual calculus methods
it can easily be shown that p(l - p) has its maximum value at p = 1/2),
it follows that regardless of what p may be,
(30)

(IS,.- PI >b) ~ _1 .

4nb2

Equation (29) is useful when we know the value of p, while (30) gives us a
bound on P (

~"

- p

> b) that is valid for any value of p. If p is near

1/2, (29) and (30) do not differ by much, but ifpis far from 1/2 the estimate
given by (29) may be much better. (Actually even the bounds given by
(29) are quite poor. We shall discuss another method in Chapter 7 that
yields much better estimates.)
Suppose b and e > 0 are given. We may use (29) or (30) to find a lower
bound on the number of trials needed to assure us that

Indeed, from (29) we see that this will be the case if p(1 - p)/nb 2 < e.
Solving for n we find that n > p(l - p)/eb2 If we use (30), then
n > (4eb 2 )- 1 trials will do. We stat,e again that these bounds on n given
by Chebyshev~s inequality are poor and that in fact a much smaller
number of trials may be sufficient.
As an illustration of the difference between these two estimates for n,
choose b = .1 and e = .01. Then b2 e = 10- 4 and from (30) we see that to
guarantee that

we would need n = 104 /4 = 2500 observations. Suppose, however, we

knew that p = .1. Then since p(1 - p) = .09 we see from (29) that
n > .09 x 104 = 900 observations will suffice. For p = 1/2, (29) gives
the same estimate as (30), namely 2500.
To illustrate that the Chebyshev bounds are really poor in the case of
the binomial distribution, suppose n = 100 and p = 1/2. From (29) we
then obtain
p

(I~" - .5 1 ;:: .1) ~ .25.

This should be compared with the exact value for this probability which
is .038.

104

Expectation of Discrete Random Variables

Exercises

1 Let N be a positive integer and let/be the function defined by

2x

f(x) =

N(N
{
0,

1),

= 1, 2, ... , N,

elsewhere.

Show that f is a discrete density and find its mean. Hint:

X=

x=l

N(N + 1)
2

and

fx

= N(N

x=1

2 Let X have a binomial density with parameters n

1X2N
6

= 4 and

E sin (rrX/2).
3 Let X be Poisson with parameter A. Compute the mean of (I
4

1).

p . Find

X)- 1

If X has mean 1 and Y has mean 3, what is the mean of 2X + 5 Y?

5 Suppose X and Y are two random variables such that

P(IX -

Yl 5:: M) = I

for some constant M. Show that if Y has finite expectation, then X has
finite expectation and lEX- EYI < M.
Let X be a geometrically distributed random variable and let M > 0
be an integer. Set Z = min (X, M). Compute the mean of Z.
Hint: Use Theorem 5.
Let X be a geometrically distributed random variable and let M > 0
be an integer. Set Y = max (X, M). Compute the mean of Y.
Hint: Compute P(Y < y) and then use Theorem 5.
Let X be uniformly distributed on {0, I, ... , N}. Find the mean and
variance of X by using the hint to Exercise 1.
Construct an example of a density that has a finite moment of order
r but has no higher finite moment. Hint : Consider the series
:E;'= 1 x-<r+l> and make this into a density.

X and Yare two independent random variables such that

EX = 2, EY 2 = 1, EX 2 = 1, and EY = 0. Compute Var (X2 Y).
11 Show that Var (aX) = a 2 Var X.
12 Let X be binomially distributed with parameters n and p. Use the
probability generating function of X to compute its mean and variance.
13 Let X be a nonnegative integer-valued random variable.
(a) Show that
4>x(t) = Etx,
-1 5:: t < 1,

10 Suppose
4

-1 < t < 1,

and
4>_i(t)

= EX(X-

1)tx- 2 ,

-1<t<l.

Exercises

105

(b) Use Theorem 4 to rederive the result that if X and Yare independent nonnegative integer-valued random variables, then
-l<t~l.

14 Let X and Y be two independent random variables having finite

second moments. Compute the mean and variance of 2X + 3 Y in
terms of those of X and Y.
, X,. be independent random variables having a common
density with mean p, and variance a 2 Set X = (X1 + + X,.)fn.
(a) By writing X" - X = (X" - p,) - (X - p,), show that

15 Let X1,

"

i.J (X" -

-2
X)
=

k=l

"

~(

i.J

X" .- p,)2 - n(X

- p,)2 .

k=l

(b) Conclude from (a) that

E (

(X" - X) 2 ) = (n - 1)u2

k=l

Suppose n balls are distributed at random into r boxes. Let X1 = 1 if

box i is empty and let Xi = 0 otherwise.
(a) Compute EX,.
(b) Fori :1: j, compute E(X1X 1).
(c) LetS, denote the number of empty boxes. WriteS, = X1 +
+ X, and use the result of (a) to compute ES,.
(d) Use the result of (a) and (b) to compute Var S,.
17 Suppose we have two decks of n cards, each numbered 1, ... , n. The
two decks are shuffled and the cards are matched against each other.
We say a match occurs at position i if the ith card from each deck has
the same number. Let S,. denote the number of matches.
(a) Compute ES,..
(b) Compute Var S,..
Hint: Let X1 = 1 if there is a match at position i and let X1 = 0
otherwise. Then S,. = X1 + + X,.. From the results in Chapter
2 we know that P(X1 = 1) = 1/n and that if i . j,

16

P(X,

1, X 1 = 1)

1
= -n(n -

1)

18 Consider the random variable S" introduced in Example 8. Compute

Var S".
19 Establish the following properties of covariance:
(a) Cov (X, Y) = Cov (Y, X);
(b) Cov

Ct. itt
a 1X1,

h1 Y1) =

~~ 1

a1b1 Cov (Xi, lj).

20 Let Xh X 2 , and X 3 be independent random variables having finite

positive variances uf, ai, and a~ respectively. Find the correlation

between xl - x2 and x2 + x3.

106

Expectation of Discrete Random Variables

21 Suppose X and Yare two random variables such that p(X, Y) = 1/2,
VarX = 1, and Var Y = 2. Compute Var (X- 2Y).
22 A box has 3 red balls and 2 black balls. A random sample of size 2 is

drawn without replacement. Let U be the number of red balls selected

and let V be the number of black balls selected. Compute p( U, V).
23 Suppose a box has 3 balls labeled 1, 2, and 3. Two balls are selected

without replacement from the box. Let X be the number on the first
ball and let Y be the number on the second ball. Compute Cov (X, Y)
and p(X, Y).
24 Suppose an experiment having r possible outcomes 1, 2, ... , r that

occur with probabilities p 1 , , Pr is repeated n times. Let X be the

number of times the first outcome occurs, and let Y be the number -of
times the second outcome occurs. Show that
p(X, Y) =

-J

P1P2 (1 - PtXl - P2)

by carrying out the following steps. Let / 1 = 1 if the ith trial yields
outcome l, and let Ii = 0 otherwise. Similarly, let J 1 = 1 if the ith
trial yields outcome 2, and let Ji = 0 elsewhere. Then X = / 1 +
+ I,. and Y = J 1 + + J,.. Now show the following:
(a) E(l/i) ...: o:
(b) If i #: j, E(IiJi) = PtP2
(c) E(XY)

=E

(t fiJi) + E (t
1

i~i

IiJJ)

= n(n - l)Pt P2
(d) Cov (X, Y) = -nPtP2

(e) p(X, Y) = -J--P-1-P2- - .

(1 - PtXl - P2)

r objects consists of r 1 objects of type 1, r2

objects of type 2, and r 3 objects of type 3, where r 1 + r 2 + r 3 == r.
A random sample of size n s r is selected without replacement from

25 Suppose a population of

this population. Let X denote the number of objects of type 1 in the

sample and let Y denote the number of objects of type 2. Compute
p(X, Y) by doing the following. Let /i = 1 or 0 according as the ith
element in the sample is of type 1 or not and let Ji = 1 or 0 according
as the ith element in the sample is of type 2 or not.
(a) Show that Eli = r1 /r and EJi = r2 /r.
(b) Show that for i #: j,

and that E(liJ1) = 0.

(c) Set X= / 1 + + 111, and Y = 1 1 + + J" and use (a) and (b)
to compute E(XY), Var X, and Var Y.

Exercises

107

(d) Use (c) to compute p(X, Y). Compare with the corresponding
correlation coefficient in Exercise 24 withp 1 = r 1 fr andp2 = r2 /r.
26 Let X be a random variable having density f given by

1/18,
f(x) = { 16/18,

= 1, 3,

X=

2.

Show that there is a value of~ such that P(IX- Jll > ~) = Var X/~ 2 ,
so that in general the bound given by Chebyshev's inequality cannot be
improved.
27 A bolt manufacturer knows that

5%

of his production is defective.

He gives a guarantee on his shipment of 10,000 parts by promising to
refund the money if more than a bolts are defective. How small can
the manufacturer choose a and still be assured that he need not give a
refund more than 1% of the time?

28 Let X have a Poisson density with parameter

A.. Use Chebyshev's

inequality to verify the following inequalities:

(b) P(X :2: 2.A.)
29 Let

~!.
A.

X be a nonnegative integer-valued random variable whose prob-

ability generating function ~x(t) = Etx is finite for all t and let x 0
be a positive number. By arguing as in the proof of Chebyshev's
inequality, verify the following inequalities:

0 < t
(b) P(X >

X 0)

< ~xx(t) ,
t

1;

t > I.

30 Let X have a Poisson density with parameter A. Verify the following

-inequalities:

;
;
( 2,t) < (2)A/2

(a) P X<

(b) P(X > 2.A.)

~ (~f

Hint: Use calculus to minimize the right sides of the inequalities in

Exercise 29. These inequalities are much sharper, especially for large
values of A., than are those given in Exercise 28.
Exercises 31-36 develop and apply the notions of conditional
density and conditional expectation.
Let X and Y be discrete random variables. The conditional density
fy,x(Y I x) of Y given X= xis defined by
r

JYIX

I X ) . -_ {P(Y
= y I X = x),
0
'

if P(X = x) > 0,
elsewhere.

108

Expectation of Discrete Random Variables

For any x such that P(X = x) > 0 it follows that fr 1x(Y I x) is a

density in y. Example 14(d) of Chapter 3 can be interpreted as saying
that if X and Y are independent and geometrically distributed with
parameter p, then, for z > 0, the conditional density of Y given
X+ Y = z is the uniform density on {0, I, . .. , z}.
Let Y have finite expectation. The conditional expectation of Y given
X = x is defined as the mean of the conditional density of Y given
X = x, i.e., as
E[YI X= x] = ~ Yfrtx(Y I x).
)I

31 Verify the following properties of the conditional density and con-

ditional expectation:
(a) fr(y)

~fx(x)fl'txCY I x);

(b) EY = ~fx(x)E[Y I X = x].

JC

JC

be independent random variables each having a geomd,ric

density with parameter p. Find E[ Y I X + Y = z] where z is a nonnegative integer. Hint: Use Example 14(d) and Exercise 8.
33 Let X and Y be two independent Poisson distributed random variables
having parameters A. 1 and A.2 respectively. Compute E[ Y I X + Y = z]
where z is a nonnegative integer. Hint: Use the result of Exercise 35
of Chapter 3.
32 Let X and Y

be a nonnegative integer-valued random variable. Let { Y,.},

n > 0, be random variables each of which has finite expectation and is
independent of N. Show that

34 Let N

E[YN
35 Let {X,.}, n

IN

n]

= EY,..

> 1, be independent random variables having a common

finite mean Jl and variance u 2 Set S0 = 0 and S,. = X 1 + + X,.,

n > 1. Let N be a nonnegative integer-valued random variable having
finite mean and variance, and suppose that N is independent of all
random variables defined in terms of {X,.}, n ~ I. Then SN has finite
mean and variance. Show that
ESN = JlEN,

and

ES~ = u 2 EN

Var SN = u 2 EN

+ Jl2 EN2 ,

+ Jl2 Var N.

Hint: Use Exercises 3l(b) and 34.

36 Obtain the results of Exercise 35 by differentiating the probability

generating function of SN, found in Example 19 of Chapter 3, and

setting t = I.

Continuous Random
Variables

In Chapter 3 we considered discrete random variables and their densities, e.g.,

binomial, hypergeometric, and Poisson. In applications, these random variables
typically denote the number of objects of a certain type, such as the number of red
balls drawn in a random sample of size n with or without replacement or the
number of calls into a telephone exchange in one minute.
There are many situations, both theoretical and applied, in which the natural
random variables to consider are "continuous" rather than discrete. Tentatively
we can define a continuous random variable X on a probability space !l as a
function X(w), wEn, such that
P({w

I X(w) = x}) = 0,

-00

<

<

00,

that is, such that X takes on any specific value x with probability zero.
It is easy to think of examples of continuous random variables. As a first
illustration, consider a probabilistic model for the decay times of a finite number
of radioactive particles. Let T be the random variable denoting the time until the
first particle decays. Then T would be a continuous random variable, for the
probability is zero that the first decay occurs exactly at any specific time (e.g.,
T = 2.()()()() .. .. seconds). As a second illustration, consider the experiment of
choosing a point at random from a subset S of Euclidean n-space having finite
nonzero n-dimensional volume (recall the discussion of this in Chapter 1). Let X
be the random variable denoting the first coordinate of the point chosen. It is clear
that X will take on any specific value with probability zero. Suppose, for example,
that n = 2 and S is a disk in the plane centered at the origin and having unit
radius. Then the set of points inS having first coordinate zero is a line segment in
the plane. Any such line segment has area zero and hence probability zero.
Generally speaking, random variables denoting measurements of such physical
quantities as spatial coordinates, weight, time, temperature, and voltage are most
conveniently described as continuous random variables. Random variables which
count objects or events are clear examples of discrete random variables.
There are cases, however, in which either discrete or continuous formulations
could be appropriate. Thus, although we would normally consider measurement of
109

110

Continuous Random Variables

length as a continuous random variable, we could consider the measurement as

being rounded off to a certain number of decimal places and therefore as being a
discrete random variable.
6.1.

Random variables and their distribution functions

In applications, a random variable denotes a numerical quantity defined

in terms of the outcome of a random experiment. Mathematically,
however, a random variable X is a real-valued function defined on a
probability space. Naturally, we want P(X < x) to be well defined for
every real number x. In other words, if (!l, .91, P) is the probability space
on which X is defined, we want
{w I X(w) < x}
to be an event (i.e., a member of d). This leads to the following definitions.

Definition 1 A random variable X on a probability space

(Q, .91, P) is a real-valued function X(w), wEn. such that for
- oo < x < oo, {w I X(w) < x} is an event.
Definition 2
is the function

The distribution function F of a random variable X

F(x) = P(X

x),

- 00

<X<

00.

The distribution function is useful in computing various probabilities

associated with the random variable X. An example of this is the formula
(1)

P(a < X < b)

= F(b)

- F(a),

a< b.

In order to verify (1), set A = {co I X(w) ~ a} and B = {w I X(w) < b}.
Then A ; Band, by the definition of a random variable, both A and B
are events. Hence {w I a < X < b} = B t1 Ac is an event and (1) is a
special case of the fact proven in Section 1.3 that if A ; B, then

P(B

t1

Ac)

P(B) - P(A).

Consider the experiment of choosing a point at random

from the disk of radius R in the plane centered at the origin. To make the
experiment more interesting, we can think of it as the result of throwing
dart at a disk-shaped target. Associated with this experiment is the
uniform probability space described in Section 1.2. Let X be the random
variable denoting the distance of the point chosen from the origin. The
distribution function of X is easily computed. If 0 < x < R, the event
{co I X(w) < x} is the disk in the plane, of radius x centered at the origin.
Its area is nx2 Thus by the definition of a uniform probability space,
Example 1.

5.1.

Random va1iablss and thsi1 distlibution functions

111

0 <X< R.
If x < 0, then P(X < x) = 0. If x > R, then P(X < x) = I. Thus the
distribution function F of the random variable X is given by

X< 0,
0 s X oe::: R,
X> R.

(2)

The graph ofF is given in Figure 1. It follows from Formulas (I) and (2)
that if 0 < a < b ~ R, then
P(a

< X < b)

= F(b)

- F(a)

b2- a2
R2

R
Figure 1

Consider a probability model for the decay times of a

finite number of radioactive particles. Let X denote the time to decay for
a specific particle. Find the distribution function of X.
As we saw in Section 1.1, for a suitable positive value for A.,
Example 2 .

0 <a< b < oo.

Since X takes on only positive values, P(X < x) = 0 for x < 0 and, in
particular, P(X < 0) = 0. For 0 < x < oo,
P(X

< x)

= P(X

< 0) + P(O < X < x)

= P(O <X~ x)

=1-

e-

lx

Thus X has the distribution function F given by

(3)

X~

0,

X> 0.

112

Continuous Random Variables

Of course, discrete random variables also have distribution functions,

two of which were computed in Examples 10 and 11 of Chapter 3.
Let X have a binomial distribution with parameters
n = 2 and p = 1/2. Then f(O) = 1/4, f(l) = 1/2, and /(2) = 1/4.
Consequently
0,
X< 0,
0 <X < 1,
F(x) =
1 < X < 2,
(
Example 3.

~~!:
I,

2 ~X.

The graph of this distribution function is given in Figur,e 2.

1

*..----1

2
Figure 2

Not all functions can

arise as distribution functions, for the latter must satisfy certain conditions.
Let X be a random variable and let F be its distribution function. Then

5.1.1.

Properties of distribution functions.

(i) 0 < F(x) < 1 for all x.

(ii) F is a nondecreasing function of x.
Property (i) follows immediately from the defining property F(x) =
P(X < x). To see that (ii) holds we need only note that if x < y, then
F(y) - F(x) = P(x < X < y) ;;;:: 0.

A function f is said to have a right-hand (left-hand) limit L at x ifI

f(x + h) -+ Lash -+ 0 when h is restricted to positive (negative) values.
The right-hand and left-hand limits, when they exist, are denoted respectively by f(x+) andf(x- ). It is not hard to show that if/is bounded and
either nondecreasing or nonincreasing, then f(x+) and f(x-) exist for
all x. Under the same conditions, f has limits f (- oo) as x -+ - oo and
f( + oo) as x -+ + oo.
From properties (i) and (ii) and the discussion in the preceding paragraph, it follows that the distribution function F has limits F(x+) and
F(x-) for all x as well as the limits F(- oo) and F( + oo).

5. 1.

Random variables and their distribution functions

113

(iii) F(- oo) = 0 and F( + oo) = 1.

(iv) F(x+) = F(x) for all x .
In order to evaluate F(- oo) and F( + co) we need only find the limits of
F(n) as n -+ -co and n -+ + oo. (This is because F is nondecreasing.)
Set
B11 = {c:o I X(c:o) =:; n}.

n ,.= 0

-oo

+oo

U B,. = n.

and

11=0

11=0

It now follows from the results of Theorem 1 of Chapter 1 that

lim P(B11)

= P(0) = 0

lim P(B,) ::: P(Q) = 1.

and

11-+-oo

11 ...

+ 00

Since F(n) = P(X =:; n) = P(B11), we have that

= lim

F(- co)

lim P(B,)

F(n) =

11-+-oo

= 0

11-+-oo

and similarly that F( + oo) = 1.

Property (iv) states that F is a right-continuous function and
(4)

= P(X =:;

x),

-00

< X<

00.

F(x-) = P(X < x),

-00

<X<

00.

F(x+)

A closely related result is

(5)

The proofs of (4) and (5) are similar to the proof of (iii). To prove (4),
for example, we need only show that F(x + 1/n) --+ P(X < x) as
n -+ + oo. This can be done by setting
B,

= (ro I X(ro) ..c::: x + ~},

noting that n11 B, = {ro I X(ro) < x} and repeating the argument of(iii).
From (4) and (5) we see immediately that
(6)

F(x+) - F(x-)

P(X

x),

-oo<x<oo.

This formula states that if P(X = x) > 0, then F has a jump at x of

magnitude P(X = x). If P(X = x) = 0, then F is continuous at x. We
recall from the introduction to this chapter the concept of a continuous
random variable.
Definition 3 A random variable X is called a continuous random
variable if
P(X = x) = 0,
-co<x<co.

Continuous Random Variables

114

We now see that X is a continuous random variable if and only if its

distribution function F is continuous at every x, that is, F is a continuous
function. If X is a continuous random variable, then in addition to (1)
we have that
(7)

P(a

< X < b)

= P(a ~ X ~ b) = P(a

= F(b)

< X < b)

- F(a),

so that < and < can be used indiscriminately in this context. The
various properties of a distribution function are illustrated in Figure 3.
(Note that the random variable having this distribution function would be
neither discrete nor continuous.)
F(+co)-1

F (x+ ) - f (x- )= P (X- x)

f(x-)

f(-oo)-0
X

Y-o
Figure 3

Consider the random variable X defined in Example 1. From Formula

(2) or Figure 1 we see that its distribution function is continuous. Thus X
is a continuous random variable. Similarly it is clear from (3) that the
random variable from Example 2 is a continuous random variable.
Most random variables arising in practical applications are either discrete or continuous. There are some exceptions. Consider Example 2.
In this example X represents the time to decay of a specific particle.
If the experiment lasts only a specified time, say until time t0 > 0, and the
particle has not decayed by this time, then its true decay time X will not be
observed. One possible way out of this difficulty is to define a new random
variable Y as follows
Y(ro) = {X(w)
to

if X(w) < t0 ,
if X(w) > t 0

5.2.

115

Densities of continuous rBndom vBriBbles

Thus Y is the decay time, if this time is observed (i.e., is less than or equal
to t 0 ), and otherwise Y = t0 The distribution function Fr of Y is given by
Fr(Y) =

< 0,
0 < y < t0 ,

0,
1 - e-;..,,

1,

y >to.

The distribution function has a jump aty = t 0 of magnitude e-;.ro. Thus it

is clear that the random variable Y we have constructed is neither discrete
nor continuous.
We have defined distribution functions in terms of random variables.
They can be defined directly.
Definition 4 A distribution function is any function F satisfying
properties (i)-{iv); that is,
(i) 0

F(x)

1 for all x,

(ii) F is a nondecreasing function of x,

(iii) F(- oo) = 0 and F( + oo)

(iv) F(x+)

= F(x) for

1,

all x.

In more advanced books it is shown that ifF is a distribution function,

there is necessarily a probability space and a random variable X defined on
that space such that F is the distribution function of X .
5.2.

Densities of continuous random variables

In practice, continuous distribution functions are usually defined m

terms of density functions.
Definition 5 A density function (with respect to integration)
is a nonnegative function f such that

J~oo f(x) dx

= 1.

Note that iff is a density function, then the function F defined by

(8)

F(x) =

J~oo f(y) dy,

-00

<X <

00,

is a continuous function satisfying properties (i)-{iv) of Section 5.1.1.

Thus (8) defines a continuous distribution function. We say that this
distribution function has density f. It is possible but difficult to construct
examples of continuous distribution functions that do not have densities.
Those that do have densities are called absolutely continuous distribution
functions.

116

Continuous Random Variables

If X is a continuous random variable having F as its distribution function, where F is given by (8), then f is also called the density of X . In the
sequel we will use "density function" to refer to either discrete density
functions or density functions with respect to integration. It should be
clear from the context which type of density function is under consideration. For example, the phrase "let X be a continuous random variable
having density f" necessarily implies that f is a density function with
respect to integration.
It follows from (I) and (8) that if X is a continuous random variable
having density f, then
(9)

P(a < X < b) =

L" f(x) dx,

a < b,

or somewhat more generally, that

P(X e A) =

(10)

JA f(x) dx

if A is a finite or countably infinite union of disjoint intervals. Thus

P(X e A) can be represented as the area under the curve f as x ranges
over the set A (see Figure 4).

Figure 4

In most applications, the easiest way to compute densities of continuous

random variables is to differentiate (8) and obtain

(11)

f(x)

= F'(x),

-oo < x< oo.

Strictly speaking, (11) holds at all points x where f is continuous.

Let X be the random variable from Example 1 having the
distribution function F given by (2). Then
Example 4.

0,
(12)

F'(x)

2x/R
0,

X< 0,
O<x<R,
X> R.

At x = R the function Fis not differentiable. If, however, we define/by

f(x) = F'(x), x -:/; R. and f(R) = 0, then this/will be a density for F.

5.2.

117

Densities of continuous random variables

We note that (8) does not definefuniquely since we can always change
the value of a function at a finite number of points without changing the
integral of the function over intervals. One typical way to define f is by
setting f(x) = F'(x) whenever F'(x) exists and f(x) = 0 otherwise. This
defines a density ofF provided that F is everywhere continuous and that F'
exists and is continuous at all but a finite number of points.
There are other ways to derive or verify formulas for the density of a
continuous distribution function F. Given a density function f we can
show that f is a density function of F by verifying that (8) holds. Alternatively, we can reverse this process and show that F can be written in the
form (8) for some nonnegative function f. Then f is necessarily a density
function of F. These methods, essentially equivalent to each other, are
usually more complicated than is differentiation. However, they are
rigorous and avoid special consideration of points wher,e F'(x) fails to
exist.
We will illustrate these methods in our first example of the following
subsection.
Let X be a continuous random
variable having density f. We will discuss methods for finding the density
of a random variable Y which is a function of X.

5.2.1 .

Change of variable formulas.

Let X be a continuous random variable having density f.

Find the density of the random variable Y = X 2
Example 5.

To solve this problem we first let F and G denote the respective distribution functions of X and Y. Then G(y) = 0 for y < 0. For y > 0
G(y)

P(Y < y)

= P(- ..}y
=

P(X 2 < y)

< X < ..}y)

F(..}y) - F( - .Jy)

and by differentiation we see that

G '(y)

lr

(F'(Jy) + F'( - Jy))

2vy
=

Thus Y
(13)

lr (f(..}y)
2v y

+ f( -Jy)).

= X 2 has density g given by

u<Y> = {~~:; <t<~Y> + t< -~Y

for y > 0,
for y

0.

118

Continuous Random Variables

Although (13) is valid in general, our derivation depended on differentiation, which may not be valid at all points. To give an elementary but
completely rigorous proof of (13), we can define g by the right side of (13)
and write for x > 0

% g(y) dy

f.%

- oo

1
, (J(JY)

2v y

z = J; (so that dz = dyJ2Jy), we

By making the change of variable

obtain

I:w g(y) dy

+ f( -Jy)) dy.

(f(z)

+ /( -

z)) dz

I.r"

f(z) dz

-./~

= F(J~) - F( -J~) = G(x),

so that g is indeed a density of G.
Hereafter we will freely use differentiation to establish formulas such
as (13), knowing that we could if necessary provide alternative derivations
via integration.
Let us now use (13) to find the density of X 2 , where X is the random
variable defined in Example 1. The density of X was found in Example 4
to be f(x) = 2xfR 2 for 0 < x < R and f(x) = 0 elsewhere. Thus by
(13), X 2 has density g given by
1

g(y)

= 2)y

2Jy
R2

1
R2'

and g(y) = 0 elsewhere. This density is a uniform density on (0, R2 )

according to the following.
Definition 6 Let a and b be constants with a < b. The uniform
density on the interval (a, b) is the density f defined by

for a < x < b,

elsewhere.

(14)

The distribution function corresponding to (14) is given by

(15)

F(x)

0,

x <a,

(x - a)j(b - a),

a ::5: x ::5: b,

I,

X> b.

It is not difficult to find other examples of uniformly distributed random

variables. If a well-balanced dial is spun around and comes to rest after a
large number of revolutions, it is reasonable to assume that the angle of

5.2.

119

Densities of continuous random variables

the dial after it stops moving (suitably defined in radians) is uniformly

distributed on ( -n, 1r) or, equivalently, on {0, 2n). In applications of
probability theory to numerical analysis, it is often assumed that the
rounding error caused by dropping all digits more than n places beyond
the decimal point is uniformly distributed on (0, 10-").
Let X be uniformly distributed on (0, 1). Find the density
of Y = - A.- log (1 - X) for A. > 0.
Example 6.

Let G denote the distribution function of Y. We observe first that Y is a

positive random variable and consequently G(y) = 0 for y < 0. For
y > Owehave

G(y)

P(Y ~ y)

P(-A.- 1 log (1 -X)< y)

= P(log (1

- X) ~ - A.y)

= P(l - X > e-;.')

= P(X ~ 1 - e-;.Y)

Hence G'(y) = A.e-;.' for y > 0 and G'(y) = 0 for y < 0. The density of
Y is therefore given by .
y > 0,
(16)
g(y) = {A.e-;.',
0,
y ~ 0.
This density is called the exponential density with parameter A. and will be
discussed further in the next section.
The above example is a special case of problems that can be solved by
means of the following theorem.

Theorem 1 Let <p be a differentiable strictly increasing or strictly

decreasing function on an interval I, and let cp(l) de_note the range of q>
and q> - 1 the inverse function to <p. Let X be a continuous random
variable having density f such that f(x) = 0 for x /. Then Y = q>(X)
has density g given by g(y) = 0 for y cp(I) and
(17)

g(y) =

/(,-'(y))

I~ "'-'(y)l

q>(l).

It is somewhat more suggestive to write (17) in the equivalent form

(18)

g(y) = /(x)

~~;I

'

cp(J)

and

(or alternatively g(y)ldyl = f(x)ldxl).

In order to derive (17), let F and G denote the respective distribution
functions of X arid Y. Suppose first that q> is strictly increasing (i.e.,

Continuous Random Va!isb/es

120

<p(x 1 ) < q>(x2) if X 1 < x 2 , x 1 e I and x 2 e /). Then <p - 1 is strictly

increasing on rp(l) and for y e rp(J),
G(y) = P(Y S y)
= P(rp(X) ~ y)

= P(X < rp - 1 (y))

=

F(rp-1(y)).

Thus by the chain rule for differentiation,

G'(y) =

~ F(rp- 1(y))

dy

F'(rp-1(y)) _!!_ <p-(y)

dy

:y

= f(cp-(y))

rp-1(y).

Now

.!! (/) -l(y) =

dy

I!!_
(/) -1(y)l
dy

because rp- 1 is strictly increasing so that (17) holds. Suppose next that rp
is strictly decreasing on 1. Then <p- 1 is strictly decreasing on rp(J), and
for y e <p(/)
G(y) = P( Y =:;; y)
= P(<p(X)
=

< y)

P(X > rp- 1(y))

= 1 - F(<p- 1(y)).

Thus
G '(y) = - F'(rp -1(y))

_!!_ cp -1(y)
dy

= f(rp-1(y))

(- :y cp-l(y)).

Now
_.!!._qJ-1(y) =
dy

l~rp-1(y)l
dy

because rp- 1 is strictly decreasing. Therefore in either case we see that G

has the density g given by (17).
I
Let X be a random variable having an exponential density
with parameter A.. Find the density of Y = X 11', where f3 ::f: 0.
Example 7.

5.2.

Densities of continuous random variables

121

According to the definition given in the previous example, X has the

density f given by f(x) = A.e-;..x for x > 0 and f(x) = 0 for x < 0. The
above theorem is applicable with cp(x) = x 1 1P, x > 0. The equation
y = x 11P has solution x = y which yields dxfdy = py- 1 Thus by (18),
Y has density g given by

> 0,
y !5: 0.

Let X be a continuous random variable having density f

and let a and b be constants such that b =F 0. Then by Theorem I, the
random variable Y = a + bX has density given by
Example 8.

(19)

ibi1 1 (Y -b a) '

g(y) =

-oo <

< oo.

As an illustration of this formula, let X be the random variable defined in

Example 1. In Example 4 we found its density function f to be given by
f(x) = 2x/R2 for 0 < x < R and f(x) = 0 elsewhere. Consider the
random variable Y = X /R and let g denote its density. Then by Formula
(19) with a = 0 and b = l j R,

= 2y,

g(y) = Rf(Ry)

and g(y)

0 < y < 1,

= 0 elsewhere.

The reader may prefer to derive formulas such as those of Examples 7

and 8 by using the direct method of Example 6 instead of Theorem I .
As we have seen in the above examples, we can construct density functions by considering functions of random variables. There is another
simple way of constructing density functions. Let g be any nonnegative
function such that
0 <

J~> g(x) dx

< oo.

Then g can always be normalized to yield a density function f = c- 1g,

where c is the constant
c =

J~> g(x) dx.

The following examples illustrate this method.

Example 9.

Let g(x)

where. Then
c =

= x(l

t x(l -

- x), 0 < x

x) dx =

1, and g(x) = 0 else-

(x2- - -x3) It =-1

2

Continuous Random Variables

122

and f = c- 1g is given by f(x) = 6x(l - x), 0 S x S 1, and f(x) ;:; 0

elsewhere. The corresponding distribution function is given by F(x) = 0
for x < 0, F(x) = 3x2 - 2x3 for 0 < x s 1, and F(x) = 1 for x > 1.
Let g(x) = 1/(1 + x 2 ), - oo < x < oo. From calculus
we know that the indefinite integral of 1/(1 + x 2 ) is arctan x. Thus
Example 10.

c=

co

dx

-co

=arctanx

leo

7t

=--

-co

Consequently f = c- 1g is given by

+ x2) '

f(x) = n(l

-oo<x<oo.

This density is known as the Cauchy density. The corresponding distribution function is given by
1
F(x) = -

+ -1 arctan x,

-oo<x< ,oo.

7t

For an illustration of a Cauchy distributed random variable we have the

following:
Let X denote the tangent of an angle (measured in radians)
chosen at random from ( -n/2, n/2). Find the distribution of X.
Example 11.

In solving this problem we will let E> be the random variable denoting
the angle chosen measured in radians. Now X = tan 0 and hence (see
Figure 5) for - oo < x < oo,
P(X

x)

= P(tan 0 s
=

x)

P(- ~ < E> :s;; arctanx)

=~

(arctan x - ( -

~))

= -1 + -1 arctan x.
2

7t

Thus X has the Cauchy distribution.

Figure 5

5.2.

Densi~s

of continuous random variables

123

We will close this section by discussing

symmetric densities and symmetric random variables. A density function
f is called symmetric iff( -x) = f(x) for all x. The Cauchy density and
the uniform density on (-a, a) are both symmetric. A random variable X
is said to be symmetric if X and -X have the same distribution function.
The next result shows that these two concepts of symmetry are very closely
related.
5.2.2.

Symmetric densities.

Theorem 2 Let X be a random variable that has a density.

Then f has a symmetric density if and only if X is a symmetric random

variable.
Proof. We will prove this result for continuous random variables.
The proof for discrete random variables is similar. In our proof we will
use the fact that for any integrable function/
-oo < x < oo.
Suppose first that X has a symmetric density f Then
P(-X

= P(X >

x)

-x)

f~xj(y) dy

= J:<X) /(- y) dy

= J~<X) f(y) dy
= P(X ~ x),

so that X and -X have the same distribution function.

Suppose conversely that X and -X have a common density g. Define f
by f(x) = (g(x) + g( -x))/2. Then f is clearly a symmetric density
function. Also

J~<X) f(y) dy = 1/2 J~<X) g(y) dy + 1/2 J~<X) g(- y) dy

- 1/2

J:oo g(y) dy + 1/2 f~x g(y) dy

- l/2[P(X

~ x)]

l/2[P(-X > -x)]

= P(X ~ x).

Thus X has the sytn.metric density f, as desired.

124

Continuous Random Variables

If a continuous distribution function F has a symmetric density f, then

F(O) = 1/2. The values ofF for negative x's can be calculated from the
values ofF for positive x's. For

F( -x) =
=

J~: f(y) dy
LCX) I<- y) dy

= LCX) f(y) dy
=

J~CX) f(y) dy - J~CX) f(y) dy

and hence
(20)

F( -x) = 1 - F(x),

- OO <X<OO.

For this reason, when tables of such a distribution function are constructed, usually only nonnegative values of x are presented.
5.3.

Normal, exponential, and gamma densities

In this section we will discuss three of the most important families of

density functions in probability theory and statistics.
2

Let g(x) = e-" ' 2 , - oo < x < oo. In

order to normalize g to make it a density we need to evaluate the constant

5.3.1.

Normal densities.

There is no simple formula for the indefinite integral of e-"212 The easiest
way to evaluate c is by a very special trick in which we write c as a twodimensional integral and introduce polar coordinates. To be specific

c2

= f oo

e-:x2f2 dx

= 21t fooo

re-r2f2 dr

= - 21te-r2f21 ~

= 27t.

foo
- oo

- QO

e-y2f2 dy

5.3. Normal, exponential, and gamma densities

125

Thus c = ~27t and the normalized form of g is given by

f(x) =

(2n)-lf2e-xlf 2,

-00

<

<

OC>.

We also record the formula

(21)

e-:xlf 2

(lO

dx =

~2n.

-QO

The density just derived is called the standard normal density and is
usually denoted by cp, so that
_
cp(x ) -

(22)

..j2n e

-x:Zj 2
,.

-OC>

< X<

00.

The standard normal density is clearly symmetric. The distribution

function of lfJ is denoted by <1>. There is no simple formula for <I> so it must
be evaluated numerically. Computer routines and tables such as Table I
at the back of this book are available for computing <1>. Since cp is symmetric, (20) is applicable and

- oo < x < oo.

<I>(- x) = 1 - <l>(x),

(23)

Let X be a random variable having the standard normal density cp and

let Y = 11 + uX, where u > 0. Then by Formula (19), Y has the density
g given by
-(y-p.)lf2al
1
-00 < y < 00.
g( y ) = ;- e
'
av27t
This density is called the normal density with mean 11 and variance u 2 and
is denoted by n(Jl, u 2 ) or n(y; Jl, u 2 ), - oo < y < oo. Thus
(24)

n(y;

J,4,

u2) =

1
u~

v'

e-<11-l'):t/2

lip
u

(y - ~-'), -

00

<y <

00

Since we have not yet defined moments of continuous random variables,

we should temporarily think of p and u 2 as the two parameters of the
family of normal densities. The corresponding distribution function can be
calculated in terms of <I>, for
P(Y

y)

P(JL +aX < y)

= P

(x ~ Y ~ JL)

It follows that if Y is distributed as n(Jl, a 2) and a < b, then

(25)

P(a < Y < b) =

~ (b ~

11) - <I>

(a ~ ll) .

126

Continuous Random Variables

= 0 and b =

For example, let Y be distributed as n(l, 4) and let a

find from Table I that
P(O < Y < 3)

= <D(l)

~( -1/2)

= <ll{l)

3. We

- (1 - <D{l/2))

= .8413 - .3085
= .5328.

If a random variable Y is distributed as n(Jl, a 2), then the random variable

a + bY, b =1= 0, is distributed as n(a + bp,, b2 a 2 ). This is a direct applica-

Jl

= (a

bp,)

tion of (19). Alternatively, we can write Y

standard normal distribution. Then

+ b(Jl + uX)
which is distributed as n(a + bJt, b 2 u 2 ).
a

bY= a

uX, where X has the

+ buX,

Normally distributed random variables occur very often in practical

applications. Maxwell's Law in physics asserts that under appropriate
conditions the components of the velocity of a molecule of gas will be
randomly distributed according to a normal density n(O, u 2 ), where u 2
depends on certain physical quantities. In most applications, however, the
random variables of interest will have a distribution function that is only
approximately normal. For example, measurement errors in physical
experiments, variability of outputs from industrial production lines, and
biological variability (e.g., those of height and weight) have been found
empirically to have approximately normal distributions. It has also been
found, both empirically and theoretically, that random fluctuations which
result from a combination of many unrelated causes, each individually
insignificant, tend to be approximately normally distributed. Theoretical
results in this direction are known as "central limit theorems" and have
developed into one of the major research topics in probability theory.
One such central limit theorem will be discussed in Chapter 7 and proved
in Chapter 8. The importance of normal distributions arises also from
their nice theoretical properties. An example is the property that the sum
of independent normally distributed random variables is itself normally
distributed. This will be proved in Chapter 6. In Volume II we will see
that normal distributions also play a fundamental role in theoretical and
applied statistics.
The exponential density with parameter
l was defined in Section 5.2. It is given by

6.3.2.

(26)

Exponential densities.

f(x) =

{le-lx,
0,

X~

0,

X< 0.

5.3.

127

Norms/, exponential, and gamma densities

The corresponding distribution function is

F(x) = { 1 - e-A..x,

(27)

0,

X~

0,
X< 0.

From the discussion in Chapter 1 and in Example 2 of this chapter we see

that exponentially distributed random variables are useful in studying
decay times of radioactive particles. They are also useful in developing
models involving many other waiting times, such as the time until a piece
of equipment fails, the time it takes to complete a job, or the time it
takes to get a new customer. Exponentially distributed random variables
are also of theoretical importance, as can be seen by studying Poisson
processes (see Chapter 9) or continuous time Markov chains (see Volume
Ill).
An important property of exponentially distributed random variables is
that if X is such a variable, then
(28)

P(X

>a + b)

= P(X

> a)P(X > b),

a> 0 and b

0.

(This formula is similar to the one obtained in Chapter 3 for geometrically

distributed random variables.) In order to see that (28) holds, let A. denote
the parameter of the exponential distribution of X. Then by (27)
P(X > a)P(X > b) = e-J.,e-Ab

= P(X >a+ b).

A more suggestive but equivalent form of (28) is

(29)

P(X > a

+ b IX >

a) = P(X > b),

a> 0 and b > 0.

Think of X as -the time it takes a piece of equipment to fail after it is

installed. Then (29) states that, conditioned on there having been no
failure by time a, the probability of no failure in the next b units of time is
equal to the unconditioned probability of no failure during the first b
units of time. This implies that the aging of the piece of equipment neither
increases nor decreases its probability of failing in a given length of time.
That (28) or (29) characterizes the family of exponential distributions is
shown by the following result.

Tlreorem 3 Let X be a random variable such that (28) holds.

Then either P(X > 0) = 0 or X is exponentially distributed.
If P(X > 0) = 0, then (28) holds trivially. Suppose (28)
holds and P(X > 0) :#= 0. Then by (28) with a = b = 0 we see that
P(X > 0) = 1, so that X is a positive random variable. Let F
Proof.

Continuous Random Vsrisb/es

128

denote the distribution function of X and define G by G(x) ! : 1 - F(x) =

P(X > x).
Then G is a right-continuous, nonincreasing function,
G(O) = 1, G( + oo) = 0, and by (28)
G(a

a > 0

b) = G(a)G(b),

and b > 0.

It follows that if c > 0 and m and n are positive integers, then

(30)

G(nc)

= (G(c))"

and

G(c) = (G(cfm))'".

We claim next that 0 < G(l) < 1. For if G(1) = 1, then G(n) =
(G(l))" = 1, which contradicts G( + oo) = 0. If G(l) = 0, then G(l/m) = 0
and by right-continuity, G(O) = 0, another contradiction.
Since 0 < G(1) < 1, we can write G(1) = e-:A where 0 < ). < oo. It
follows from (30) that if m is a positive integer, then G(lfm) = e-:AJm. A
second application of (30) yields that if m and n are positive integers, then
G(nfm) = e-;."1'". In other words G(y) = e-J.y holds for all positive
rational numbers y . By right-continuity it follows that G(y) = e-J.y for
all y > 0. This implies that F = 1 - G is the exponential distribution
function with parameter A..
I
Before defining gamma densities in general
we will first consider an example in which they arise naturally.

5.3.3.

Gamma densities.

Let X be a random variable having the normal density

Find the density of the random variable Y = X 2

Example 12.

n(O, u

).

In solving this problem we note first that the density of X is

/",(x) = _ 1_
_e -x2{2t12,
1
cry 2n

-co < x <co.

By Formula (13), Y has density g given by g(y)

g(y) =

1
r (/(../y)
2~y

+ /( -.J y)),

= 0 for y
y

0 and

> 0.

This implies that

(31)

g(y) =

1
u../2ny

e-y/ 2t12'

y > 0.

In order to define gamma densities in general, we first consider functions

g of the form
X> 0,
X~ 0.
Here we require ex > 0 and ). > 0 in order that g be integrable. The
density in (31) corresponds to the special case ex = 1/2 and). = 1/2a2 In
normalizing g to make it a density we must evaluate

5.3.

Normal, exponential, and gamma densities

129

Make the change of variable y = A.x. Then

c = _;_
A.

fJ yx-1e-y dy.

Jo

There is no simple formula for the last integral. Instead it is used to define
a function called the gamma function and denoted by r. Thus
c

= A_
-I r(a),

where

fo(() X

r(a) =

(32)

11

e-x

dx,

a> 0.

The normalized function is called the gamma density with parameters a

and A. and is denoted by r(a, A.) or r(x; a, A.). We see that

(33)

- A_ x -1 e -Ax,
r(x;a,A.) = { ~(ex)

X> 0,

X< 0.
'
We also record the following formula, which will prove to be useful:

(f. )

(34)

r(cx)
x c:r-1 e -).x d x=--.
).

The exponential densities are special cases of gamma densities. Specifically, the exponential density with parameter A. is the same as the gamma
density r(l, A.). The density given by (31) was also seen to be a gamma
density with parameters a = 1/2 and A. = If2u2 In other words, if X
has the normal density n(O, u 2 ), then X 2 has the gamma density r(I/2, l/2u2).
By equating (31) and (33) with a = I /2 and A. = 1/2(/ 2 we obtain the useful
fact that

r(l/2)

(35)

= .J;.

An important property of the gamma function is

r(a

(36)

1) = ar(a),

a> 0.

This formula follows from (32) by a simple application of integration by

parts. To be specific

r(a

1) =

fo(() xa.e-x dx

-xe-x~~ +

= ar(a).

fo(() cxx-le-x dx

Continuous Random Variables

130

Since r(t) = 1 it follows easily from (36) that if n is a positive integer,

r(n)

(37)

(n - 1)!.

It also follows from (35), (36) and some simplifications that if n is an odd
positive integer, then

n) .J~n - 1)!
(
r 2 2._, (n ; 1)

(38)

There are no simple formulas for the distribution function corresponding

to r((X, A.) except when a = m is a positive integer. In this case we can
integrate by parts to obtain for x > 0

rx l'"y'"-1e-A.>' dy =

Jo

(m -

1)!

-(A.y)'"-1e-A.)'IX
(m - 1)!

rx A.'"-1y'"-2e-A.>' dy

o Jo

J.o

(m - 2)!

Am- 1 y'"- 2e- A.7 d

(AX)"'- 1e- b

y-

(m - 2)!

(m - 1)!

'

provided that m > 2. If we integrate by parts m - 1 times in this manner

and observe that

we obtain the formula

A.'"y'"-1 e-A.y
- - dy
o (m - 1)!

(39)

= 1-

m-1

(A.x)"e- .b

Jc=o

k!

'

X> 0.

This formula provides an interesting connection between a random

variable X having the gamma density r(m, A.) and a random variable Y
having a Poisson distribution with parameter A.x. Specifically, (39) states
that
(40)

P(X < x) = P(Y > m).

This connection is relevant to the theory of Poisson processes, as we will

see in Chapter 9.
The qualitative behavior of the gamma density, illustrated in Figure 6,

Figure 6.

The Gamma Density

is easily obtained by methods of calculus. One important property of

gamma densities is that if X and Y are independent random variables

5.4.

Inverse distribution functions

131

having respective densities r(ah A.) and r(a2, A.), then X + Y has the
gamma density r(a1 + a 2 , A.). This result will be proven in Chapter 6.
This and other properties of gamma densities make them very convenient
to work with. There are many applied situations when the density of a
random variable X is not known. It may be known that X is a positive
random variable whose density can reasonably well be approximated by a
gamma density with appropriate parameters. In such cases, solving a
problem involving X under the assumption that X has a gamma density
will provide an approximation or at least an insight into the true but
unknown situation.
5.4.

Inverse distribution functions

Important applications of the change of variable formulas of Section

5.2.1. can be obtained by letting the function q> be related to a distribution
function F.
Let X be a continuous random variable having distribution function F
and density function f. We will apply the change of variable formula to
the function cp = F. If y = F(x), then dyfdx = F'(x) = f(x) and hence
dxfdy = 1/ f(x). Thus according to (18), the random variable Y = F(X)
has density g where
g(y)

= ~~:~ =

1,

0 < y < 1,

and g(y) = 0 otherwise. In other words, the random variable Y = F(X)

is uniformly distributed on (0, 1). This result is valid-even if the function
q> = F does not satisfy all the assumptions of Theorem 1. By using a
direct argument, one can show that if X is a continuous random variable
having distribution function F, then F(X) is uniformly distributed on (0, 1).
(If F is discontinuous at some point x 0 , then P(X = x 0 ) > 0, so that
P(F(X) = F(x0 )) > 0 and F(X) could not possibly be uniformly
distributed on (0, 1).)
One can also proceed in the other direction. Let F be a continuous
distribution function that is strictly increasing on some interval I and
such that F = 0 to the left of I if I is bounded from below and F = I to the
right of I if I is bounded from above. Then for 0 < y < 1, by the intermediate value theorem of calculus, there is a unique value of x such that
y = F(x). Thus F- 1 (y), 0 < y < 1, is well defined. Under these
assumptions, if Y is a uniformly distributed random variable on (0, 1), then
the random variable F- 1( Y) has F as its distribution function.
Two of the examples from Section 5.2.1 can be used . to illustrate the
above result. In Example 6 we obtained exponentially distributed random
variables as transforms of a uniformly distributed random variable. The

Continuous Random Variables

132

reader should check to see that these transformations can be obtained by

the method of the above paragraph. In Example 11 we showed that if 0
is uniformly distributed on ( -n/2, n/2), then tan 0 has the Cauchy
distribution. Let Ybe uniformly distributed on (0, 1). Then 0 = 1tY- 7t/2
is uniformly distributed on ( -7t/2, n/2), so that

= tan 0 =

~)

tan ( 1r Y -

has the Cauchy distribution. This is exactly what we would get by using
the result of the previous paragraph. According to Example 10, the
Cauchy distribution function is given by
1

F(x) = -

+ -1 arctan x,

-00

<X <

00,

1t

and the equation y = F(x), or

= -21 + -1 arctan x,
1t

has solution
x = F- 1(y) =tan

(1ty- ~).

For some purposes it is desirable to generate a random variable X having

a prescribed distribution function F. One way of doing this is to first
generate a uniformly distributed random variable Y and then set
X = F- 1( Y). This method is especially useful on a digital computer
since there are very satisfactory methods for generating (what act like)
uniformly distributed random variables on such computers. Suppose for
example we want a routine for generating a random variable X having the
standard normal density n(O, 1). We would use a subroutine for generating
a random variable Yuniformly distributed on (0, 1) and a subroutine for
computing the numerical function <I>- 1 , and then compute X = <I>- 1( Y).
To generate a random variable X having the normal density n(p,, u 2 ) we
would set X = Jl + u<I>- 1(Y).
Inverse distribution functions are useful for other purposes. To see this
let X have the normal density n(p,, u 2 ) and recall from Section 5.3.1 that
P(X < b) = <I>

(b : Jl) .

Suppose we want to choose b such that P(X < b)

forb in the equation

= .9.

We need to solve

5.4.

Inverse distribution functions

133

The solution is given by

b - .Jl

<1>-1(.9)

(1

or

b = Jl

u<I>- 1 (.9).

Froin Table I we see that <I>- 1(.9) = 1.28. Thus b = Jl

+
In applied statistics the number b =
P(X

:s:

Jl

1.28u and

1.28u) = .9.
Jl

+ 1.28u is called the upper decile

for the n(Jl, u ) distribution.

Let F be any distribution function that satisfies the requirements for
p- 1 (y), 0 < y < 1, to be well defined, as discussed above. Then
m = F- 1 (1/2) is called the median ofF, F- 1 (3/4) and F- 1 (1/4) are called
the upper and lower quartiles ofF, F- 1 (.9) is called the upper decile and
F- 1 (k/100) is called the upper k-percentile. These definitions can be
modified to apply to arbitrary and, in particular, discrete distribution
functions.
If X has a symmetric density then X clearly has median m = 0. For a
more interesting example~ let X be exponentially distributed with parameter
A.. Then its median misgiven by 1 - e-.tm = 1/2, which has the solution
m = A. - 1 log 2. Suppose X represents the time for a radioactive particle
to decay. Then if we have a very large number of such particles we would
expect that by time m one half of the particles would have decayed. In
physics this time is called the half-life of the particle. If we observe the
half-life m we can use it to compute the rate of decay A., since A. = m- 1 log2.
For a final application of inverse distribution functions, let X have the
normal density n(Jl, u 2 ) and suppose we want to find a > 0 such that
P(Jc - a ~ X~ Jl + a) = .9. Then by (25) we have to solve for a in
the equation

~ (;)

- <I> (- ;)

.9.

Since <I>( -x) = I - <I>(x) for all x, we have

2<1> (;) - 1 = .9
and hence a = 011>- 1 (.95). From Table I we see that <1- 1(.95) = 1.645.
In other words,

P(Jl - l.645u

X < Jl

1.645u) = .9.

.675u) = .5

By using the same technique we obtain

P(Jt - .675u

X ::; Jl

Continuous Random Variables

134

or equivalently,
P(IX -

PI

:S .675u)

.5.

This says that if X has the normal density n(Jl~ u 2 ), then X will differ from
11 by less than .675u with probability one-half and by more than .675u with
probability one-half. If we think of J1. as a true physical quantity and X as a
measurement of Jl, then IX- 111 represents the measurement error. For
this reason .675u is known as the probable error.
Exercises

1 Let X be a random variable such that P(l X - 11 = 2) = 0. Express

P(IX - 111 > 2) in terms of the distribution function Fx.
2

Let a point be chosen randomly from the interior of a disk of radius R

in the plane. Let X denote the square of the distance of the point
chosen from the center of the disk. Find the distribution function of X.

3 Let a point be chosen uniformly from a solid ball in three-dimensional

space of radius R. Let X denote the distance of the point chosen from
the center of the ball. Find the distribution function of X.
4

Let a point be chosen uniformly over the interval [0, a]. Let X denote
the distance of the point chosen from the origin. Find the distribution
function of X.

5 Let a point be chosen uniformly from the interior of a triangle having

a base of length I and height h from the base. Let X be defined as the
distance from the point chosen to the base. Find the distribution
function of X .
6 Consider an equilateral triangle whose sides each have length s. Let a
point be chosen uniformly from one side of the triangle. Let X denote
the distance of the point chosen from the opposite vertex. Find the
distribution function of X.
7 Let the point (u, v) be chosen uniformly from the square 0 < u :S 1,
0 :S v :S 1. Let X be the random variable that assigns to the point
(u, v) the number u + v. Find the distribution function of X.
8 Let F be the distribution function given by Formula (3).

Find a

number m such that F(m) = 1/2.

9

Let X denote the decay time of some radioactive particle and assume
that the distribution function of X is given by Formula (3). Suppose A.
issuchthatP(X ~ .01) = 1/2. FindanumbertsuchthatP(X ~ t) =
.9.

10

Let X be the random variable in Exercise 4. Find the distribution

function of Y = Min(X, a/2).

Exercises

135

11

Let X be a random variable whose distribution function F is given by

F(x) =

0,

X< 0,

0 <

3'
X

< 1,

~ X

< 2,

2'

1,

X> 2.

Find:
(a) P(l/2 < X < 3/2);
(b) P(l/2 ~ X < I);
(c) P(l /2 ~ X< I) ;
(d) P(l ~ X ~ 3/2) ;
(e) P(l < X < 2).
12 If the distribution function of

X was defined in one of the following

ways, describe how properties (i)-{iv) of Section 5.1.1 would have to

be modified in each case :
(a) F(x) = P(X < x);
(b) F(x) = P(X > x);
(c) F(x) = P(X ~ x).
13 A point is chosen uniformly from ( -10, 10). Let X be the random

variable defined so that X denotes the coordinate of the point if the

point is in [- 5, 5], X = -5 if the point is in ( -10, - 5), and X = 5
if the point is in (5, 10). Find the distribution function of X .
14

Let X be a continuous random variable having density f given by

f(x) = (l/2)e-lxl,

-00

<X<

00.

Find P(I ~ lXI < 2).

15 Let F be the distribution function defined by
1

F(x) =

2 + 2(1xl + 1)'

-oo < x < oo.

Find a density function f for F. At what points x will F'(x) = f(x)?

16 Find a density function for the random variable in Exercise 3.
17 Find a density function for the random variable in Exercise 7.
18 Let X be a continuous random variable having density

formula for the density of Y

19 Let X and Y = X

f . Find a

= lXI.

be positive continuous random variables having

densities f and g respectively. Find fin terms of g and find g in terms
off.
20 Let X be uniformly distributed on (0, 1). Find the density of Y = X 1' ' ,
where fJ :1:- 0.

136

Continuous Random Variables

21 Let X be a positive continuous random variable having density

Find a formula for the density of Y

= lf(X + 1).

f.

22 Let X be a random variable, g a density function with respect to

integration, and ({J a differentiable strictly increasing function on

(- oo, oo ). Suppose that
tp(x)

P(X < x) =

-oo

g(z) dz,

-co < x < co.

Show that the random variable Y = q>(X) has density g.

23 Let X be a random variable that is uniformly distributed on (a, b).
Find a linear function q> such that Y = q>(X) is uniformly distributed
on (0, 1).
24 Let X have an exponential density with parameter ..t Find the density

of Y = eX, where c > 0.

25 Let g(x) = x(l - x)2 , 0 :s:; x :S:: 1, and g(x) = 0 elsewhere.
should g be normalized to make it a density?

How

Let X have the Cauchy density. Find the density of Y = a + bX,

b #= 0.
27 Let X denote the sine of an angle chosen at random from ( -7t/2, 7t/2).
Find the density and distribution function of X.

26

X be a continuous random variable having symmetric density f

and such that X 2 has an exponential density with parameter A.. Find f.

28 Let

29 Let X be a continuous random variable having distribution function F

and density function f Then f is said to be symmetric about a if

f(a + x) = f(a - x), -co < x < co. Find equivalent conditions
in terms of the random variable X and in terms of the distribution
function F.
30 The error function is defined by

erf(x)

r~

~; Jo e -yl dy,

-co < x < co.

Express ci> in terms of the error function.

31 Let X have the normal density n(O, u 2 ). Find the density of Y
2

32 Let X have the normal density n(Jl, a ).

= lXI.
Find the density of Y = eX.

This density is called a lognormal density.

33 Let X be normally distributed with parameters Jl and a 2
P(IX - Ill < a).

Find

34 Let X be normally distributed with parameters Jl and u 2 Find numbers

a and b such that a + bX has the standard normal distribution.

35 Let X be normally distributed with parameters J.l = 0 and u 2 = 4.
Let Y be the integer-valued random variable defined in terms of X by
Y = m if m - 1/2 < X< m + 1/2, where m is an integer such that

137

Exercises

- 5 < m ~ 5, Y = -6 if X < - 5.5, and Y = 6 if X > 5.5. Find

fr and graph this density.
36

Suppose that the weight of a person selected at random from some

population is normally distributed with parameters J.l and CT. Suppose
also that P(X < 160) = 1/2 and P(X < 140) = 1/4. Find Jl and r1
and find P(X ;;:::: 200). Of all the people in the population weighing at
least 200 pounds, what percentage will weigh over 220 pounds?
tP be the number such that <l>(tp) = p, 0 < p < 1. Let X have the
normal density n(p,, CT2 ). Show that for 0 < p 1 < p 2 < I,

37 Let

P(Jl

tPl(l

~ X ~ Jl

+ tPlCT)

= P2 - P1

38 Suppose a very large number of identical radioactive particles have

decay times which are exponentially distributed with some parameter A..
If one half of the particles decay during the first second, how long will it
take for 75% of the particles to decay?

A.. Let Y be the

integer-valued random variable defined in terms of X by Y = m if
m < X< m + 1, where m is a nonnegative integer. How is Y
distributed?

39 Let X be exponentially distributed with parameter

T be a positive continuous random variable denoting the failure

time of some system, let F denote the distribution function of T, and
suppose that F(t) < 1 for 0 < t < oo. Then we can write F(t) =
I - e~G<r>, t > 0. Suppose G'(t) = g(t) exists for t > 0.
(a) Show that T has a density f given by

40 Let

f(t)
(t)
1 - F(t) - g '

0 < t <

00.

The function g is known as the "failure rate," for heuristically,

~t+

dt I T > t) = f(t) dt
1 - F(t)
(b) Show that for s > 0 and t > 0,
P(t

P(T > t +sIT> t)

= g(t) dt.

= e-s: .. ,<u>

du.

(c) Show that the system improves with age (i.e., for fixed s the expressions in (b) increase with t) if g is a decreasing function, and
the system deteriorates with age if g is an increasing function.
(d) Show that

00

g(u) du =

00.

(e) How does g behave if Tis exponentially distributed?

(f) If G(t) = ,t~, t > 0, for which values of (X does the system
improve, deteriorate, and stay the same with age?

Continuous Random Variables

138
41 Let X have the gamma density r(ex,

where c

A.). Find the density of Y

eX,

> 0.

> 1, the gamma density has a maximum at (ex - 1)/A..

have the gamma density r{ex, A.). Find the density of Y = .JX.

42 Show that if ex
43

Let X

Y be uniformly distributed on (0, 1). Find a function q> such that

X= q>(Y) has the density f given by f(x) = 2x, 0 ~ x < 1, and
f(x) = 0 elsewhere.
45 Let Y be uniformly distributed on (0, 1). Find a function q> such that
q>(Y) has the gamma density r(1/2, 1/2). Hint: Use Example 12.
44 Let

46 Find ~- 1 (t) fort = .1, .2, . .. , .9, and use these values to graph ~- .
1

47 Let

X have the normal density n(Jl, u2 ). Find the upper quartile for X

48 Let X have the Cauchy density. Find the upper quartile for X .

49 Let X have the normal density with parameters Jl and u 2 = .25. Find
a constant c such that
P(IX - Jll :s; c) = .9.
so Let X be an integer-valued random variable having distribution
function F, and let Y be uniformly distributed on (0, 1). Define the
integer-valued random variable Z in terms of Y by

=m

if F(m - 1) < Y < F(m),

for any integer m. Show that Z has the same density as X.

Jointly Distributed
Random Variables

In the first three sections of this chapter we will consider a pair of continuous
random variables X and Y and some of their properties. In the remaining four
sections we will consider extensions from two ton random variables X 1 , X 2 , . ,
X 11 The discussion of order statistics in Section 6.5 is optional and will not be
needed later on in the book. Section 6.6 is mainly a summary of results on sampling
distributions that are useful in statistics and are needed in Volume II. The material
covered in Section 6.7 will be used only in proving Theorem 1 of Chapter 9 and
Theorem 1 of Chapter 5 of Volume II.
6.1.

Properties of bivariate distributions

Let X and Y be two random variables defined on the same probability

space. Their joint distribution function F is defined by
F(x, y) = P(X < x, Y < y),

-oo < x,y < oo.

To see that F is well defined, note that since X and Yare random variables,
both {w I X(w) ::5: x} and {w I Y(w) < y} are events. Their intersection
{w I X(w) < x and Y(w) < y} is also an event, and its probability is
therefore well defined.
The joint distribution function can be used to calculate the probability
that the pair (X, Y) lies in a rectangle in the plane. Consider the rectangle
R

{(x, y) I a < x ::;; b, c < y ::;; d},

where a ::;; b and c < d. Then

P((X, Y) e R) = P(a < X

(1)

::5:;

b, c < Y < d)

F(a, d) - F(b, c) + F(a, c).

= F(b, d) -

To verify that (1) holds observe that

P(a

<

<

b, Y

< d)

= P(X

<

b, Y

< d) -

= F(b, d) - F(a, d).

139

P(X

<

a, Y

< d)

140

Jointly Distributed Random Variables

Similarly
P(a < X < b, Y < c)

= F(b, c)

- F(a, c).

Thus
P(a < X ~ b, c < Y ~ d)
= P(a

< X

= (F(b, d) -

b, Y

d) - P(a < X < b, Y < c)

F(a, d)) - (F(b, c) - F(a, c))

and (1) holds as claimed.

The one-dimensional distribution functions Fx and Fr defined by
Fx(x) = P(X < x)

Fr(Y) = P(Y ~ y)

and

are called the marginal distribution functions of X and Y. They are related
to the joint distribution function F by
Fx(x) = F(x, oo) =lim F(x,y)
y-+ 00

and
Fr(Y)

F(oo, y)

lim F(x, y) .

.x-+ 00

If there is a nonnegative function f such that

(2)

F(x, y) =

J~oo (J~oo f(u, v) dv) du,

- 00

< x, y < oo,

then f is called a joint density function (with respect to integration) for the
distribution function F or the pair of random variables X, Y. Unless
otherwise specified, throughout this chapter by density functions we shall
mean density functions with respect to integration rather than discrete
density functions.
IfF has density f, then Equation (I) can be rewritten in terms of j, to
give
(3)

P(a < X < b, c < Y

d)

J.b

(J:

f(x, y) dy) dx.

By using the properties of integration and the definition of a probability

space, it can be shown that the relation
(4)

P((X, Y) e A)

Jf

f(x, y) dx dy

holds for subsets A in the plane of the type considered in calculus. By

letting A be the entire plane we obtain from (4) that
(5)

J~oo J~oo f(x, y) dx dy =

1.

6.1.

Properties of bivariate distributions

141

We also obtain from (4) that

Fx(x) = P(X

< x)

s:>

(J:> f(u, y) dy) du

and hence Fx has marginal density fx given by

s:<Xl f(x,

fx(x) =

which satisfies

J:>

F x(x) =

y) dy

fx(u) du.

Similarly Fr has marginal density fr given by

fr( y) =

J:> f(x, y) dx.

As in the one-dimensional case, f is not uniquely defined by (2). We

can change fat a finite number of points or even over a finite number of
smooth curves in the plane without affecting integrals off over sets in the
plane. Again as in the one-dimensional case, F determines .f at the continuity points of f. This fact can be obtained from (3).
By differentiating (2) and applying the rules of calculus we obtain

~ F(x,
ay

y) =

f"

- >

(j_ay f'

f(u, v) dv) du

- >

J:> f( u, y) du

and
(6)

02

- - F(x, y) = f(x, y).

ax oy

Under some further mild conditions we can justify these operations and
show that (6) holds at the continuity points of f. In specific cases instead
of checking that the steps leading to (6) are valid, it is usually simpler to
show that the function f obtained from (6) satisfies (2).
Let us illustrate the above definitions and formulas by
reconsidering Example 1 of Chapter 5. We recall that in that example, we
chose a point uniformly from a disk of radius R. Let points in the plane be
determined by their Cartesian coordinates (x, y). Then the disk can be
written as
Example 1.

{(x, y) I x2

y2 < R2}.

Let X and Y be random variables denoting the random coordinates of the

point chosen. Corresponding to the assumption of uniformity, we suppose
that X and Y have ajoint density.f given by

142

Jointly Distributed Random Variables

(7)

f(x, y) =

{n!2 '
0,

elsewhere.

Then for any subset A of the disk (say of the type considered in calculus),
P((X, Y) e A) =

JJf(x , y) dx dy
A

area of A
which agrees with our assumption of uniformity. The marginal density fx
is given by
l
2.../R2 _ X2
fx{x)

oo

f(x, y) dy

-oo

JJR2-x2
_
- - 2 dy =
. -JR2-x2 nR

nR

for - R < x < R and fx(x) = 0 elsewhere. The marginal density f r(Y)
is given by the same formula with x replaced by y.
The variables X and Y are called independent random variables if
whenever a < b and c 5: d, then
(8)

P(a < X < b, c < Y ~ d) = P(a < X < b)P(c < Y ~ d).

By letting a = c = - oo, b = x, and d = y, it follows that if X and Yare

independent, then
(9)

F(x, y)

Fx(x)Fy(y),

-oo < x,y < oo.

Conversely (9) implies that X and Y are independent. For if (9) holds,
then by (1) the left side of (8) is
F(b, d) - F(a, d) - F(b, c)

F(a, c)

= Fx(b)Fy(d) -

Fx(a)Fy(d) - Fx(b)Fy(c)

= (Fx{b) - Fx(a))(Fy(d) = P(a <X:::;;; b)P(c

Fx(a)Fy(c)

Fy(c))

< Y <d).

More generally, it can be shown that if X and Yare independent and A

and B are unions of a finite or countably infinite number of intervals, then
P(X e A, Y e B) = P(X e A)P( Y e B)

or, in other words, the events

{ru I X(ru) e A}

are independent events.

and

{w

I X(ru) e B}

6. 1.

Properties of bivariate distributions

143

Let X and Y be random variables having marginal densities fx and fr

Then X and Y are independent if and only if the function f defined by
f(x, y) = fx(x) fr(Y),

-00

< x, y < oo,

is a joint density for X and Y. This follows from the definition of independence and the formula

As an illustration of dependent random variables, let X and Y be as in

Example 1. Then for -R < x < Rand -R < y < R,
(10)

.; '

)~" (

J'X\X'Jr

) =

4JR2

x2

J R2 -

y2

n2R4

'

which does not agree with the joint density of these variables at x = 0,
y = 0. Since (0, 0) is a continuity point of the functions defined by (7) and
(I 0), it follows that X and Yare dependent random variables. This agrees
with our intuitive notion of dependence since when X is close to R, Y must
be close to zero, so information about X gives us information about Y.
Density functions can also be defined directly, as we have seen in other
contexts. A two-dimensional (or bivariate) density function f is a nonnegative function on R 2 such that

J~> J:> f(x, y) dx dy

= 1.

Corresponding to any bivariate density function f, there is a probability

space and a pair of random variables X and Y defined on that space and
having joint density f
The easiest way to construct two-dimensional density functions is to
start with two one-dimensional densities / 1 and / 2 and define the function
fby
(11)

f(x, y)

= ft (x)f2(y),

-oo < x , y < oo.

Then f is a two-dimensional density function since it is clearly nonnegative

and

J:> J~> f(x, y) dx dy = J:> / (x) dx J:> / (y) dy = 1.

1

If random variables X and Yhave thisf as their joint density, then X and Y
are independent and have marginal densitiesfx = / 1 andfr = / 2
As an illustration of (11), let / 1 and f 2 both be the standard normal
density n(O, 1). Then f is given by

144

Jointly Distributed Random Variables

!(x, y ) -

-.x'-/ 2

../2n e

-y'-{2

../ire e

or
(12)

- __!_ e -(.x'-+y'-)/ 2
/(x y) -

'

21t

-00

'

< x, y <

00.

The density given by (12) is called the standard bivariate normal density.
In our next example we will modify the right side of(l2) slightly to obtain a
joint density function that corresponds to the case where the two random
variables having normal marginal densities are dependent.
Let X and Y have the joint density function f given by

Example 2 .

!( x, y) -- ce -(.x'--.xy+yl)/ 2 ,

<

- 00

X,

y <

00,

where cis a positive constant that will be determined in the course of our
discussion. We first "complete the square" in the terms involving y and
rewrite! as
f(x, y) =

ce-[(y-.xf2)2+J.x2f41/ 2'

- 00

<

X,

<

00,

and then note that

J~<Xl f(x,

fx(x) =

y) dy

Making the change of variable u

oo

2
e-(y-.x/ l ) / 2

dy

- oo

= y -

J~<Xl e-(y-.x/ 2)2/ 2 dy.

ce-3x2f8

Jco

x /2, we see that

e-"'"12 du

= ../27t.

-oo

Consequently
fx(x) =

c../27te-3x

2
/

It is now clear that fx is the normal density n(O, cr2 ) with cr2 = 4/3 and
hence

or c
(13)

= .JJ/47t.

Consequently

f( x, y ) --

../3 e -(.xl-.xy+yl)/ 2 ,
4n

-oo < x,y < oo.

The above calculations now show that fx is the normal density n(O, 4/3).
In a similar fashion, we can show that fr is also n(O, 4/3). Since f(x, y) =F
fx(x)jy(y), it is clear that X and Yare dependent.

6.2.

145

Distribution of sums and quotients

6.2.

Distribution of sums and quotients

Let X and Y be random variables having joint density f. In many

contexts we have a random variable Z defined in terms of X and Y and
we wish to calculate the density of Z. Let Z be given by Z = <p(X, Y),
where <p is a real-valued function whose domain contains the range of X and
Y. For fixed z the event {Z :S z} is equivalent to the event {(X, Y) e Az},
where Az is the subset of R 2 defined by

= {(x, y) I <p(x, y)

A:

:::;; z}.

Thus
Fz(z) = P(Z < z)
= P((X, Y) E A:)

= JJt(x, y) dx dy.
A~

If we can find a nonnegative function g such that

JJt(x, y) dx dy

J~<X) g(v) dv,

-oo < z < oo,

A~

then g is necessarily a density of Z. We will use this method to calculate

densities of X + Y and Y/ X.
6.2.1.

=X +
= {(x, y) I x + y :S

Distribution of sums.

Az

Set Z

Y. Then
z}

is just the half-plane to the lower left of the line x

Figure 1. Thus

Fz(z) = Jftcx, y) dx dy =

+y =

z as shown in

J~ro (J~:x f(x, y) dy) dx.

A~

Make the change of variable y = v - x in the inner integral. Then

Fz(z)

= f~oo (J~oo f(x, v- x) dv) dx

=

J~oo (J:oo f(x, v- x) dx) dv,

where we have interchanged the order of integration. Thus the density of

Z = X + Y is given by
(14)

fx+y(z)

J:oo f(x, z -

x)

dx,

oo

< z < oo.

148

Jointly Dist1ibuted Rsndom Vs1isb/es

y

Figu1e 1

In the main applications of (14), X and Y are independent and (14) can be
rewritten as

(IS)

fx+r(z) =

J~co fx(x)fy(z

- x) dx,

- oo < z < oo.

If X and Y are nonnegative independent random variables, then

fx+r(z) = 0 for z ~ 0 and

(16)

fx+r(z) =

J:

fx(x)fy(z - x) dx,

0 < z < oo.

The right side of (IS) suggests a method of obtaining densities. Given

two one-dimensional densities f and g, the function h defined by

h(z) =

J~CX) f(x)g(z ~

x) dx,

-oo < z < oo,

is a one-dimensional density function, which is called the convolution off

and g. Thus the density of the sum of two independent random variables
is the convolution of the individual densities.
Let X and Y be independent random variables each having
an exponential distribution with parameter A.. Find the distribution of
X+ Y.
Example 3.

The density of X is given by fx(x) = .A.e-Ax for x ~ 0 andfx(x) = 0 for

x < 0. The density of Yis the same. Thus fx+r(z) = 0 for z ~ 0 and,
by (16), for z > o
fx+r<z) =

J:

A_e-.txA.e-'-<.:r:-x) dx

= .A?e-A.%

J:

dx = .A?ze-.tz.

6.2.

147

Distribution of sums and quotients

We see that X

Y has the gamma density r(2, A.).

Let X and Y be independent and uniformly distributed

over (0, 1). Find the density of X + Y.
Example 4.

The density of X is given by fx(x) = 1 for 0 < x < 1 and fx(x) = 0

elsewhere. The density of Y is the same. Thus fx+y{z) = 0 for z ~ 0.
For z > 0 we apply (16). The integrandfx(x)fy{z - x) takes on only the
values 0 and 1. It takes on the value I if x and z are such that 0 ~ x ~ I
and 0 s z - x S I. If 0 < z s I, the integrand has value I on the set
0 ~ x ~ z and zero otherwise. Therefore we obtain from (16) that
O~z~

fx+r(z) = z,

1.

If 1 < z ~ 2 the integrand has value I on the set z - 1 < x < 1 and
zero otherwise. Thus by (16)

I < z < 2.

fx+r(z) = 2 - z,

If 2 < z < oo the integrand in (16) is identically zero and hence

2 < z <

fx+r(z) = 0,

00.

In summary

z,
fx+r(z)

2 - z,
0,

z < 1,

1<z~2,

elsewhere.

Figure 2

The graph off is given in Figure 2. One can also find the density of X + Y
by computing the area of the set
Az = {(x, y) I 0 < x < 1, 0 ~ y ~ 1 and

+y <

(see Figure 3) and differentiating the answer with respect to z.

1 <z < 2
Figure 3

z}

Jointly Distributed Random Variables

148

Example 3 has an important generalization, which may be stated as

follows.
Theorem 1 Let X and Y be independent random variables
such that X has the gamma density r(a 1, A.) and Y has the gamma
density r(a2 , A.). Then X + Y has the gamma density

r(a1

Proof.

+ a2, A.).

We note that X and Yare positive random variables and that

;.~~~x~~t-1e-;..x

fx(x) =
and

X> 0,

'

;.1y1-1 e-;.,

fr(y) =
Thusfx+rCz)

r(ocl)

= 0 for z <
fx+r(z) =

r(a2)

'

> 0.

0 and, by (16), for z > 0

A_tzt +tz1e-).z i z
X

r(al)r(a2)

111

(z - x)111 -

dx.

In the preceding integral we make the change of variable x

dx = z du) to obtain

= zu (with

z > 0,

(17)
where
(18)

c =

JA u~~~-1(1

- u)l-1 du

r(al)r(a2)

The constant c can be determined from the fact that fx + r integrates out
to 1. From (17) and the definition of gamma densities, it is clear that f x +r
must be the gamma density r(a 1 + cx 2 , A.) as claimed.
I
From (17) and the definition of the gamma density we also see that
c = 1jr(o: 1 + a 2 ). This together with (18) allows us to evaluate the
definite integral appearing in (18) in terms of the gamma function:

(19)

1 ut-1(1 -u)l-1 d u= r(a1)r(a2) .

0
r(al + a2)

This formula permits us to define a new two parameter family of densities

called Beta densities. The Beta density with parameters a 1 and cx2 is given
by
r(at + a2)xt-1(1 - x)2-1
0 < X < 1,
(20)
f(x) =
r(a 1)r(oc2)
{
0,
elsewhere.

6.2.

149

Distribution of sums and quotients

The reason for this terminology is that the function of a 1 and a 2 defined by
B(<Xh

<X2) -

r(al)r(a2)
r(al + a2)

'

is called the Beta function.

Our final application of the convolution formula is to normally distributed random variables.

Theorem 2 Let X and Y be independent random variables

having the respective normal densities n(J.L 1 , ar) and n(p2 , a~). Then
X + Y has the normal density

n(J.l1
Proof.

Jl2, uf

+ ui).

We assume first that J.L 1 = 112 = 0. Then

-oo < x < oo,

and
-oo < y < oo.
Thus by (15)

fx+r(z) =

Joo

1
21t0' 1 0' 2

exp

-00

[ --1 (x2 2 +
2

(z - x)

0' 1

2
)]

dx.

0' 2

Unfortunately an evaluation of this integral requires some messy computations (which are not important enough to master). One way of proceeding
is to first make the change of variable

.J0'21 + 0'22 x.
0'10'2

After some simple algebra we find that

fx+r(z) =

2nJaf + ai

Joo
-oo

exp

[-

-1 ( u 2 -

a2 ../af + ai

We next complete the square in u and observe that

Then by making a second change of variable

1
2uzu
.

+ 2z

0'2

)]

du.

150

Jointly Dist1ibuted Random Valisb/es

we see that

e -z2/2(ahai)

.J2n.Juf + u~ '

which is just the normal density n(O, uf + u~).

In general, X - J.l1 and Y - Ji. 2 are independent and have the respective
normal densities n(O, uD and n(O, ui). Thus by the above special case,
(X - /it) + ( Y - p 2 ) = X + Y - (p 1 + p 2 ) has the normal density
n(O, uf + ui), and hence X + Y has the normal density

as claimed.

The preceding proof is elementary but messy. A less computational

proof involving more advanced techniques will be given in Section 8.3.
Another proof is indicated in Exercise 36 at the end of this chapter.
Let X and Ybe independent random variables each having
the normal density n(O, u 2). Find the density of X + Y and X 2 + Y 2
Example 5.

From Theorem 2 we see immediately that X + Y has the normal density

n(O, 2u2 ). By Example 12 of Chapter 5, X 2 and Y 2 each have the gamma
density r(l/2, l/2u2 ). It is easily seen that X 2 and Y 2 are independent.
Thus by Theorem 1, X 2 + Y 2 has the gamma density f(l, l/2u 2), which
is the same as the exponential density with parameter l/2CT2
As before, let X and Y denote
random variables having joint density f. We will now derive a formula for
the density of the random variable Z = Y/ X. The set

6.2.2.

Distribution of quotients*.

Az = {(x, y) I yfx s; z}

is shown in Figure 4. If x < 0, then yfx < z if and only if y > xz. Thus
Az = {(x, y) I x < 0 and y > xz} u {(x, y) I x > 0 and y < xz}.

Consequently

Frrx<z) =

JJf(x, y) dx dy
...

r (L:
OJ

f(x, y)

dy) dx + foaJ (J:~ f(x. y) dy) dx.

6.2.

151

Distribution of sums and quotients

z<O

z>O

Figure 4

In the inner integrals, we make the change of variable y = xv (with

dy = x dv) to obtain
Fr1x(z) =

[oo (i-oo xf(x, xv) dv) dx

J (f~
00

+
=

xf(x, xv) dv) dx

00

f~oo (f~oo (-x)f(x, xv) dv) dx

J (J~oo
00

+
=

xf(x, xv) dv) dx

J~oo (f~oo lxlf(x, xv) dv) dx.

By interchanging the order of integration we see that

(21) Fy1x(z)

J~oo (J:oo lxlf(x, xv) dx) dv,

-oo < z < oo.

It follows from (21) that Y/X has the density fr1x given by
(22)

fr1x(z) =

J~oo lxlf(x, xz) dx,

-oo < z < oo.

In the special case when X and Y are independent positive random

variables, (22) reduces to jy1x(z) = 0 for z < 0 and

J
00

(23)

/r,x(z) =

xfx(x)fy(xz) d~,

0 < z

Our next theorem is a direct application of (23).

<

00.

152

Jointly Disttibuted Random Variables

Theorem 3 Let X and Y be independent random variables

having the respective gamma densities f(oc 1 , .'.) and f(oc 2 , .'.). Then
Y/X has the density glven by fr 1x(z) = Ofor z < 0 and

(24

.)

fi (z) - r(ocl + oc2)

Yf X

f(ocl)f(oc2) (z

0 < z <

z2- 1
l)IZI +2'

00

Proof. Recall that

X> 0,
and
fr(Y)

;.2y2-1e-ly
r(oc2)
,

y > 0.

Formula (23) is applicable, so for 0 < z < oo,

A_t +2z2-1
r(ocJ)r(oc2)

J.oo x' +2-1e-.d(::+ 1) dx.

0

By Equation (34) of Chapter 5

fa) x' +2 -le-d<::+ 1) dx

f(<X1

<X2)

(A.(z + 1))t+2

Jo

Consequently (24) holds as claimed.

Since (24) defines a density function we see that for ocb oc2 > 0

roo z2-l(z + 1)-(t+2) dz

= r(ocl)f(oc2) .

Jo

r(ocl

oc2)

Let X and Y be independent random variables each having

the normal density n(O, a 2 ) . Find the density of Y 21X 2
Example 6.

The random variables are the same as those of Example 5. Thus again
X and Y 2 are independent, and each has the gamma density r(l/2, I /2a2 ).
Theorem 3 is now applicable and Y 2 IX 2 has the density /r21x2 given by
/r2;x2(z) = 0 for z < 0 and
2

r(l)
21

/r x (z)

z-l /l

f(l l2)f(ll2) (z + 1)

1
- ---1t(Z + l).J Z

0 < z <

00.

'

(Here we recall from Equation (35) of Chapter 5 that r(l/2) = .j;.) We

leave it to the reader to show as an exercise that under the same conditions
both Y/X and Y/IXI have the Cauchy density.

6.3.

153

Conditional densities
6.3.

Conditional densities

In order to motivate the definition of conditional densities of continuous random variables, we will first discuss discrete random variables.
Let X and Y be discrete random variables having joint density f If x is
a possible value of X, then
_
IX _ ) _ P(X = x, Y = y) _ f(x, y)
P(y -Y
x.
P(X = x)
fx(x)
The function fr 1x defined by
(25)

frlx(Y I x) =

f(x, y)
fx(x) '

fx(x)

0,

fx(x) = 0,

i= 0,

is called the conditional density of Y given X. For any possible value x of

X,

Lfrlx(Y I x) = 'L,f(x, y) = fx(x) = 1,

'
fx(x)
fx(x)

so that for any such x, fr 1x(Y I x) defines a discrete density function of y

known as the conditional density of Y given X = x. In the discrete case
conditional densities involve no really new concepts.
If X is a continuous random variable, however, then P(X = x) = 0 for
all x so that P(Y = y I X= x) is always undefined. In this case any
definition of conditional densities necessarily involves a new concept.
The simplest way to define conditional densities of continuous random
variables is by analogy with Formula (25) in the discrete case.
Definition 1 Let X and Y be continuous random variables
having joint density f. The conditional density fr 1x is defined by
f(x, y)

(26)

fnx<Y I x) =

fx(x) '

0,

0 < fx(x) < oo,

elsewhere.

It follows immediately from this definition that, as a function of y,

fr 1x(Y I x) is a density whenever 0 < fx(x) < oo (again called the
conditional density of Y given X = x). Conditional densities can be used
to define conditional probabilities. Thus we define
(27)

P(a < Y < b I X= x)

ib

frjx(Y I x) dy,

a< b.

Alternatively, we could attempt to define the conditional probability

appearing in (27) by means of the following limit:
{28)

P(a s; Y ~ b

IX =

x)

= lim P(a ~ Y ~ b
11~0

Ix - h < X

h).

154

Jointly Distributed Random Variables

The right-hand side of (28) can be rewritten in terms off as

J~~~ (J!J(u, y) dy) du - l" (1/2h) s~~: (J!f(u, y) dy) du
.
ltm J + 11
)
1m
J +,.
.
11!0 ~- 11 (f~ro f(u , y) dy du
11!0
(1/2h) ~- 11 /x(u) du

If

L" f(u, y) dy
is continuous in u at u = x, the numerator of the last limit converges to

i"

f(x, y) dy

as h ! 0. If fx is continuous at x the denominator converges to fx(x) as

h ! 0. Under the additional condition that fx(x) =I= 0, we are led from (28)
to
P(a

~ b IX

= x) =

J!f~x, y) dy,
fx(x)

which agrees with (27). In summary, we have defined conditional densities

and conditional probabilities in the continuous case by analogy with the
discrete case. We have also noted that, under further restrictions, a
limiting process would yield the same definition of conditional probabilities. It turns out that such limiting processes are difficult to work with
and will not be used further.
It follows immediately from the definition of conditional density functions that
(29)

f(x, y)

= fx(x)fYJx(Y I, x),

- 00

<

X,

y <

X,

y <

00.

If X and Y are independent and

(30)

f(x, y) = fx(x)fy(y),

- 00

<

00,

then
(31)

fr1x<Y I x) = fy(y),

0 < f x(x) < oo and - oo < y < oo.

Conversely if (31) holds, then it follows from (29) that (30) holds and X
and Y are independent. Thus (31) is a necessary and sufficient condition
for two random variables X and Yhaving a joint density to be independent.
Example 7.

Let X and Y have the bivariate density f given by Formula

(13), namely

!(x, y ) -_ ,J"j e -(x~-x)'+ 7~)/2 ,

4n

- oo < x, y < oo.

Then as we saw in Example 2, X has the normal density n(O, 4/3). Thus for
-00 < x, y < 00

6.3.

Conditional densities

155

_ 1 -(y-x/2)1/2
- .J2n e
.

In other words, the conditional density of Y given X = x is the normal

density n(x/2, l).
We have been starting with joint densities and using them to construct
marginal densities and conditional densities. In some situations we may
reverse this by starting with marginal densities and conditional densities
and using them to construct joint densities.
Let X be a uniformly distributed random variable over
(0, 1), and let Y be a uniformly distributed random variable over (0, X).
Find the joint density of X and Y and the marginal density of Y.
Example 8.

From the statement of the problem, we see that the marginal density of
Xis given by
for
0 < X < 1,
fx(x) = {
elsewhere.

~.

The density of Y given X = x is uniform on (0, x), so that

for

1/x
frrx(Y I x) = {O,

0 < y < X< 1,

elsewhere.

Thus the joint density of X and Y is given by

1/x
f(x, y) = {O,

for

0 < y <X< 1,
elsewhere.

The marginal density of Y is

fr(Y)

andfr(y)

= f~

-~

f(x, y) dx

Jyf. X! dx =

-logy,

= 0 elsewhere.

Of course, we can reverse the roles of X and Y and

define the conditional density of X given Y = y by means of the formula

6.3.1.

(32)

Bayes rule.

"

JXfY(

y) -

f(x, y)
fr(Y) '

0 < fr(Y) <

Since

f(x, y)

= fx(x)frrx<Y I x)

00.

166

Jointly Distributed Random Variables

and
/r(Y)

J~oo f(x, y) dx

f~oo fx(x)frlx(Y I x) dx,

we can rewrite (32) as

fxlr(X I y) =

(33)

fx(x)frlx(Y I x)
J~oofx(x)frlx(Y I x) dx

This formula is the continuous analog to the famous Bayes' rule discussed
in Chapter 1.
In Chapters 3 and 4 we considered random variables X and Y which
were both discrete. So far in Chapter 6 we have mainly considered random
variables X and Y which are both continuous. There are cases when one is
interested simultaneously in both discrete and continuous random variables. It should be clear to the reader how we could modify our discussion
to include this possibility. Some of the most interesting applications of
Formula (33) are of this type.
Suppose the number of automobile accidents a driver will
be involved in during a one-year period is a random variable Y having a
Poisson distribution with parameter A., where A. depends on the driver.
If we choose a driver at random from some population, we can let A. vary
and define a continuous random variable A having density fA The
conditional density of Y given A = A. is the Poisson density with parameter
A. given by
Example 9.

A.'e-.A.

Y!
{

IYII\(Y I A.) =

for

0,

= 0, 1, 2, .. . '

elsewhere.

Thus the joint density of A and Y is

/A(A.)A.'e-;.
f(A.,y) =
y!
{
0,

for

y = 0, 1, 2, . . . '
elsewhere.

In general we cannot find a nice formula for the marginal density of Y or

the conditional density of A given Y = y, since we cannot evaluate the
required integrals. We can find simple formulas, however, in the special
case when f is a gamma density r(ex, p), so that
pa;_r~.-le-AP

fA(A.) =
In this case,

r(cx)
0,

for

A. > 0,
elsewhere.

6.4.

157

Properties of multivariate distributions

co

p~~ ).- 1

J.o

e- J.fl

T(rx)

= P

).>'e- J.
-dJ.

y!

J.co .A.cz+y-te-J.(fl+l) d.A.

y!r(a) o
r(rx + y)p
- y! r(a)(ft + l)ll+)'.
11

The value of the last integral was obtained by using Formula (34) of
Chapter 5. We leave it as an exercise for the reader to show that fr is the
negative binomial density with parameters rx and p = P/(1 + p). We also
have that for A. > 0 andy a nonnegative integer,
'"

(.A.

I y)

= f(A., y)

fy(y)

JAfY

pll;.a+y- 1e- J.(fJ+ l)y! r(rx)(p

r(a)y!r(rx

(p +

1)11+ y

y)p

l)ll+>',{ll+y-te-l(/1+1)

r(cx

y)

which says that the conditional density of A given Y = y is the gamma

density r(cx + y, p + 1). If someone in the insurance industry wanted to
solve problems of this type he would quite possibly try to approximate the
true density fA by a gamma density r(a, p), where rx and Pare chosen to
make the approximation as good as possible.
6.4.

Properties of multivariate distributions

The concepts discussed so far in this chapter for two random variables X
and Y are readily extended to n random variables. In this section we
indicate briefly how this is done.
Let X 1, , X" ben random variables defined on a common probability
space. Their joint distribution function F is defined by
- oo < xh ... , x,. < oo.
The marginal distribution functions Fxm' m = 1, ... , n, are defined by
-00

<

X 111

<

00.

The value of Fxm(x111) can be obtained from F by letting x 1 , , X 111 _ 1,

X 111 + 1 , . , x,., all approach + oo.
A nonnegative function f is called a joint density function (with respect
to integration) for the joint distribution function F, or for the random
variables X1 , . . . , X11, if

Jointly Distributed Random Variables

158

J::

Xl

(34)

F(xh . .. , x,.) =

-oo

f(u1, .. . , u,.) du 1 du,.,

- 00

<

<

X 1 , ... , X 11

00.

Under some further mild conditions the equation

f(x 1 ,

. ,

x,.)

0"

ox ... ox
1

F(x 1 ,

x,.)

II

is valid at the continuity points of F. If (34) holds and A is any subset of

R" of the type considered in calculus, then
P((X 1 ,

X,.) e A) =

f ~ f f(x

1 , ... ,

x,.) dx 1 dx,..

In particular

(35)
and if am

oo Joo f(x 1 , , x,.) dx 1

-oo
-oo
hm for m = 1, ... , n, then

dx,. = 1

P(a 1 < X 1 ~ b~t ... , a,. ~ X,. < b,.)

=

bl fb" f(xh ... , x,.) dx

4J

dx,..

4n

The random variable Xm has the marginal density fxm obtained by integrating/ over the remaining n - 1 variables. For example,
fxix2) =

f~

>

J:

f(x 1 , , x,.) dx 1 dx 3

dx,..

00

In general, the random variables X1 , .. , X,. are called independent if

whenever am < bm form = 1, ... , n, then
P(a 1 < X 1 ~ h 1 ,

... ,

a,. < X, < b,.)

= P(a 1 < X 1 < b 1)

P(a,.

< X,. < b,).

A necessary and sufficient condition for independence is that

<

- 00

X 1, . . . , X 11

<

00.

The necessity is obvious, but the sufficiency part for n > 2 is tricky and
will not be proved here. If Fhas a density j, then X1 , . . , X,. are independent if and only iff can be chosen so that
- 00

<

X 1> , X 11

<

00.

One can also define ann-dimensional density directly as a nonnegative

function on R!' such that (35) holds. The simplest way to construct ndimensional densities is to start with n one-dimensional densities / 1 , . . . , j,.
and define/by
(36) j(x1, .. . , x,.) = ft(x 1)

'/,.(x,.),

- 00

<

X 1, ... , X 11

<

00.

6.4.

Properties of multivariate distributions

159

If X 1 , , X,. are random variables whose joint density f is given by (36),

then X 1 , . , X,. are independent and x.. has the marginal density f ...
Let X 1 , , X,. be independent random variables, each
having an exponential density with parameter A.. Find the joint density of
Example 10.

X1,

X,..

The density of X m is given by

A.e- .t.x,.
fx...(x,.) = { O,

for

0 < x,. <

elsewhere.

00,

Thusfis given by
for

xh ... , x,. > 0,

elsewhere.

In order to c ompute the density of the sum of n independent random

variables, and for several other purposes, we need the following fact.

Theorem 4 Let X 1 , , X,. be independent random variables.

Let Y be a random variable defined in terms of X 1 , , X,., and let Z
be a random variable defined in terms of X,.+ 1 , , X,. (where
I ~ m < n). Then Y and Z are independent.
The proof of this theorem will not be given since it involves arguments
from measure theory.
Using this theorem and an argument involving mathematical induction,
we can extend Theorems 1 and 2 to sums of independent random variables,
as follows.

Theorem 5 Let X 1 , . , X,. be independent random variables

such that X,. has the gamma density r(a111, A.) for m = 1, ... , n.
Then X 1 + + X,. has the gamma density r(a, A.), where

Recall that the exponential density with parameter A. is the same as the
gamma density r(l, A.). Thus as a special case of this theorem we have the
following corollary: If X 1 , , X, a.re independent random variables, each
having an exponential density with parameter A., then X 1 + + X,. has
the gamma density r(n, A.).

Theorem 6 Let X 1 , , X,. be independent random variables

such that Xm has the normal density n(J,t111, u!). m = I, .. . , n. Then
X 1 + + X, has the normal density n(p,, u 2 ), where
J.l = p, 1

+ .. + Jl

11

and

u2 = uf + .. + u;.

If X., ... , X, has a joint density f, then any subcollection of these

random variables has a joint density which can be found by integrating
over the remaining variables. For example, if 1 < m < n,

Jointly Distributed Random Variables

160

The conditional density of a subcollection of X 1 , , X,. given the

remaining variables can also ,b e defined in an obvious manner. Thus the
conditional density of X,.+., . .. , X,. given X 1 , , X,. is defined by

'"

J Xm+t.XniXJ, ... , X,.. X,.+ h ' X,.

Xh , X,.

I"

f(xl, ... , x,.)

(

JX~o ... , Xm X1, . . ' X,.

) ,

where f is the joint density of X 1 , , X,..

Often conditional densities are expressed in terms of a somewhat different
notatwn. For example, let n + 1 random variables X 1 , , X,., Y have
joint density f. Then the conditional density of Y given X 1 , . , X,. is
defined by
I
)
f(x .. . ,x,.,y)
fiYIX..... . Xn( y x., . .. ' Xn = I"
(x . . . x)

1x...... x"

6.5.

t; '

Order statistics

Let U., . . . , U,. be independent continuous random variables, each

having distribution function F and density function f. Let X 1 , , X,.
be random variables obtained by letting X 1 (ro), . .. , X,.(ro) be the set
U1 (ro), .. . , U,.(ro) permuted so as to be in increasing order. In particular,
xl and x. are defined to be the functions
and
X,.(w) = max (U1 (ro), . . . , U,.(ro)).
The random variable Xt is called the kth order statistic. Another related
variable of interest is the range R, defined by
R(ro) = X,.(ro) - X 1 (ro)
= max (U1 (m), . .. , U,.(ro)) - min (U1(ro), ... , U,.(ro)).

It follows from the assumptions on U 1 , .. , U, that, with probability one,

the U,'s are distinct and hence X 1 < X 2 < < X,..
To illustrate these definitions numerically, suppose U1(ro)
U2 (ro) = 3.5, and U3 (w) = 4.3. Then X 1 (ro) = 3.5, X 2 (ro)
X3 (ro) = 4.8, and R(ro) = 1.3.

= 4.8,
= 4.3,

Consider a machine having n parts whose failure times

u., ... , U,. satisfy the assumptions of this section. Then Xt is the time it
takes for k of the parts to fail. If the entire machine fails as soon as a
single part fails, then X 1 = min (U1 , , U,.) is the failure time of the
machine. If the machine does not fail until all its parts have failed, then
X,. = max ( U 1 , .. . , U,.) is the failure time of the machine.
Example 11.

6.5.

Order statistics

161

Let n hopefully identical parts be manufactured in a

single run of an assembly line and let U1 , .. , U, denote the lengths of the
n parts. An inspector might be interested in the minimum length x~. and
maximum length X, to check if they are within certain tolerance limits.
If the parts are to be interchangeable the amount of variation in the
lengths may have to be kept small. One possible measure of this variation
is the range R of the lengths.
Example 12.

We will now compute the distribution function of the kth order statistic
xk. Let -00 < X < 00. The probability that exactly j of the u,s lie in
( -oo, x] and (n - j) lie in (x, oo) is
(;) Fi(x)(l - F(x))"- 1,

because the binomial distribution with parameters n and p = F(x) is

applicable. The event {X" < x} occurs if and only if k or more of the
U,'s lie in (- oo, x]. Thus
(37)

Fx~c(x) = P(X"
=

~ x)

(~)

j=lc

F'(x)(1 - F(x))"-1,

- oo

< x < oo.

In particular the distribution functions of X, and X1 can be written very

simply as
-oo < x < oo,
Fx"(x) = (F(x))",
and
-OO<X<OO.
Fx,(x) = 1 - (1 - F(x))",
In order to find the corresponding density functions, we must differentiate
these quantities. We easily find that

-oo < x < oo,

and

fx.(X) = nf(x)(I - F(x))"- 1,

-oo<x<oo.

The corresponding derivation for X" in general is slightly more complicated. From (37),

fx~c(x) =

n!
f(x)F 1 - 1 (x)(1 - F(x))"- 1
J=k (j - 1)! (n - j)!

,.-1

. n:
J=kJ!(n - J ~

L" .

J=k (J -

l:

1)!

I)! (n - J)!

f(x)F 1(x)(l - F(x))"_ 1 _ 1

f(x)F1 - 1(x)(1 - F(x))"- 1

n.
. f(x)Fi- 1(x)(l - F(x))"- 1
J=k+1 (J - 1)! (n - J)!

Jointly Distributed Random Variables

162

and by cancellation

f Xk(X )

( 38)

n!

= (k _ 1)! (n _ k)!

/( )F"X

1(

X1

F( ))"-lc
X

-ex:>

<

<

OC).

In order to find the density of the range R we will first find the joint
density of X 1 and X,. We assume that n > 2 (since R = 0 if n = 1). Let
x ~ y. Then
P(X. >

X,

X, < y) = P(x < ul < y, ... '

<

u, < y)

(F(y) - F(x))",

and of course
P(X, :::;; y)

= F"(y).

Consequently
F x,,x"(x, y) = P(X 1 < x, X, :::;; y)

= P(X, <

y) - P(X 1 > x, X 11 < y)

= F"(y) - (F(y) - F(x))".

The joint density is given by

02
fx,,x"(x, y) = ox oy F x,,xJx, y)

= n(n -

1)/(x)f(y)(F(y) - F(x))"- 2 ,

X< y.

It is obvious and easily shown that

fx,,xJx, y)

0,

x > y.

By slightly modifying the argument used in Section 6.2.1 to find the density
of a sum, we find that the density of R = X, - X 1 is given by

fR(r)

= J~co fx,,x"(x, r +

x) dx.

In other words

fR(r) =

{n(n - 1) J~00 f(x)f(r + xXF(r + x) -

F(x))"- dx,

r>O
r

0,

< 0.

These formulas can all be evaluated simply when U 1, . , U, are independent and uniformly distributed in (0, 1). This is left as an exercise.
There is a uheuristic" way for deriving these formulas which is quite
helpful. We will illustrate it by rederiving the formula for fxk Let dx
denote a small positive number. Then we have the approximation

fx"(x) dx ~ P(x ~ X~; ~ x

dx).

163

6.6. Sampling distributions

The most likely way for the event {x ~ X11: < x + dx} to occur is that
k - 1 of the U/s should lie in (- oo, x], one of the U/s should lie in
(x, x + dx], and n - k of the U/s should lie in (x + dx, oo) (see Figure
5). The derivation of the multinomial distribution given in Chapter 3 is
applicable and the probability that the indicated number of the U;'s will
lie in the appropriate intervals is
r_ (x) dx ~
JXk

n!

(k- 1)! l!(n- k)!

X(J~
~

aJ

j(u) du

r-l J.x+dx j(u) du (J.:dx j(u) du) n-It

n!

f(x) dxFlt-l(x)(l - F(x))"-k,

(k - 1)! (n - k)!

from which we get (38). We shall not attempt to make this method
ngorous.
k-1

1 ]

n-k

x+dx

Figure 5
6.6.

Sampling distributions

Let X1, .. . , X" be independent random variables, each having the

normal density n(O, u 2 ). In this section we will find the distribution
functions of several random variables defined in terms of the X's. Besides
providing applications of the preceding material, these distribution
functions are of fundamental importance in statistical inference, and will
be needed in Volume II.
The constant u 2 is convenient but unessential since X 1fu, ... , Xnfu are
independent and each has the standard normal density n(O, I). Thus we
could always take u 2 = 1 with no loss of generality.
By Theorem 6 the random variable X 1 + + X" has the normal
density with parameters 0 and nu2 If we divide this sum by various
constants we can get alternative forms of this result. Thus
Xt

+ ... +X"
n

is normally distributed with parameters 0 and u2 fn, and

xl + + x"
u.Jn
has the standard normal density n(O, 1).

164

Jointly Distributed Random Variables

Since X 1 /u has the standard normal density, it follows from Example 12

of Chapter 5 that Xi/u 2 has the gamma density r(I/2, 1/2). Thus by
Theorem 5
x~

+ + x;
(]2

has the gamma density r(n/2, 1/2). This particular gamma density is very
important in statistics. There the corresponding random variable is said
to have a chi~square (X 2 ) distribution with n degrees of freedom, denoted
by X2 (n). By applying Theorem 5 we will obtain the following result about
x2 distributions.
Theorem 7 Let Y1 , . ~ Y, be independent random variables
such that Ym has the x2 (km) distribution. Then Y1 + + Y,. has .the
x2(k) distribution, where k = kl + . .. + k,.
Proof. By assumption, Ym has the gamma distribution r(km/2, 1/2).

Thus by Theorem 5, Y1 + + Y, has the gamma distribution r(k/2, 1/2)

where k = kt + .. . + k,. But this distribution is x2 (k) by definition. I
We can also apply Theorem 3 to find the distribution of the ratio of two
independent random variables Y1 and Y2 having distributions x2(kt) and
x2(k 2 ) respectively. It is traditional in statistics to express the results in
terms of the normalized variables Y 1 fk 1 and Y2 /k 2 The distribution of
Ytfk 1

Y2/k2
is known as the F distribution with k 1 and k 2 degrees of freedom, denoted
by F(kt. k 2 ).
Theorem 8 Let Y1 and Y2 be independent random variables
having distributions x2 (k 1) and x2(k 2 ). Then the random variable

-Ytfkt
-,
Y2/k2

which has the distribution F(k 1 , k 2 ), has the density f given by f(x) = 0
for x s; 0 and

(39) f(x) = (k1 /k2) f[(kl + k2)/2) (ktxfk 2 )Ckt/l)- l

r(ktf2) r(k 2 /2) [I + (k 1xf k2)J<k1+kl>t z

> O.

'

By Theorem 3, the random variable YtfY2 has density g, where

g is given by (24) with ll 1 = ktf2 and cx 2 = k 2 /2. Thus the density f of
k2 Y1/kt Y2 is given by
Proof.

and (39) now follows from (24).

6.6.

Sampling distributions

165

We can apply this result to the random variables X 1 , , X, defined at

the beginning of this section. Let I < m < n. By Theorem 4, the random
variables
and
are independent. Since they have the respective distributions
x2(n - m) we see that the random variable

+ + X!)/m
+ + X~)/(n -

x2(m) and

(Xf

(X;+ 1

m)

has the F(m, n - m) distribution and the density given by (39), where
k 1 = m and k 2 = n - m. Tables ofF distributions are given in Volume
II.
The case m = 1 is especially important. The random variable

Xf
(Xi

+ + ~:)/(n

1)

has the F(1, n - 1) distribution. We can use this fact to find the distribution of

y =

x1
../(Xi + + x;)/(n - 1)

Since X 1 has a symmetric density function and is independent of the

random variable ../(Xi + + x;)/(n - 1), it follows easily from
Theorem 2 of Chapter 5 that Y has a symmetric density function fr By
Example 5 of Chapter 5 the density fr2 is related to fr by
frl(z)

1
=

r (fy( -.Jz) + fr(../ z)),

z > 0.

2yz

By using the symmetry of fr and letting z = y 2 we see that

fr(Y) = IYI/r:~(Y 2 ).

Since Y 2 has the F(l, n - I) density given by (39) with k 1

k 2 = k = n - 1, we now find that
fr(Y) =

Since r(l/2)
(40)

= ..;;,,

M~=

I and

I yl (1/k) r[(k + 1)/2] (y 2 /k)- 112

r(1/2) r(k/2) [1 + (y 2 fk)]<k+ 1 >12

this expression reduces to

r[(k

+ t)/2] [1 + (y 2 fk)]- <"+ 1 >12

I

v kn: r(k/2)

'

-oo < Y < ro.

A random variable whose density is given by (40) is said to have a t

distribution with k degrees of freedom. We observe that the t distribution

Jointly Distributed Random Variables

166

with 1 degree of freedom is the same as the Cauchy distribution discussed

in Chapter 4. Tables of t distributions are given in Volume II.
The distribution of the random variable

Y =

x~

.J(X~ + + x;)f(n - 1)

which is a t distribution with n - 1 degrees of freedom, depends only on

the fact that
and
are independent and distributed respectively as n(O, 1) and , x2 (n - 1).
Thus we have the following result.

Theorem 9 Let X and Y be independent variables with the

respective distributions n(O, 1) and x2 (k). Then

JY/k
has a t distribution with k degrees of freedom.
6.7.

Multidimensional changes of variables

Let X 1 , , X,. be continuous random variables having joint density f.

Let Y1 , , Y,. be random variables defined in terms of the X's. In this
section we will discuss a method for finding the joint density of the Y's
in terms of f. We will consider mainly the case when the Y's are defined
as linear functions of the X's.
Suppose then that
It

Yi

I:

j=l

i = 1, . . . , n.

auX1,

The constant coefficients ail determine an n x n matrix

[a,J]

[a~l
a,.l

Associated with such a matrix is its determinant

la11

a 11

detA =

If det A ::f: 0 there is a unique inverse matrix B = [b 11] such that BA = I

or equivalently
1,
i = j,
(41)
{0,
i #: j.

6. 7.

Multidimensional changes of variables

167

The constants b 11 can be obtained by solving for each i the system (41) ofn
equations in then unknowns bil~ . . ~ bi,. Alternatively~ the constants b11
are uniquely defined by requiring that the equations
II

Yt

= }=1
l: a,Jxi'

i = 1, ... , n,

have solutions
i = 1, . .. , n.

(42)

Theorem 10 Let X 1 , . . . , X,. be continuous random variables

having joint .density f and let random variables Y1 , . , Y,. be defined by
II

Yi

~ a,jxj,

i = 1, . . . , n,

j= 1

where the matrix A = [au] has nonzero determinant det A. Then

Y1 , , Y,. have joint density fr 1 , .... r" given by
(43)

fr ... rJY1, , Y11) = ldet AI f(xl, 'x,.),

where the x's are defined in terms of the y's by (42) or as the unique
solution to the equations y 1 = 1:j= 1 a1ix1.

This theorem, which we will not prove here, is equivalent to a theorem

proved in advanced calculus courses in a more general setting involving
"Jacobians." From the general result proved in advanced calculus, we can
extend the above theorem to nonlinear changes of variables. We will
describe this extension briefly, although it will not be needed later.
Let the Y's be defined in terms of the X's by

Yi

g/...X1 ,

X,.),

1, .. . , n.

Consider the corresponding equations

(44)

i = 1, ... , n.

Suppose that these equations define the x's uniquely in terms of the y's,
that the partial derivatives oyJox1 exist and are continuous, and that the
Jacobian
iJyl
oy1
ox,.
iJxl
.
..
.
J(x1 , . , x,.) = .
.
oy,.
oy.,
ox,.
oxl
is everywhere nonzero. Then the random variables Y1 ,
tinuous and have a joint density given by

. . .

Y,. are con-

Jointly Distributed Random Variables

168

(45)

fr ...... r,.(Yh ... , y,.) = IJ(

)I f(xh , x,.),

Xt, .. , X 11

where the x's are defined implicitly in terms of the y's by (44). This change
of variable formula can be extended still further by requiring that the
functions Ui be defined only on some open subsetS of lf' such that
P((Xh ... , X,.) e S)

1.

In the special case when Yi = I:j= 1 aiix1, we see that oy,jox1 = ail and
J(x 1 , , x,.) is just the constant det A = det [a 11]. So it is clear that
(45) reduces to (43) in the linear case.
Let X 1 , , X,. be independent random variables each
having an exponential density with parameter l . Define Yt> ... , Y,. by
Yi = X 1 + + X~> 1 < i < n. Find the joint density of Y1 , . , Y,..
The matrix [a11] is
Example 13.

1 0

1 1 0

0
1 1

Its determinant is clearly 1. The equations

i = 1, ... , n,

have the solution

i = 2, . .. , n.

The joint density of X 1 ,

(46)

f(x b

. . . ,

. ,

X,. is given by

_ {.A."e-A.<x + +x,.>,
0,

xh ... , x,. > 0,

elsewhere.

x,.) -

Thus the joint density fr 1 , , y,. is given by

(47)

A."e-A.y,.

fr ...... r,.(Yt> , y,.) = {O,

'

< Yt < .. ' < y,.,

elsewhere.

Of course, one can apply the theorem in the reverse direction. Thus-If
Y~> . .. , Y,. have the joint density given by (47), and random variables
Xb ... , X,. are defined by X 1 = Y1 and X, = Yi - Y1_ 1 for 2 ~ i < n,
then the X's have the joint density f given by (46). In other words,
Xb ... , X,. are independent and each has an exponential distribution with
parameter A.. This result will be used in Chapter 9 in connection with
Poisson processes.

Exercises

111
Exercises

1 Let X and Y be continuous random variables having joint density

function f. Find the joint distribution function and the joint density
function of the random variables W = a + bX and Z = c + dY,
where b > 0 and d > 0. Show that if X and Y are independent, then
W and Z are independent.
2 Let X and Y be continuous random variables having joint distribution

function F and joint density function f Find the joint distribution

function and joint density function of the random variables W = X 2
and Z = Y 2 Show that if X and Yare independent, then Wand Z are
independent.
3 Let X and Y be independent random variables each uniformly distributed on (0, 1). Find
(a) P(IX - Yl S .5),

(b)

P(/; - t/~ .s),

(c) P(Y >XI Y ~ 1/2).

4 Let X and Y be independent random variables each having the normal

density n(O, u 2) . Find P(X2

Y 2 < 1). Hint: Use polar coordinates.

5 Let X and Y have a joint density f that is uniform over the interior of
the triangle with vertices at (0, 0), (2, 0), and (1, 2). Find P(X < 1 and
y ~ 1).
6 Suppose the times it takes two students to solve a problem are indepen-

dently and exponentially distributed with parameter A.. Find the

probability that the first student will take at least twice as long as the
second student to solve the problem.
7 Let X and Y be continuous random variables having joint density f
given by f(x, y) = l 2e-A', 0 ~ x ~ y, and f(x, y) = 0 elsewhere.
Find the marginal densities of X and Y. Find the joint distribution

function of X and Y.

< x < y ~ 1, and f(x, y) = 0 elsewhere.

(a) For what values of r.x. can c be chosen to make fa density function?
(b) How should c be chosen (when possible) to make/a density?

8 Letf(x, y)

c(y - x), 0

(c) Find the marginal densities off.

9 Let f(x, y) = ce-<xl-xy+ 412 >12, - oo < x, y < oo. How should c be
chosen to make fa density? Find the marginal densities of f.
10 Let X and Y be independent continuous random variables having joint

density f. Derive a formula for the density of Z = Y - X.

11 Let X and Y be independent continuous random variables having the

indicated marginal densities. Find the density of Z = X

Y.

170

Jointly Distributed Random Variables

(a) X and Yare exponentially distributed with parameters A. 1 and A.2 ,

where A. 1 :1= A.2
(b) X is uniform on (0, I), and Y is exponentially distributed with
parameter A..
12 Let X and Y have a joint density f given in Exercise 8. Find the
density of Z = X + Y.
13 Let X and Y be independent and uniformly distributed on (a, b).
Find the density of Z = IY - XI.
14

15
16

17

18
19

20
21
22

23

24
25

26

Let X and Y be continuous random variables having joint density f .

Derive a formula for the density of Z = aX + bY, where b :1= 0.
Let f be a Beta density with parameters cx 1 > I and cx2 > I. Where
does f take on its maxfmum value?
Let X and Y be independent random variables having respective
Find the density of
normal densities n(~h ui} and n(p.2,
Z=Y-X.
Let a point be chosen randomly in the plane in such a manner that its
x and y coordinates are independently distributed according to the
normal density n(O, cr2 ). Find the density function for the random
variable R denoting the distance from the point to the origin. (This
density occurs in electrical engineering and is known there as a Rayleigh
density.)
Let X and Y be continuous random variables having joint density f .
Derive a formula for the density of Z = XY.
Let X and Y be independent random variables each having the normal
density n(O, u 2 ). Show that Y/X and Y/IXI both have the Cauchy
density.
Let X and Y be as in Exercise I9. Find the density of Z = I YI/1 XI.
Let X and Y be independent random variables each having an exponential distribution with parameter A.. Find the density of Z = Y/X.
Let X and Ybe independent random variables having respective gamma
densities r(cxh A.) and r(cx2, ..t). Find the density of Z = X/(X + Y).
Hint: Express Z in terms of Y/ X.
Let X and Y have joint density f as indicated below. Find the conditional density fr 1x in each case:
(a) f as in Exercise 7,
(b) f as in Exercise 8,
(c) f as in Exercise 9.
Let X and Y be distributed as in Example 7. Find P( Y < 2 1 X = 1).
Show that the marginal density fr in Example 9 is negative binomial
with parameters ex and p = fJ/(fJ + 1). Hint: Use Formula (36) of
Chapter 5.
Let Y be a discrete random variable having a binomial distribution
with parameters n and p. Suppose now that p varies as random variable

un.

Exercises

171

TI having a Beta density with parameters cx 1 and cx2 Find the conditional density of II given Y = y.

Y be exponentially distributed with parameter A.. Let A. vary as a

random variable A having the gamma density r(ex, p). Find the
marginal density of Y and the conditional density of A given Y = y.

27 Let

28 Let

xl, x2, x3

denote the three components of the velocity of a

molecule of gas. Suppose that X 1 , X2 , X3 are independent and each
has the normal density n(O, cr2). In physics the magnitude of the
112
is said to have a Maxwell distribuvelocity Y = (X~ + Xi +
tion. Find fr
29 Let X 1, . , Xn be independent random variables having a common
normal density. Show that there are constants An and Bn such that

xn

Xt

+ .. . + Xn- An
Bn

has the same density as X1 .

30 Let

xlt x2, x3

31 Let

X 1 be chosen uniformly on (0, 1), let X 2 be chosen uniformly on

be independent random variables each uniformly

distributed on (0, 1). Find the density of the random variable
Y = X 1 + X 2 + X3 Find P(X1 + X 2 + X3 ~ 2).

(0, X 1), and let X 3 be chosen uniformly on (0, X 2 ). Find the joint
density of xl' x2, x3 and the marginal density of x3.
32 Let U~> .. . , Un be independent random variables each uniformly
distributed over(O, 1). Let Xk, k = I, ... , n, and R be as in Section 6.5.

(a) Find the joint density of X 1 and Xn.

(b) Find the density of R .
(c) Find the density of Xk.

Ut> . .. , Un be independent random variables each having an

exponential density with parameter A.. Find the density of X 1 =
min (U1 , , Un).

33 Let

34 Find a formula for the x2 (n) density.

35 Let X and Y be independent random variables distributed respectively

as x2 (m) and x2 (n). Find the density of Z = Xj(X + Y). Hint: Use
the answer to Exercise 22.
36 Let X and Y be independent random variables each havjng the standard
normal density. Find the joint density of aX + bY and bX - a Y,
where a2 + b2 > 0. Use this to give another derivation of Theorem 2.
37 Let X and Y be independent random variables each having density f.

Find the joint density of X and Z

=X +

Y.

38 Let X and Y be independent random variables each having an exponential density with parameter A.. Find the conditional density of X
given Z = X+ Y = z. Hint : Use the result of Exercise 37.
39 Solve Exercise 38 if X and Yare uniformly distributed on (0, c).

Jointly Distributed Random Variables

172

Let U and V be independent random variables each having the standard

normal density. Set Z = pU + J1 - p 2 V, where -1 < p < 1.
(a) Find the density of Z.
(b) Find the joint density of U and Z.
(c) Find the joint density of X= p. 1 + u 1 U and Y = p.2 + u 2Z,
where u 1 > 0 and u 2 > 0. This joint density is known as a bivariate normal density.
(d) Find the conditional density of Y given X = x.
41 Let X and Y be positive continuous random variables having joint
density f Set W = Y/ X and Z = X + Y. Find the joint density of
Wand Z in terms off Hint: Use Equation (45).
42 Let X and Y be independent random variables having the respective
gamma densities r(a 1 , .A.) and r(a2 , A.). Use Exercise 41 to show that
Yf X and X + Y are independent random variables.
43 Let R and E> be independent random variables such that R has the
Rayleigh density
40

> 0,
r < 0,

and 9 is uniformly distribut,ed on ( -n, n). Show that X = R cos 9

and Y = R sin E> are independent random variables and that each has
the normal density n(O, u 2 ). Hint: Use Equation (45).

Expectations and the

Central Limit
Theorem

In the first four sections of this chapter we extend the definition and properties
of expectations to random variables which are not necessarily discrete. In Section
7.5 we discuss the Central Limit Theorem. This theorem, one of the most important
in probability theory, justifies the approximation of many distribution functions by
the appropriate normal distribution function.
7.1.

Expectations of continuous random variables

Let us recall from Chapter 4 our definition of the expectation of a

discrete random variable X having density f We say that X has finite
expectation if Lx lxlf(x} < oo, and in that case we define its expectation
EX as
EX= L xf(x).
X

The easiest way to define expectations of continuous random variables

having densities is by analogy with the discrete case.
Definition 1 Let X be a continuous random variable having
density f We say that X has finite expectation if

J::> lxlf(x) dx

< oo,

and in that case we define its expectation by

EX =

s:) xf(x) dx.

Using this definition we can easily calculate expectations of continuous

random variables having the various densities discussed in Chapters 5
and 6.
Example 1.

EX

Let X be uniformly distributed on (a, b). Then

iba x ( b-a
1 ) dx = ( 1 )
b-a
173

x21

b=

2a

_+ab
2

174

Expectations and the Central Limit Theorem

Let X have the gamma density r((X, A.). Then

Example 2.

where we have used Formulas (34) and (36) of Chapter 5. By setting

(X = 1 we see that if X has an exponential density with parameter A., then
EX= A.- 1
Let X have the Cauchy density f given by

Example 3.

f(x) = n(l

-oo<x<oo .

+ x2) '

Then X does not have finite expectation. For

-~

lxl

1
n(l

X )

dx

= -2 f.~
0

1t

= -2

lim

1t c-> ~

ic
0

= -1 lim log (1
1t c ... ~

=
7.2.

dx

dx

+ x 2 ) lc
0

00 .

A general definition of expectation

The definition of expectation given in Section 7.1 is certainly appropriate

from a computational point of view for the case of continuous random
variables having densities. In order to define expectation in general,
however, it is better to extend the notion of e?'pectation directly from the
discrete case to the general case. We will present only the basic ideas that
motivate the general definition of expectation. The precise details require
further background in the theory of measure and integration. We will
assume in our discussion that all random variables under consideration
are defined on a fixed probability space (!l, .91, P).
Let X and Y be discrete random variables such that, for some e > 0,
P(IX- Yl < e) = I. It follows from Theorems 2(iii) and 3 of Chapter 4

7.2.

A general definition of expectation

175

that if Y has finite expectation, then X has finite expectation and

lEX - EYI ~ e. It also follows that if Y does not have finite expectation,
then neither does X. When expectation is defined in general these properties should continue to hold.
Let us assume this to be the case and let X be any random variable.
Suppose we want to calculate EX with an error of at most e, for some
e > 0. All we need to do is find a discrete random variable Y such that
P(IX- Yl ~ e) = '1 and calculate EY according to the methods of
Chapter 4.
It is easy to find such approximations to X. Let X, be the discrete
random variable defined by
(1)

X, = ek if ek < X < e(k

1)

for the integer k.

This random variable can also be defined in terms of the greatest integer
function [ ] as X, = e[X/e]. If e = Io-n for some nonnegative integer
n, then X,(w) can be obtained from X(ro) by writing X{ro) in decimal form
and dropping all digits n or more places beyond the decimal point. It
follows immediately from (I) that
X(ro) - e < X,(ro)

and hence P(IX - X, I < e)

= I.

fx.(x) = {:(ek ~ X < e(k

wen,

X(ro),

The density function of X, is given by

if x = ek for the integer k,
elsewhere.

1))

The random variable X, has finite expectation if and only if

~ lxlfx.(x) =
Jl:

I: lekiP(ek <

X < e(k

+ I)) < oo,

"

in which case
EX, = ~ ekP(ek ~X < e(k

1)).

"

These expressions can be written in terms of Fx. For

P(ek

< X < e(k + 1))

P(X < e(k

1)) - P(X < ek)

and by Equation (5) of Chapter 5, P(X < x) = F(x-) holds for all x.
The following theorem, which we state without proof, will be used to give a
general definition of expectation.

Tlleorem 1 Let X be a random variable and let X,, e > 0, be

defined by (1 ). If X, has finite expectation for some e > 0, then X,
luls finite expectation for all e > 0 and
lim EX,

exists and is finite.

Expectations and the Central Limit Theorem

176

This theorem and our preceding discussion suggest the following general
definition of expectation.
Definition 2 Let X be a random variable and let X,, B > 0, be
defined by (1). If X, has.finite expectation for some e > 0, we say that
X has finite expectation and define its expectation EX by
EX= lim

EX~..

e-+0

Otherwise we say that X does not have finite expectation.

From the discussion preceding Theorem 1 it follows that the definition

of EX can be given in terms of the distribution function of X and that if
two random variables have the same distribution function, then their
expectations are equal (or both not finite). Using techniques from the
theory of measure and integration, we can show that Definition 2 gives
the same values as do our previous definitions for the special cases when
X is discrete or when X is a continuous random variable having a density.
There is an analog of Theorem 1 of Chapter 4 which we state without
proof. In this theorem, qJ can be any function of the type considered in
calculus.
Theorem 2 Let Xh . .. , X,. be continuous random variables
having joint density f and let Z be a random variable defined in terms of
X., . . . , X, by Z = qJ(X1 , , X,.). Then Z has finite expectation
if and only if

s:oo .J_

00

jqJ(X1, , X,.)jf(xl , . . . , X,.) dx1 dx,. < 00,

00

in which case

EZ =

Joo Joo
- CXl

qJ(x 1, . . . , x,.)/(xtt .. . , x,.) dx 1 dx,..

- CXl

We can show that the basic properties of expectation proven in Chapter

4 for discrete random variables are valid i~ general. In particular Theorems
2, 3, and 4 of Chapter 4 are valid and will be freely used.
As in the discrete case we sometimes refer to EX as the mean of X. The
definition of moments, central moments, variance, standard deviation,
covariance, and correlation given in Chapter 4 for discrete random
variables depend only on the notion of expectation and extend immediately
to the general case.
In general, as in the discrete case, if X has a moment of order r, then X
has a moment of order k for all k < r. Theorems 6 and 7 of Chapter 4
are also true in general. The reader should review these definitions and
theorems in Chapter 4 before going on to the next section.

7.3.

Moments of continuous random variables

7.3.

177

Moments of continuous random variables

Let X be a continuous random variable having density f and mean p..

If X has finite mth moment, then by Theorem 2
EX"' =

s:oo x"'f(x) dx

and

In particular, if X has finite second moment, its variance u 2 is given by

U

s:<X) (x

- p.} /(x) dx.

Note that (] 2 > 0. For if (] 2 = 0, then it follows by the argument of

Section 4.3 that P(X = p.) = 1, which contradicts the assumption that X
is a continuous random variable.
Let X have the gamma density r(a, A.). Find the moments
and the variance of X.
Example 4.

The mth moment of X is given by

= - ,t

r(a)

J.<Xl X m+-1 e -A.x dX,

0

so by Formulas (34) and (36) of Chapter 5

EX"' = ,tr(m + a)
,tm+r(a)

(2)

= a(a

1) (a

m -

1)

,tm

The variance of X is given by

u2

= EX 2- (EX)2 =

a(a ,t~ 1) -

(i) 2= ;2 .

By setting a = I, we see that if X has the exponential density with

parameter ,t, then EX"' = m! A. -m and X has variance A.- 2 For a second
special case, let X have the x2 (n) distribution which, according to Section
6.6, is the same as the r(n/2, 1/2) distribution. Then

EX= n/2 = n

1/2

and

Var X

= - n/2- 2 = 2n.
(1/2)

178

Expectations and the Central Limit Theo1em

We can often take advantage of symmetry in computing moments.

For example, if. X has a symmetric density, if EX"' exists, and if m is an
odd positive integer, then EX"' = 0. To see this, note that by Theorem 2
of Chapter 5~ X and -X have the same distribution function. Thus X"'
and (-X)"' = -X"' have the same distribution function and consequently the same expectation. In other words EX"' = E(- X"') = -EX"',
which implies that EX"' = 0.
Let X have the normal density n(J.l, a 2 ) . Find the mean
and central moments of X.
Example 5.

The random variable X - J.l has the normal density n(O, u 2 ), which is a
symmetric density. Thus E(X - Jl)"' = 0 form an odd positive integer.
In particular E(X - Jl) = 0, so we see that the parameter J1 in the normal
density n(Jl, u 2 ) is just the mean of the density. It now follows that all the
odd central moments of X equal zero. To compute the even central
moments we recall from Section 5.3.3 that Y = (X - p,) 2 has the gamma
density r(I /2, l/2a2 ). Since for m even E(X - J.l)"' = EY"'12, it follows
from Example 4 that

(m; 1)

(2~2) m/ 2 r (~)
- -------:::--(

_1

2a

)"''2

= u'"l 3 (m - 1).

By using Formulas {35) and (38) of Chapter 5, we obtain the alternative

formula
(3)

E(X - J.l)"' =

m!
2"'' 2

(;)!

a"'.

In particular a 2 denotes the variance of X and E(X - Jl)4 = 3a4

Let X and Y be continuous random variables having joint density f,
means Jlx and Jlr, and finite second moments. Then their covariance is
given by
(4)

E(X - JlxXY - Jly) =

f~oo f~oo (x

- J.lrXY - Jlr)f(x, y) dx dy.

Let X and Y have the joint density f in Example 2 of

Chapter 6. Find the correlation between X and Y.
Example 6.

7.3.

Moments of continuous random variables

179

According to Example 2 of Chapter 6

I( x, y )

./J e-[(xl-xy+yl)/2]

=-

4n

-_ ..j"j
- e -3xlf8e-[(y-x/ 2)1/ 2].
4n

We saw in that example that X and Y each have the normal density
n(O, 4/3). Thus Jlx = Jly = 0 and Var X= Var Y = 4/3. From Equation (4) and the second expression for f, we have

f oo
2-Jhr -

..j]
Exy = -

foo

xe -(3x2/ 8) d x

00

y -1- e -[(y-.x/ 2)2/ 2] dy .

- 00

...[i;c

Now

J
oo

- oo

foo (u + X)
1 e
- --=

1 e -[(y -xf2)l j2] d yY --=

J2n

-(ul/2)

2 J2n

-oo

du- -X ,
2

and hence
1

EXY =

(JJ) ...J2;c
J:oo x

1/2

1/2 . 4/3

foo

x2e-<3xlf8)

dx

-oo

n(x; 0, 4/3) dx

2/3.

The correlation p between X and Y is given by

p=

EXY

Jvar X .Jvar Y

2/3

J4i3 ..j4/3

=-

Let U1, , U,. be independent random variables each

uniformly distributed over (0, 1) and set
Example 7.

X = min (U1 ,

U,.)

Y = max (U1 ,

.. ,

U,.).

and
Find the moments of X and Y and the correlation between X and Y.
These random variables were studied in Section 6.5 (where they were
denoted by X 1 and X,). Specializing the results of that section to U/s
which are uniformly distributed, we find that X and Y have a joint density
fgiven by
0 <X~ y ~ 1,
(5)
f(x, y) = {~~n
l)(y - x)"-2,
elsewhere.

Expectations and the Central Limit Theo1em

180

Those readers who have skipped Section 6.5 can think of the present
problem as that of finding the moments of, and the correlation between
two random variables X and Y whose joint density is given by (5).
The mth moment of X is given by
-

1)

J:

x'" dx

= n(n -

1)

x'" dx Y - x

= n(n

EX'"

J:
(

)" _ 1

I.,=

n - 1

(y - x)"- 2 dy
1

y-x

x'"(1 - x)"- 1 dx.

The definite integral appearing in this expression is a Beta integral and

was evaluated in Formula (19) of Chapter 6. From this formula we find
that

= nr(m + l)r(n) =

EX'"

r(m
In particular, EX
that

1/(n

Var X

(EX 2 )

1)

1) and EX 2
(EX) 2

m! n!
(m + n)!

= 2/(n

(n

l)(n

1) (n

2). It follows

+ 2)

The mth moment of Y is given by

EY'" = n(n -

1)

= n(n -

1)

J:

y'" dy

J: (

y - x)"- 2 dx

1
1

y'" d y ( Y - x )"-1( - l) rx=y

= n

s:

n -

n
1)2 (n

x=O

y'"+n-1 dy

n
m+n

- - --

Thus EY = nf(n

1) and

Var y = n

2 -

)
1

= (n

2)

Alternatively, these quantities could be computed from the marginal

densities of X and Y.
To find the covariance of X and Y we start with

J J:
1

EXY = n(n -

1)

y dy

x(y - x)"- 2 dx.

7.4.

Conditional expectation

181

Since
x(y - xt- 2 = y(y - x)"- 2

we find that
EXY = n(n - 1)

J:

(y - x)"-t,

J:
J J: (
2

y dy

(y -

x)"- dx

- n(n - 1)

y dy

y - x )"- dx

= n(n- 1)

yzdy(y- x)"-t(-1) 1"='

n- 1
x=O

Jo

J.

- n(n - 1) o y d y y -

=n

Ll

x)"( l) lx =y
n

x =o

y" + 1 d y - ( n - 1) Jot y" + 1 d y

1
- -n

Consequently
Cov (X, Y)

EX Y- EXEY

= -

- --- -- 2
n + 2
(n + 1)

(n

11)2 (n

2)

Finally we obtain for the correlation between X and Y,

Cov (X, Y)

p = ---,=:-=======
../var X VarY

= (n

+ l)!(n + 2)/ (n +

l)~(n + 2)

n
7.4.

Conditional expectation

Let X and Y be continuous random variables having joint density f and

suppose that Y has finite expectation. In Section 6.3 we defined the
conditional density of Y given X = x by

fr1xCY I x) = {

f(x, y)
fx(x) '

0 < fx(x) < oo,

0,

elsewhere.

182

Expectations and the Central Limit Theorem

For each x such that 0 < fx(x) < oo the function fr1xCY I x), - oo <
y < oo, is a density function according to Definition 5 of Chapter 5. Thus
we can talk about the various moments of this density. Its mean is called
the conditional expectation of Y given X = x and is denoted by
E[Y I X= x] or E[Y I x]. Thus

J_

00

(6)

E[Y

I X = x] =

yf(y I x) dy
00

= s~oo yf(x, y) dy

fx(x)

when 0 < fx(x) < oo. We define E[Y I X= x] = 0 elsewhere. In

statistics the function m defined by m(x) = E[ Y I X = x] is called the
regression function of Y on X.
Conditional expectations arise in statistical problems involving prediction and Bayesian estimation, as we will see in Volume II. They are also
important, from a more general viewpoint, in advanced probability theory.
We will confine ourselves to some elementary illustrations of conditional
expectations. The general theory is quite sophisticated and will not be
needed in this book.
Let X and Y have the joint density f in Example 2 of
Chapter 6. Find the conditional expectation of Y given X = x.
Example 8.

In Example 7 of Chapter 6 we found that the conditional density of Y

given X = x is the normal density n(x/2, 1) which we now know has
mean x/2. Thus

E[ y I X

= X

J=

- .

In this example the conditional variance of Y given X = xis the constant 1.

Let X and Y be continuous random variables having joint
density f given by (5). In the previous section we computed various
moments involving X and Y. Here we will compute the conditional density
and conditional expectation of Y given X = x.
Example 9.

The marginal density of X is given by

L
1

fx(x)

n(n - 1)

= n( 1 -

(y - x)"- 2 dy

x )"- 1 ,

0 <X< 1,

and fx(x) = 0 elsewhere. Thus for 0 < x < 1,

f(y

I x)

(n - l)(y - x)"- 2
(1 - x)"- 1

0,

X
'

< y < 1,

elsewhere.

7.5.

The Central Limit Theorem

183

Consequently, for 0 < x < 1,

E[Y I X = x]

J:oo yf(y I x) dy

1
1

= (n -

1)(1 - x) 1 -

"

= (n -

1)(1 - x) 1 -

"

= (n _ 1)(l _ x) 1 - ,
=

1)(1 - x)

(n -

n- 1
n

y(y - x)"[(y- x)"-

dy

+ x(y -

[(1 - x)" + x(1 -

x)"n- 1

x)"- 2 ] dy
1
]

+x

+x

It is sometimes convenient to calculate the expectation of Y according to

the formula
(7)

EY

J:oo E[Y I X

= x]fx(x) dx.

This fQrmula follows immediately from (6). For

J~oo E[Y I X=

00

x]fx(x) dx =

J:oo dx J_ yf(x, y) dy
00

J:oo J:oo yf(x, y) dx dy

= EY.

Applying this formula to Example 9, we get

EY =

J: (n -

: + x) n(1 - x)"- 1 dx

J:

=n
=

(1 - x)"- 1 dx -

1-

n+1

n+1

(1 - x)" dx

'

which agrees with the answer found in Example 7.

Naturally, conditional expectations can be defined similarly for discrete
random variables. Some exercises involving this were given in Chapter 4.
7.5.

The Central limit Theorem

Throughout this section X 1 , X 2 , will denote independent, identically

distributed random variables having mean J.l and finite nonzero variance

Expectations and the Central Limit Theotem

184

u 2 We will be interested in studying the distribution of S,. = X 1 +

+ X,.. First of all we note that S,. has mean nJ1 and variance nu2
Suppose next that X 1 has density f. Then, for all n > I, S,. will have a
density Is,. Now Is, = J, and the other densities can be calculated successively by using formulas obtained in Chapters 3 and 6 for the density of
the sum of two independent random variables. We have that
ls,.(x)

L ls,._,(y)l(x

- y)

or

according as X 1 is a discrete or a continuous random variable. For certain

choices of I (e.g., binomial, negative binomial, Poisson, normal, and
gamma), we can find simple formulas for Is,. In general, however, we have
to resort to numerical methods.
One of the most important and most remarkable results of probability
theory is that for large values of n the distribution of S,. depends on the
distribution of X 1 essentially only through p. and u 2 Such a result is more
easily discussed in terms of the normalized random variable

S*

_ S,. - ES,. _ S,. -

" - .Jvar S,.

nJL

u.Jn

'

which has mean 0 and variance I .

To get some idea of how the distribution function of s,. behaves as
n -+ oo, let us first consider a special case in which this distribution
function can be found easily and exactly. Suppose, then, that X 1 is
normally distributed with mean p. and variance u 2 Then, by results in
Chapter 6,
is normally distributed with mean 0 and variance 1 or, in
other words, S,.* has the standard normal distribution function <1>.
Suppose next that X 1 takes values I and 0 with respective probabilities
p and I - p. Then as we saw in Chapter 3, S,. has a binomial distribution
with parameters n and p; that is,

s:

P(S,.

k)

= (~)

pt(l - p)"-t.

It was discovered by DeMoivre (1667-1754) and Laplace (1749-1827)

that in this case the distribution function of s,.* approaches <1>, the standard
normal distribution function, as n -+ oo .
In more recent times there have been many extensions of the DeMoivn>Laplace limit theorem, all known as 'centrallimit theorems." The simplest
and best known of these results was proven by Lindeberg in 1922:
4

7.5.

The Central Limit Theorem

185

Theorem 3 Central Limit Theorem. Let X., X 2 , be independent, identically distributed random variables having mean Jl and
finite nonzero variance u 2 SetS,. = X 1 + + X,.. Then

(8)

lim P
,. .... oo

(Su~nnp ,;; x) = ~x),

- 00

<

<

00 .

The generality of this theorem is remarkable. The random variable X 1

can be discrete, continuous, or neither of these. Moreover, the conclusion
holds even if no moments of X 1 exist beyond the second. Another very
surprising part of the theorem is that the limiting distribution function of
S,.* is independent of the specific distribution of X 1 (provided, of course,
that the hypotheses of the theorem are satisfied). We should not be
surprised, however, that <I> is that limiting distribution. For we have seen
that this is true if X 1 has either a normal or a binomial distribution.
The proof of the Central Limit Theorem will be postponed to Chapter 8,
since it requires advanced techniques yet to be discussed which involve
characteristic functions. It is possible to give an elementary but somewhat
laborious proof of the DeMoivre- Laplace limit theorem, the special case
of the Central Limit Theorem when X 1 is binomially distributed. There
are elementary ways to make the Central Limit Theorem plausible, but
they are not proofs. One such way is to show that, for any positive integer
m, if X 1 has finite mth moment, then

np.)"'

lim E ( 8 " , .... oo

u.Jn

exists and equals the mth moment of the standard normal distribution.
At this stage it is more profitable to understand what the Central Limit
Theorem means and how it can be used in typical applications.
Let X 1 , X 2 , be independent random variables each
having a Poisson distribution with parameter A.. Then by results in
Chapter 4, J1 = u 2 = l and S, has a Poisson distribution with parameter
n.A.. The Central Limit Theorem implies that
Example 10.

lim P
,. .... oo

(s".J-n.A.nl ~

) = <l>(x),

-00

<X<

00.

One can extend the result of this example and show that if X, is a
random variable having a Poisson distribution with parameter A. = t, then
(9)

lim
, ... tO

P(X, - EX, ~ x)
.Jvar X,

Cl(x),

-00

<X<

00 .

Expectations and the Csntral Limit Theorem

186

Equation (9) also holds if X, is a random variable having the gamma

distribution r(t, ..t) for fixed A., or the negative binomial distribution with
parameters rx = t and p fixed.
Normal approximations.
The Central Limit Theorem strongly
suggests that for large n we should make the approximation
7.5.1.

p ( S,(f~nnJ.l

:$;

X) "' <>(x),

<X<

-00

00,

or equivalently
P(S, <

(10)

x) "' ~ (x"~=!')
= <b (x - ES,) '

-00

<X<

00 .

.../Var S,

We will refer to (10) as a normal approximation formula. According to

this formula we approximate the distribution function of S, by the normal
distribution function having the same mean and variance. One difficulty
in applying the normal approximation formula is in deciding how large n
must be for (10) to be valid to some desired degree of accuracy. Various
numerical studies have indicated that in typical practical applications
n = 25 is sufficiently large for (I 0) to be useful.
As an example where normal approximation is applicable, let Xh
X 2 , be independent random variables each having an exponential
density with parameter l = 1. Then (10) becomes
P(S, < x)"'

(11)

~ (x :;n n),

-oo < x < oo.

Graphs showing the accuracy of this approximation are given in Figure 1

for n = 10.
1.0

.8

.6

.4
-

.2

10
Figure 1

True Distribution Function

Normal Approximation

15

20

7.5.

The Central Limit Theorem

187

Suppose the length of life of a certain kind of light bulb,

after it is installed, is exponentially distributed with a mean length of
10 days. As soon as one light bulb burns out, a similar one is installed in
its place. Find the probability that more than 50 bulbs will be required
during a one-year period.
Example 11.

In solving this problem we let Xn denote the length of life of the nth
light bulb that is installed. We assume that X 1 , X 2 , are independent
random variables each having an exponential distribution with mean I 0 or
parameter A. = 1/10. Then Sn = X 1 + + Xn denotes the time when
the nth bulb burns out. We want to find P(S50 < 365). Now S 50 has
mean 50A.- 1 = 500 and variance 50A.- 2 = 5000. Thus by the normal
approximation formula (1 0)
P(S 50 < 365) "' <I>

e6~~OO)

= 4>( -1.91) = .028.

It is therefore very unlikely that more than 50 bulbs will be needed.

Suppose that Sn is a continuous random variable having density fs... If
we diff~rentiate the terms in (10) we obtain

(12)

fs.(X)

0:<

tl~; <p (X tl~=l') , -00

< X<

00.

Though the derivation of (12) is far from a proof, {12) is actually a good
approximation for n large (under the further mild restriction that, for some
n, Is.. is a bounded function).
As an example of this approximation let X 1 be exponentially distributed
with parameter A. = 1, so that (11) is applicable. Then (12) becomes
(13)

1
fs ..(X) ~ .j~

(/)

(x .j-n n) ,

- oo < n < oo.

Graphs showing the accuracy of this approximation are given in Figure 2

for n = 10.
- T r u e Density Function
.15

Normal Approximation

.10

.05

10
Figure 2

15

20

188

Expectations and the Central Limit Theorem

Forms of the Central Limit Theorem involving densities instead of

distribution functions are known as "local" central limit theorems. They
are also important, especially in the advanced theory of probability.
There is an approximation similar to (12) for discrete random variables.
Naturally, a precise statement of such an approximation depends on the
nature of the possible values of S,., that is, those values of x such that
fs"(x) = P(S, = x) > 0. For simplicity we make the following two
assumptions :
(i) If xis a possible value of X 1 , then xis an integer;
(ii) if a is a possible value of X" then the greatest common divisor of

the set
{x - a I xis a possible value of Xd
is one.
We exclude, for example, a random variable X1 such that P(X1 = 1) =
P(X1 = 3) = 1/2, for then the greatest common divisor of the indicated
set is 2. Under assumptions (i) and (ii), the approximation
(14)

x an integer,

is valid for large n.

Let X1 be the binomial random variable taking on -values
1 and 0 with probabilities p and 1 - p respectively. Then (i) and (ii)
hold and (14) is applicable with J.L = p and a2 = p(I - p). Since S,. has
the binomial distribution with parameters n and p, we have the approximation
Example 12.

(15)

fs"(x) =

(~) i~:(l
1

- p)"-x
(

x - np

)np(l - p) q> ) np(l - p) '

x an integer.

This approximation is plotted in Figure 3 for n = 10 and p = .3.

From Figure 3 we are led to another method for approximating fs"(x)
in the discrete case, that is, the integral of the right side of (14) over the set
[x - 1/2, x + 1/2]. By expressing this integral in terms of <I> we obtain
as an alternative to (14)
(16)

fs.(x)

~ ~ (x + (~3:-

"Jl)

_"" (x -- (1/2) - np,)

"'

u.J~

'

x an integer.

7.5.

189

The Central Limit Theorem

.3

.2

.1

10

Figure 3

The area of the shaded region of Figure 3 is an approximation to

P(S,. = 5).
Finally, if we sum (16) on the set { ... , x - 2, x - I, x} we are led to
the approximation
11
(17)
P(S. ,;; x) "' <I> ("' + (~3~) ,
x an integer.

"

When S,. is discrete and conditions (i) and (ii) hold, then (17) is usually
more accurate than is the original normal approximation Formula (10).
In Figure 4 we compare the approximations in Formulas (10) and (17)
when S,. has the binomial distribution with parameters n = 10 and
p = .3.
1.0
.8

.6
.4

.2
0
0

10

Figure 4

A certain basketball player knows that on the average he

will make 60 percent of his freethrow attempts. What is the probability
that in 25 attempts he will be successful more than one-half of the time?
Example 13.

We will interpret the problem as implying that the number S,. of successes
in n attempts is binomially distributed with parameters n and p = .6.

Expectations and the Central Limit Theorem

190

Since P(S,. ~ x)
approximation

1 - P(S,. ~ x - 1) we are \ed from (17) to the

P(S, ;;;, x) "' 1 - C!> (x -

(18)

In our case np.

(~~ - np} ,

= 25(.6) = 15 and u.J~ = .J25(.6)(.4)

P(S2s > 13) ~ 1 - <b

(13 - (1/2) -

x an integer.
= 5.J.24. Thus

15)

s.J.24

1 - <b( -1.02)

= <1>(1.02) = .846.
The Central Limit Theorem and the
corresponding normal approximation formulas can be regarded as refinements of the Weak Law of Large Numbers discussed in Chapter 4. We
recall that this law states that for large n, S,./n should be close to p. with
probability close to 1. The weak law itself, however, provides no information on how accurate such an estimate should be. As we saw in Chapter
4, Chebyshev's Inequality sheds some light on this question.
The normal approximation formula ( 10) is also useful in this context.
For c > 0
7.5.2.

Applications to sampling.

(I~" - P
I ~ c)

= P(S,. :S np. -

., "'c;~)
= 2 [1 _

nc)

+ P(S,.

np.

nc)

"'cj~)
C!> (c:~} J.
+1-

In other words
(19)
where

(20)
A sample of size n is to be taken to determine the percentage of the population planning to vote for the incumbent in an election.
Let X 1 = 1 if the kth person sampled plans to vote for the incumbent and
Xk = 0 otherwise. We assume that X., .. . , X,. are independent, identically
distributed random variables such that P(X1 = 1) = p and P(Xl = 0) =
1 - p. Then p. = p and u 2 = p(I - p). We will also assume that pis
Example 14.

close enough to .5 so that u =

.Jp(1 -

p) can be approximated satis-

7.5.

The Central Limit Theorem

191

factorily by u ~ 1/2 (note that a has a maximum of 1/2 at p = .5, and

that, asp ranges over .3 ~ p < .7, u stays above .458 which is close to
1/2). The random variable S,jn denotes the fraction of people sampled
that plan to vote for the incumbent and can be used to estimate the true
but unknown probability p. We will use normal approximations to so]ve
the following three problems:
(i) Suppose n

= 900.

Find the probability that

I~"-

PI ~

.025.

(ii) Suppose n = 900. Find c such that

(iii) Find n such that

Solution to (i). By (20)

(J = ( .025).J900 = 1. 5'

.5
so by (19)
p

(I~'! - PI ~

.025)

~ 2(1

~1.5))

= 2(.067) = .134.
Solution to (ii). We first choose lJ so that 2(1 - ~(/J)) = .01 or
cJ)(5) = .995. Inspection of Table I shows that 15 = 2.58. Solving (20) for
c we get
= <5u = (2.58)(.5) =
043
c

J~

,J9oo

Solution to (iii). As in (ii) we have lJ = 2.58. Solving (20) for n we find

n

IJ2u2
2

= (2.58)2(.25) = 2663.
(.t25) 2

It is worthwhile to compare the results of (ii) and (iii). In both cases

In (ii), c = .043 and n = 900, while in (iii), c = .025 and n = 2663. In

going from (ii) to (iii), in order to decrease c by the factor 43/25 we are
forced to increase n by the square of that factor. This is true generally

Expectations and the Central Limit Theorem

192

whenever we want to keep

p (

~n- ~1 ~c)

constant. For then lJ is determined by (19) and, from (20), n is related to c

by n = lJ 2u 2 fc 2 In the same context, if we increase n by some factor, we
decrease c only by the square root of that factor.

Exercises

1 Let

X have a Beta density with parameters cx 1 and cx2 Find EX.

2 Let X and Ybe independent random variables having respective gamma

densities r(cx 1 , A.) and r(cx2 , A.). Set Z = Y/X. For which values of cx 1

and cx2 will Z have finite expectation? Find EZ when it exists. Hint:
See Theorem 3 of Chapter 6 and related discussion.
3 Let X have the normal density n(O, u 2 ). Find EIXI. Hint: Use the

result of Exercise 31 of Chapter 5.

4 Let X have an exponential density with parameter A. and let X, be

defined in terms of X and e > 0 by (1). What is the distribution of

X,/e? Find EX, and evaluate its limit as e -+ 0.
5 Let X have a Beta density with parameters cx 1 and cx2 Find the

moments and the variance of X.

6 Let X have a

of y =

rx.

x2 distribution with n degrees of freedom.

Find the mean

7 Let X be the random variable as in Example 7. Find EXm from the

marginal density fx
8

Let Z be as in Exercise 2. Find the variance of Z.

9 Let U1 and U2 be independent random variables each having an

exponential density with parameter A., and set Y = max (U1 , U2 ) .

Find the mean and variance of Y (see Section 6.5).

10

Let X be the random variable from Example 1 of Chapter 5. Find the

mean and variance of X.

11

Let X be the random variable from Example 1 of Chapter 6. Find the

mean and variance of X. Hint: Reduce EX 2 to a Beta integral.

12 Find the mean and variance of the random variable Z from Exercise 17

of Chapter 6.
13

Find the mean and variance of the random variable Y from Exercise 28
of Chapter 6.

14 Let X be the sine of an angle in radians chosen uniformly from

( -7t/2, 7t/2). Find the mean and variance of X.

Exercises

193

15 Let X have the normal density n(O, u 2 ). Find the mean and variance of

each of the following random variables:

(a) lXI;
(b) X2;
(c) erx.
16 Let

X have the gamma density r((X, A.). For which real t does etX have

finite expectation? Find Eerx for these values of t.

17 Let X have the gamma density r(ct, A.). For which real numbers r does
have finite expectation? Find Er for these values of r.

xr

18 Let X be a nonnegative continuous random variable having density

and distribution function F. Show that X has finite expectation if and

only if

J (1 00

F(x)) dx < oo

and then
EX =

fooo

(1 - F(x)) dx.

Hint: See the proof of Theorem 5 of Chapter 4.

19 Let Xk be the kth order statistic in a sample from random variables

U1 , . , U, which are independent and uniformly distributed over

(0, 1). Find the mean and variance of Xk.
20 Let X and Y be as in Example 7 and let R

Y - X. Find the mean

and variance of R. Hint: Use Equation (16) of Chapter 4.

21

Let X and Y have density f as in Exercise 9 of Chapter 6. Find the

correlation between X and Y.

22 Let X and Y be independent random variables such that X has the

normal density n(p., u 2 ) and Y has the gamma density r(o:, A.). Find
the mean and variance of the random variable Z = XY.
23 Let X and Y be random variables having mean 0, variance 1, and
correlation p. Show that X - p Y and Y are uncorrelated, and that
X - p Y has mean 0 and variance I - p 2
24 Let X, Y, and Z be random variables having mean 0 and unit variance.
Let p 1 be the correlation between X and Y, p 2 the correlation between
Y and Z, and p 3 the correlation between X and Z. Show that

Pa ~ P1P2 -

.Jt -

pf

.Jt -

p~.

Hint: Write
XZ = [p 1 Y +(X- P1Y)][p2Y

+ (Z-

P2Y)],

and use the previous problem and Schwarz's inequality.

25

Let X, Y, and Z be as in the previous problem. Suppose p 1 > .9 and

p 2 =::: .8. What can be said about p 3 ?

Expectations and the Central Limit Theorem

194

26 Let X and Y have a density f that is uniform over the interior of the
triangle with vertices at (0, 0), (2, 0), and {1, 2). Find the conditional
expectation of Y given X.
27 Let X and Ybe independent random variables having respective gamma

densities r(a 1 , A.) and r(a2 , A.), and set Z = X + Y. Find the conditional expectation of X given Z.
28 Let n and Y be random variables as in Exercise 26 of Chapter 6.
Find the conditional expectation of II given Y.
29 Let X and Y be continuous random variables having a joint density.
Suppose that Y and ~(X) Y have finite expectation. Show that

Ecp(X)Y

= J~>

cp(x)E[Y I X

x]fx(x) dx.

30 Let X and Y be continuous random variables having a joint density,

and let Var [ Y I X = x] denote the variance of the conditional
density of Y given X= x. Show that if E[Y I X= x] = p, independently of X, then EY = p. and

VarY =
31 Let

J~> Var [YI X=

x]fx(x) dx.

xh x2,...

be independent, identically distributed random

variables ha~ing mean 0 and finite nonzero variance u 2 and set
S,. = X 1 + + X,.. Show that if X 1 has finite third moment, then
ES! = nEXf and
8 3
lim E ( ;-) = o,

uv n

,. ... >

which is the third moment of the standard normal distribution.

32 Let X., ... , X,., and S,. be as in Exercise 31. Show that if X 1 has finite
fourth moment, then

Es: =

nEXt + 3n(n - 1)u4

and
lim E (
,. ... oo

;-) = 3,

av n

which is the fourth moment of the standard normal distribution.

Hint: The term 3n(n - 1) comes from the expression

(~) 2;~,.
33 Let X have the gamma density r(a, A.). Find the normal approximation
for P(X ~ x).
34

Let X1 , X 2 , . . . be independent, normally distributed random

variables having mean 0 and variance u 2

Expectations and the Central Limit Theorem

196

Suppose candidate A from the preceding problem insisted that the

sample size be increased to a number n such that if 51% of all voters
favored him he could be 95% sure of getting a majority of the votes
sampled. About how large would n have to be?
46 Solve Exercise 27 of Chapter 4 by using normal approximation.
45

Moment Generating
Functions and
Characteristic
Functions

Some of the most important tools in probability theory are borrowed from other
branches of mathematics. In this chapter we discuss two such closely related tools.
We begin with moment generating functions and then treat characteristic functions.
The latter are somewhat more difficult to understand at an elementary level because
they require the use of complex numbers. It is worthwhile, however, to overcome
this obstacle, for a knowledge of the properties of characteristic functions will
enable us to prove both the Weak Law of Large Numbers and the Central Limit
Theorem (Section 8.4).
8.1.

Moment generating functions

The moment generating function Mx(t) of a random variable X

defined by
Mx(t) = Eex.

IS

The domain of M x is all real numbers t such that etx has finite expectation.
Example 1.

Let X be normally distributed with mean p. and variance

Then

et(y+p) - 1---e -yl/2tll d y

oo

u.J2n

-oo

-Ut

e-

foo

1 ety- (yl/2t1l) d y.
u.J2n
---

-oo

Now

Consequently
pt t~ltl/2
M .X\'t) =ee

fco
- oo

197

1
-[(y-t~lt)l/2t12] dt.
-_-e
u.J2n

Moment Generating Functions and Characteristic Functions

198

Since the last integral represents the integral of the normal density
n(u2 t, u 2 ), its value is one and therefore
-oo < t < oo.

(1)
Example 2.

Let X have the gamma density with parameters ex and A..

Then

J.ex>

= -A_-

r(oc)

11-1

e -(l-t)x d X

A11
r(cx)
= - -...0........:..r(oc) (l - t)
for - oo < t
(2)

< A.. The integral diverges for A.

Mx(t)

(-A.
),
A. -

S: t

< ro. Thus

-oo<t<A..

Suppose now that X is a discrete random variable, all of whose possible

values are nonnegative integers. Then
<X>

Mx(t) =

e"'P(X

n).

11=0

In Chapter 3 we defined the probability generating function for such

random variables as
<X>

<})x(t)

= L

t 11P(X

n).

11=0

From these two formulas it is clear that

M x(t) = ~x(e').

(3)

Formula (3) allows us to determine the moment generating function directly

from the probability generating function. For example, if X has a binomial
distribution with parameters n and p, then as was shown in Example 16 of
Chapter 3,
Cl>x{t) = (pt + 1 - p)11
It follows immediately that
Mx(t) = (pe'

1 - p)11

Similarly, if X has a Poisson distribution with parameter A., then according

to Example 18 of Chapter 3,
~ x(t)

Consequently~

eA(t- 1

>.

8. 1.

199

Moment generating functions

Of course, in these two examples Mx(t) could also easily be obtained

directly from the definition of the moment generating function.
- If X and Y are independent random variables, then erx and err are also
independent. Consequently

Mx+rtt) = Et<X+Y> = EetXetY = Ee'xEe'1

= Mx(t)My(t).
It follows easily that if X 1 ,

. .. ,

X,. are independent and identically

distributed, then

(4)
In order to see why Mx(t) is called the moment generating function we
write

L -t"X"
.
n!
oo

Mx(t) = EetX = E

n=O

Suppose Mx(t) is finite on -t0 < t < t0 for some positive number t0
In this case one can show that in the last expression for Mx(t) it is permissible to interchange the order of expectation and summation. In other
words
(5)

Mx(t) =

oo

EX"

I: n =O n!

t"

for -t0 < t < t0 In particular, if Mx(t) is finite for all t, then (5) holds
for all t. The Taylor series for Mx(t) is

Mx(t) =

(6)

t ... d"
I .
I:
Mx(t)
rt=O n! dt"
t=O
00

By comparing the coefficients oft" in (5) and (6), we see that

d"
Mx(t)
dt"

EX"= -

(7)

t=O

Let X be normally distributed with mean 0 and variance

Use moment generating functions to find the moments of X.

Example 3.

Observe first from (1) that

Mx(t)

= e(l2t2f2 =

(u2t2)" _!_

n=O

n!

Thus the odd moments of X are all zero, and the even moments are given by

EX 211

u2"

-- = (2n)!
2"n!

200

Moment Generating Functions and Characteristic Functions

or
EX 2n

= 0'2"(2n)! .
2"n!

This agrees with the result obtained in Chapter 7.

This example can also be used to illustrate (7). Since
d crlt2f2 =O'e
2t crltl/2
-e
dt

and

it follows that

and

which are just the first two moments of X.

8.2.

Characteristic functions

The characteristic function of a random variable X is defined as

(/)x(t)

Ee''x.

-oo < t < oo,

where i = .J- 1. Characteristic functions are slightly more complicated

than moment generating functions in that they involve complex numbers.
They have, however, two important advantages over moment generating
functions. First, q>x(t) is finite for all random variables X and all real
numbers t. Secondly, the distribution function of X and usually the density
function, if it exists, can be obtained from the characteristic function by
means of an "inversion formula." Using properties of characteristic
functions we will be able to prove both the Weak Law of Large Numbers
and the Central Limit Theorem, which we would not be able to do with
moment generating functions.
Before discussing characteristic functions we will first briefly summarize
some required facts involving complex variables.
We can write any complex number z in the form z = x + iy, where x
and y are real numbers. The absolute value lzl of such a complex number
is defined by lzl = (x 2 + y 2 ) 112 The distance between two complex
numbers z1 and z2 is defined to be lz1 - z2 1.
If a function of a real variable has a power series expansion with a
positive radius of convergence, we can use that power series to define a
corresponding function of a complex variable. Thus we define

8.2.

Characteristic functions

201

for any complex number z. The relation

remains valid for all complex numbers z 1 and z 2 Letting z = it, where t
is a real number, we see that

e''

(it)"

n!

n=O

- (1
-

(1 -

lt -

it 3

it s

2 - 3! + 4! + 5! - ...

~~ + ~~ -

... ) +

(t - ~~ + ~~ - ... ) .

Since the two power series in the last expression are those of cos t and
sin t, it follows that
e1t = cos t

(8)

+ i sin t.

Using the fact that cos (- t) = cos t and sin (- t) = -sin t, we see that

e -It = cos t - i sin t.

From these formulas we can solv,e for cos t and sin t, obtaining
(9)

e't + e-ft

and

cost= - - 2

e't

e-it

smt = - - 2i

It also follows from (8) that

le1tl

(cos2 t

sin 2 t) 1 ' 2

1.

If f(t) and g(t) are real-valued functions oft, then h(t) = f(t) + ig(t)
defines a complex-valued function of t. We can differentiate h(t) by
differentiating f(t) and g(t) separately; that is,
h'(t)

= f'(t) +

ig'(t),

provided thatf'(t) and g'(t) exist. Similarly we define

J:

h(t) dt =

J.b f(t) dt + i

J:

g(t) dt,

provided that the indicated integrals involving/ and g exist. The formula

is valid for any complex constant c. The fundamental theorem of calculus

continues to hold and, in particular, if c is a nonzero complex constant,

202

Moment Generating Functions and Characteristic Functions

then

A complex-valued random variable Z can be written in the form

Z = X + i Y, where X and Y are real-valued random variables. Its
expectation EZ is defined as
EZ

= E(X +

i Y)

= EX + iEY

whenever EX and EY are well defined. Just as for real-valued random

variables, Z has finite expectation if and only if EIZI < oo, and in that
case

IEZI

EIZI.

The formula

E(a 1 Z 1

+ a2 Z 2 ) = a1EZ1 +

a2 EZ2

is valid whenever a 1 and a2 are complex constants and Z 1 and Z 2 are

complex-valued random variables having finite expectation.
We will let X and Y, with or without subscripts, continue to denote realvalued random variables. Thus in the phrase "let X be a random
variable ..." it is understood that X is real-valued.
Suppose now that X is a random variable and tis a constant (we reserve
the symbol t for real constants). Then le1'xl = I, so that e1'x has finite
expectation and the characteristic function (/Jx(t), - oo < t < oo, given
by
-oo < t < oo,
is well defined. We see that (/Jx(O) = Ee0 = El = 1 and, for - oo <
t < oo,
l<px(t)l = IEe1'xl ~ Ele1'xl = El = 1.
The reason characteristic functions are finite for all t whereas moment
generating functions ar~ not finite in general is that e~r, - oo < t < oo, is
bounded while e', - oo < t < oo, is unbounded.
Let X be a random variable taking on the value a with
probability one. Then
Example 4.

-00

<

<

00 .

In particular, if X takes on the value zero with probability one, then its
characteristic function is identically equal to 1.
If X is a random variable and a and b are real constants, then
<l'a+bx(t) = Ee't(a+bX)

=
=

Eeiraeibtx
e''IJEeibr x,

8.2.

Characteristic functions

203

and hence
(10)
Example 5.

-00

< t<

00.

Let U be uniformly distributed on ( -1, 1). Then for

t ::1: 0

J
1

(/Ju(t) =

-1

"!2 du

11

eit"ll

2 it

-1

siin t

=-t

For a< b let

Then Xis uniformly distributed on (a, b), and by (10) fort '# 0
_

lt(a+b)/2

(/Jx( t ) - e

sin ((b - a)t/2)

(b - a)t/2

Alternatively
q>x(t) =

f" e1'x l dx
Ja b- a
l
eitxlb

=b-aUa
it(b - a)

It is easy to check by means of (9) that th~e two answers agree.

Example 6.

Let X have an exponential distribution with parameter )..

Then
q>x{t) =

fooo e1'x;.e-.tx dx

= ).

fooo e-<J.-ir)x dx

- A e-<J.-Ir)xlo.
A - it
oo

Since limx... oo e-J.x = 0 and e1"' is bounded in x, it follows that

Moment Generating Functions and Chsrscteristic Functions

204

Thus
qJ:x(t)

..t
..t -

..

lt

Suppose X and Y are independent random variables. Then e1'x and

e1'r are also independent random variables; consequently
(/Jx+r{t)

Ee''<X+r> = Ee''xeltY

Ee''xEe''r

and hence
(11)

(/Jx+r(t) = (/Jx(t)qJy(t),

-00

< t <

00.

Formula (11) extends immediately to yield the fact that the characteristic
function of the sum of a finite number of independent random variables
is the product of the individual characteristic functions.
It can be shown that ({Jx(t) is a continuous function oft. Moreover, if X
has finite nth moment, then qJ<;>(t) exists, is continuous in t, and can be
calculated as
cp<">(t)
X

d" EeitX

dt"

E d/1 eitX
dt"

E(iX)"eitx.

In particular
qJ<;>(O) = i"EX".

(12)

We can attempt to expand (/Jx(t) into a power series according to the

formula
CfJx(t) = Ee"x

(13)

(itX)"

n!

11=0

11=0

i"EX" t".

n!

Suppose that
Mx(t) =

Lco EX"
- t"
rt=O

n!

is finite on - t 0 < t < t 0 for some positive number t 0 Then (13) also
holds on -t0 < t < t 0
Example 7.

Let X be normally distributed with mean 0 and variance

Find fPx(t).

From Chapter 7 we know that EX" = 0 for any odd positive integer n.
Furthermore, if n = 2k is an even integer, then
EX"

EXn

= u2k(2k)! .
2kk!

Therefore

8.3.

Inversion formulas and the Continuity Theorem

205

More generally let X be normally distributed with mean p. and variance

u Then Y = X - p. is normally distributed with mean 0 and variance
u 2 Since X= p. + Y we see from Formula (10) and Example 7 that
(14)
(t) _ itp -a"lt~/2
-oo < t < oo.
2

lf>x

- e e

Let X be a random variable whose mo.ment generating function Mx(t)

is finite on - t0 < t < t0 for some positive number t 0 Since
Mx(t) = Ee'x

and

lfJx(t)

Eettx,

we would expe.c t that

(15)

lfJx(t) = Mx(it).

In other words, we would expect that if we replace t by it in the formula

for the moment generating function, we will obtain the corresponding
formula for the characteristic function. This is indeed the case, but a
thorough understanding of the issues involved requires a sophisticated
concept (analytic continuation) from complex variable theory.
As an example of (15), let X be normally distributed with mean Jl and
variance u 2 Then as we have already seen
Mx(t) = ell'ea2t2f2

and hence
iut -a2t2f2
= e"'e

which by (14) is lfJx(t).

8.3.

Inversion formulas and the Continuity Theorem

Let X be an integer-valued random variable. Its characteristic function

is given by
00

lf>x(t) = ~ e'i'Jx(j).
-co

One of the most useful properties of q>x(t) is that it can be used to calculate
fx(k). Specifically we have the "inversion formula"
(16)

fx(k) = _!_
27t

Jx

e-iktlfJx(t) dt.

-x

In order to verify (16) we write the right side of this formula as

_!_

2n

fx

-x

e-ikt [

-co

e'i'JxU>] dt.

206

Moment Generating Functions and Characteristic Functions

A theorem in integration theory justifies interchanging the order of

integration and summation to yield the expression

fxU)

_!_ f~r
21t

-<Xl

ei<i-ll;)t

dt.

-n

In order to complete the proof of (16) we must show that the last expression
equals fx(k). To do so it is enough to show that

_!_

(17)

27t

JK

ei<i-k)t

dt = { 1

-x

if j = k,
if j ::1: k.

Formula (17) is obvious whenj = k, for in that case e'<i-">' = I for all t.
If j :F k, then
7t
l(j- ll;)tlw
-1
e'<i-k)t dt = e
-K
21t - 7t
21ti(j - k)

- - - - - --21ti(j - k)

sin (j - k)n
7t(j -

= O,

k)

since sin m1t = 0 for all integers m. This completes the proof of (17) and
hence also of ( 16).
Let X 1 , X 2 , . , X, be independent, identically distributed
integer-valued random variables and set S,. = X 1 + + X,. Then
(/Jsn(t) = (q>x,(t))", and consequently by (16)
Example 8.

(18)

fs"(k) =

_!_

21t

fn

e-lkt(({Jx 1(t))"

dt.

-7r

Formula (18) is the basis of almost all methods of analyzing the behavior
of fs"(k) for large values of n and, in particular, the basis for the proof of
the "local" Central Limit Theorem discussed in Chapter 7.
There is also an analog of (16) for continuous random variables. Let
X be a random variable whose characteristic function q>x(t) is integrable,
that is,

I~

oo

lcpx(t)l dt < oo.

It can be shown that in this case X is a continuous random variable having

a density fx given by
.
fx(x) = - 1 J<Xl e-'"'q>x(t)
dt.
21t - <Xl

(19)

Let X be normally distributed with mean 0 and variance

We will show directly that (19) is valid for such a random variable.

Example 9.

8.3.

207

Inversion formulas and the Continuity Theorem

From Example 7 we know that X has characteristic function Cf>xCt)

e -(1'1.tlf2. Thus by the definition of characteristic functions,

-flltl/2 _

Joo eir.x J1
-.xl/2fll d
e
x.
- oo
u 2n

If we replace t by - t and u by 1/u in this formula it becomes

-tl/2f12

foo

-ir.x

(1

;- e

-(12.xlf2

v 2n

- 00

or equivalently,
1-- e -r2/ 2f12
-u.J2n

foo

1
2n

= -

oo

e -it.xe -fll.xl/2 d x .

Finally, if we interchange the role of the symbols x and t in the last

equation we obtain

- 1--- e -.xlf2t12 = - 1 foo e -it.xe -tl2tlf2 d t,

uJ2n
2n - oo
which is just (19) in this special case.
Let X now denote any random variable. Let Y be a random variable
that is independent of X and has the standard normal distribution, and let
c denote a positive constant. Then X + c Y has the characteristic function
(t) -cltlf2
Cf>x

Since cpx(t) is bounded in absolute value by 1 and e-c21212 is integrable, it

follows that X + c Y has an integrable characteristic function. Consequently (19) is applicable and X + c Y is a continuous random variable
having a density given by
fx+cr(x) =

_!_ Joo e-itxcpx(t)e-c2tlf2 dt.

2n -

oo

If we integrate both sides of this equation over a < x < b and interchange the order of integration, we conclude that
P(a < X

c Y < b)

= _!_ J.b
2n

_!_
2n

(foo

-oo

e-lt.x(/Jx(t)e-c21212 dt) dx

Joo ( rb e-it.x dx) (/Jx(t

-oo Ja

)e-cltl/2

dt

or
(20)

P(a

~ X + cY ~

b)

= _!_ foo
21t

-oo

(e-ib' - . e-iar) (/Jx(t)e-cltl f2 dt.

- Jt

The importance of (20) is that it holds for an arbitrary random variable X.

The right side of (20) depends on X only through Cf>x(t). By using this
fact and letting c -+ 0 in (20), one can show that the distribution function

Moment Generating Functions and Characteristic Functions

208

of X is determined by its characteristic function. This result is known as a

"uniqueness theorem" and can be stated as follows:
Theorem 1 If two random variables have the same characteristic
function, they have the same distribution function .

Use the uniqueness theorem to show that the sum of two

independent normally distributed random variables is itself normally
distributed.
Example 10.

Let X and Y be independent and distributed respectively according to

n(Jl 1 , ui) and n(p.2 , ui). Then
and
Consequently
Thus the characteristic function of X + Y is the same as that of a random
variable having a normal distribution with mean p. 1 + p.2 and variance
uf + ui. By the uniqueness theorem X+ Y must have that normal
distribution.
The most important application of the inversion formula (20) is that it
can be used to derive the next result, which is basic to the proof of the
Weak Law of Large Numbers and the Central Limit Theorem.
Theorem 2

(21)

Let X,., n > I, and X be random variables such that

lim lfJx,.(t) = cpx(t),

- 00

,.-.oo

<

<

00.

Then

(22)

lim FxJx)
,. ..... 00

= Fx(x)

at all points x where Fx is continuous.

This theorem states that convergence of characteristic functions implies

convergence of the corresponding distribution functions or, in other
words, that distribution functions "depend continuously" on their
characteristic functions. For this reason Theorem 2 is commonly known
as the "Continuity Theorem."
The proof of this theorem is rather involved. We will not present the
details of the proof, but we will indicate briefly some of the main ideas of
one method of proof.
We first choose a random variable Y that has the standard normal
distribution and isindependent of each of the random variables X,., n > 1.

8.4.

209

The Weak Law of Large Numbers and the Central Limit Theorem

Let a < b and let c be a positive constant. Then by the inversion

formula (20)
(23)

P(a

X,.

+ cy ~

b) = __!__

21t

and
(24)

P(a ~ X

cY ~ b)

= -1

oo (e-ibr -. e-iat) (/Jx"(t)e-c2t2f2 dt

-oo
-lt

oo ( -ibt
-iat)
22
e
-. e
cpx(t)e-c ' ' 2 dt.
-oo
-lt

21t

By assumption (/Jx"(t) -+ ({Jx(t) as n -+ oo. It follows from this by a

theorem in integration theory that the right side of (23) converges to the
right side of (24) as n -+ oo. Consequently
(25)

lim P(a

X,.

cY < b)

= P(a

~ X

+ cY

~ b).

n-+oo

There are two more steps to the proof of the theorem. First one must
show (by letting a -+ - oo in (25)) that
(26)

lim P(X,.

cY < b) = P(X

cY < b).

n-+oo

Finally, one must show (by letting c

-+

lim P(X, < b)

0 in (26)) that

= P(X

< b)

n-+oo

whenever P(X == b) == 0. The last result is equivalent to the conclusion

of the theorem.
8.4.

The Weak law of large Numbers and the Central limit Theorem

In this section we will use the Continuity Theorem to prove the two
important theorems in probability theory stated in the title to this section.
Both theorems were discussed without proof in earlier chapters. In order
to prove these. theorems we first need to study the asymptotic behavior of
log CfJxCt) near t = 0.
Let z be a complex number such that lz - 11 < 1. We can define
log z by means of the power series
log z

= (z

- 1) - (z - 1)2

(z - 1)3

(for lz - 11 ~ 1, other definitions of log z are needed). With this definition

we have the usual properties that log I
0,

elog z

z,

lz - 11 < 1,

and if h(t), a < t < b, is a differentiable complex-valued function such

that lh(t) - 11 < 1, then
d
h'(t)
- log h(t) = .

dt

h(t)

Moment Generating Functions and Characteristic Functions

210

Let X be a random variable having characteristic function ({Jx(t). Then

(/Jx(t) is continuous and (f}x(O) = I. Thus log (f}x(t) is well defined for t
near 0 and log ({Jx(O) = 0.
Suppose now that X has finite mean Jl. Then q>x(t) is differentiable and,
by (12), cp X(O) = iJl. Hence
. log ({Jx(t) - log (/Jx(O)
. log lfJx(t) = tm
I1m
1
, ... o
t
, ... 0
t - 0
_.;:___;_:_~--~..=...c..-

= .!!.._ log cpx(t) l

dt

t =O

= <l'x(o)
(/Jx(O)
= ijl.

Consequently,
lim log (/Jx(t) - i}lt = O.
t-+0
t

(27)

Suppose now that X also has finite variance u 2 Then ({Jx(t) is twice
differentiable and by (12)
cp;(O) = -EX 2 = - (Jl 2

+ u 2 ).

We can apply !'Hospital's rule to obtain

r1m log (/Jx(t)

, ... o

t2

- ijlt

(/Jx<t)
.
- - - lJl
(/Jx(t)
1m _;....--t-t-+0
2

= lim ({Jx(t) - iJLq>x(t)

t- o
2tcpx(t)

= lim (/Jx(t)

- iJlq>x(t) .

2t

t-+0

By a second application of !'Hospital's rule we see that

-(JL2

u2) _ (i/l)2

In other words
(28)

8.4.

211

The We,ak Law of Large Numbers and the Central Limit Theorem

Theorem 3 (Weak Law of Large Numbers). Let X 1 , X 2 , .

be independent, identically distributed random variables having finite
mean Jl and setS, = X1 + + X,. Then for any e > 0

!~~ p (I~" - Jll > e)

(29)

= 0.

Proof The characteristic function of

S
n

-II- j l =

X 1 ++XII - jl
n

IS

e- ip.t((/)x ,(tfn))".

Let i be fixed. Then for n sufficiently large, t/n is close enough to zero so
that log (/Jx 1(tfn) is well defined and
(30)

e-ip.t(q>x 1(tfn))" = exp [n(log (/Jx 1(tfn) - iJl(t/n))].

We claim next that

(31)

lim n(log (/Jx,(tjn) - iJl(t/n)) = 0.

Equation (31) is obvious for t = 0 since log

t # 0 we can write the left side of (31) as

t Ilffi
, .... oo

(/Jx 1 (0) =

log I = 0. If

log Cf'x(t/n) - ijl(tjn)

t/n

But tfn -+ 0 as n -+ oo, so the last limit is 0 by (27). This completes the
proof of (31 ). It follows from (30) and (31) that the characteristic function
of

approaches 1 as n -+ oo. Now 1 is the characteristic function of a random

variable X such that P(X = 0) = 1. The distribution function of X is
given by
if X 2: 0,
if X < 0.
This distribution function is continuous everywhere except at x = 0.
Choose e > 0. Then by the Continuity Theorem,
(32)

8
lim P ( " - Jl
, .... oo
n

-e)

= Fx.( -e) = 0

and

8
lim P ( " - Jl

w-+oo

e) =

Fx(e) = 1.

212

Moment Generating Functions and Characteristic Functions

The last result implies that

8
lim P ( " - J1. >
,. .... w
n

e) = o,
I

which together with (32) implies that (29) holds as desired.

For the next theorem it is necessary to recall that c:I>(x) denotes the
standard normal distribution function given by

f"

c:I>(x) =

-oo

~ e-y2f2 dy,

-00

.J21t

< X<

00.

We recall also that this distribution function is continuous at all values

of x .

Theorem 4 (Central Limit Theorem). Let X 1 , X 2 , be

independent, identically distribute.d random variables each having mean
J1. and.finite nonzero variance a 2 Then
lim P
,. ... oo

(S,.' a JnnJl. ~ x) =

<ll(x},

-OO

<X<

00.

Proof Set
S* _ S,. - nJl.
"
a.Jn

Then for t fixed and n sufficiently large,

(/)s=(t) = e-i,.Jltf~.;~(/Js"(t/u.J~)
= e-inJltfrr./~(({Jx1(tja.J~))",

or
(33)

q>5 =(t)

= exp [n(log (/)x 1(t/(J.J~)

- iJJ.(tfu.J~))] .

We claim next that

(34)

lim n(log ({Jx 1 (t/u.J~) - iJJ.(t/u.J~)) =

,. ... w

If t = 0, then both sides of (34) equal zero and (34) clearly holds. If
t =F 0 we can write the left side of (34) as
t 2 . log q>x.(tfu.J~) - iJJ.(t/uJ~)
2 1lffi
I
'
a ,.... oo
(tfuy n)2

which by (28) equals

213

Exercises

Thus (34) hQlds for all t. It follows from (33) and (34) that
-00

<

<

00 .

n-+OO

According to Example 7, e-'212 is the characteristic function of a random

variable X having the standard normal distribution function <l>(x). Thus
by the Continuity Theorem
lim P(Sn* < x)

,.-co

= <l>(x),

-oo < x < oo,

which is the desired conclusion.

Exercises

b). Find Mx(t).

2 Express the moment generating function of Y = a + bX in terms of
Mx(t) (here a and bare constants).

1 Let X be uniformly distributed on (a,

3 Let X have a Poisson distribution with parameter A.. Use moment

generating functions to find the mean and variance of X.

4

Let X have a negative binomial distribution with parameters rx and p.

(a) Find the moment generating function of X.
(b) Use this moment generating function to find the mean and variance
of X.

5 Let X be a continuous random variable having the density fx(x) =

(1/ 2)e-lxl, -oo < .X< 00.
(a) Show that Mx(t) = 1/(1 - t 2 ), -1 < t < 1.

(b) Use this moment generating function to find a formula for EX2 n
(note that the odd moments of X are all zero).
6

Let X have a binomial distribution with parameters n and p.

(a) Find dMx(t)jdt and d 2 Mx(t)jdt 2
(b) Use (a) and Formula (7) to compute the mean and variance of X.
. . . , Xn be independent, identically distributed random
variables such that Mx,{t) is finite for all t. Use moment generating
functions to show that

7 Let X1 ,

E(X 1

+ .. +

Xn)3 = nEX~

+ n(n

3n(n- 1)EXfEX 1

- l)(n - 2)(EX 1) 3

Hint : Find (d 3 /dt 3 )(Mx 1 (t))nlr=O

8 Let X be a random variable such that Mx(t) is finite for all t. Use the

same argument as in the proof of Chebyshev's Inequality to conclude

that
t > 0.

Moment Gene1ating Functions and Cha!actefistic Functions

It follows that
~

P(X ;;:: x)

min e-':xMx(t),
t:i!':O

provided that e-'xMx(t) has a minimum on 0 < t < oo.

9 Let X have a gamma distribution with parameters ex and A.. Use the

result of Exercise 8 to show that P(X ;;:: 2cx/A.) ~ (2/e)"'.

10 Let X have a Poisson distribution with parameter A.. Find ({Jx(t).
11 Let X have a geometric distribution with parameter p. Find lfJx(t).
12 Let X1 , X 2 , , X" be independent random variables each having a
geometric distribution with parameter p. Find the characteristic
function of X= X1 + + X".
13 Let X 1, X 2 , , X" be independent random variables each having an
exponential distribution with parameter A.. Find the characteristic
function of X = X1 + + X".
14 Let X be a discrete random variable all of whose possible values are
nonnegative integers. What relationship should we expect to hold
between the characteristic function of X and the probability generating
function of X (recall Formulas (3) and (15))?
15 Let X be any random variable.
(a) Show that ({Jx(t) = E cos tX + iE sin tX.
(b) Show that {{J-x1.0 = E cos tX- iE sin tX.
(c) Show that (/J-x(t) = ({Jx(- t).
16 Let X be a symmetric random variable, that is, such that X and -X
have the same distribution function.
(a) Show that E sin tX = 0 and that lfJx(t) is real-valued.
(b) Show that ({Jx(- t) = ({Jx(t).
17 Let X and Y be independent, identically distributed random variables.
Show that (/Jx-r(t) = I({Jx(t)l 2 Hint: Use Exercise 15.
18 Let X be a random variable such that ({Jx(t) is real-valued.
(a) Show that X and -X have the same characteristic function
(use Exercise 15).
(b) Why does it follow that X and -X have the same distribution
function?
19 Let X be a continuous random variable having the density fx(x) =
(l/2)e-lxl, -00

<

<

00.

(a) Show that ({Jx(t) = 1/(1 + t 2 ).

(b) Use (a) and the inversion formula (19) to conclude that
e-lxl =

Jex>

e-ixr

n(l

-oo

t 2)

dt.

(c) Show by using (b) that

e-lxl =

fco
-<X>

n(l

t 2)

dt.

Exercises

215

20 Let X be a random variable having the Cauchy density

fx(x) = n(l

x2),

-00

<

<

00.

Show that CfJxCt) = e-lrl, - oo < t < oo. Hint: Interchange the role
of x and t in Exercise 19.
21

Let X and Y be independent random variables having the Cauchy

density.
(a) Find the characteristic functions of X + Y and of (X + Y)/2.
(b) Why does it follow that (X + Y)/2 also has the Cauchy density?

22

Extend the result of Exercise 21 by showing that if X1, X 2 , , X,. are

independent random variables each having the Cauchy density, then
(X1 + + X,.)/n also has the Cauchy density.

23

For A. > 0 let XA. be a random variable having a Poisson distribution

with parameter)..
(a) Use arguments similar to those used in proving the Central Limit
Theorem to show that for - oo < t < oo,
lim
A.-oo

Eeit(XJ.-A.)Jv'"i =

lim exp [A.(eirJv'i - 1 - it/.J~)]

e-'2 ' 2

A.-+oo

(b) What conclusion should follow from (a) by an appropriate modification of the Continuity Theorem?

Random Walks and

Poisson Processes

In this chapter we discuss two elementary but important examples of stochastic

processes. A stochastic process may be defined as any collection of random
variables. Usually, however, in referring to a stochastic process we have in mind a
process that has enough additional structure so that interesting and useful results
can be obtained. This is certainly true of the two examples treated in this chapter.
The material on Poisson processes, our second example, does not depend on the
first two sections where we discuss random walks.
9 .1.

Random walks

Consider a sequence of games. such that during the nth game a random
variable X,. is observed and any player playing the nth game receives the
amount X,. from the "house" (of course, if X,. < 0 the player actually pays
-X,. to the house).
Let us follow the progress of'a player starting out with initial capital x.
Let S,., n > 0, denote his capital after n games. Then S0 = x and
n > 1.

The collection of random variables S 0 , S 1 , S 2 , . is an example of a

stochastic process. In order to get interesting results we will assume that
the random variables X., X 2 , are independent and identically distributed. Under this assumption the process S 0 , S 1 , . is called a random
walk.

We will further assume that the Xk's have finite mean p. If a player plays
the first n games, his expected capital at the conclusion of the nth game is
(1)

ES,. = x

np..

Suppose, however, that the player chooses numbers a < x and b ~ x

and makes a bargain with himself to quit playing when his capital becomes
not greater than a or not less than b. Then the number of times T that he
216

9.1.

Random walks

217

wtll play the game is a random variable defined by

T = min (n

> 0 I S,. < a or S,. > b).

In order to guarantee that S,.

(2)

:::5::

P(Xk

a or S,. > b for some n, we assume that

= 0)

< 1.

It is possible to prove that the random variable Tis finite (with probability
1) and, in fact, P(T > n) decreases exponentially as n -+ oo. This means
that for some positive constants M and c < 1,
P(T > n) < Me",

(3)

n = 0, 1, 2, .. ..

The proof of (3) is not difficult but will be omitted to allow room for
results of much greater interest. From (3) and Theorem 5 of Chapter 4, it
follows that ET and all higher moments ofT are finite.

x-4

a-0 ----------------------------

T- 10

Figure 1

If the player quits playing after the Tth game, his capital will be Sr
(see F igure 1). A famous identity due to Abraham Wald relates the expected capital when the player quits playing to the expected number of
times he plays the game. Specifically, Wald's identity asserts that
(4)

EST =

p.ET.

Wald's identity is remarkably similar to (1).

In proving Wald's identity it is convenient to introduce a new notation.
Let A be any event. By 1.4 we mean the indicator random variable of A,
that is, the random variable that is 1 if A occurs and 0 if A does not

218

Random Walks and Poisson Processes

occur. By definition 1"'

l...tc = 1. Using this notation we can write

T

sT

oo

= x + }=1
~ x1 t<T~J}

~ x1

=x+

}=1

Since the complement of the event {T

that

j} is the event {T < j}, we see

00

(5)

ST =

~ Xj(l - l{T<J}),

}=1

and hence
00

(6)

EST

=X+

E ~ Xj(l - l{T<j})
}= 1

It can be shown by using measure theory that the order of expectation and
summation can be reversed in (6). Thus
00

(7)

EST

E[Xj(l - l{T<J})].

j=1

In determining whether or not T < j, it suffices to look at the random

variables X 1 , X 2 , , X1 _ 1 It follows that the random variables X1 and
(1 - 1tT<Jl) are independent. Consequently
l{T<J})] = EX1E(l -

E(Xj{l -

= p(l - P(T

= pP(T

l(T<Jl)

< j))

> j)..

We now conclude from (6) and Theorem 5 of Chapter 4 that

00

EST

=X +

Jl

L P(T ;;:: j)
j=1

=X+ p.ET,
which completes the proof of Wald's identity.
If the X,.'s have mean Jl = 0 and finite variance a 2 , there is a second
identity due to Wald, namely
Var ST = u 2ET.

(8)

Since EST = x by (4), Formula (8) is equivalent to

E(ST - x) 2 = u 2 ET.

(9)

We will now verify (9). By (5)

00

ST -

~ Xj(l -

l{T<J}).

)=1

and hence
00

(ST - x) =

J=l
00

= L1
j=

00

X 1(1 - 11T<Jl) ~ X~c(l - lfT<kl)

k=1

00

~ Xj(1 - l{T<j})Xk(l - l{T<l:}).

It= 1

9.2.

Simple random walks

219

In taking expectations it is again permissible to interchange expectation

and summation. We conclude that
00

(10)

E(Sr - x)

00

= L L

E[Xj(I - l{T<J})Xk(l - l{T<k})].

j= 1 k= 1

We will now evaluate the individual terms in this double sum. Consider
first terms corresponding to values ofj and k such thatj < k. The random
variable
Xj(l - l{T<j})(l -

l{T<k})

depends only on X 1 , X 2 , , Xk_ 1 , and hence is independent of the

random variable Xk. Since EXk = Jl = 0, we see that
E[X j(l - l{T<J})Xk(l - l(T<k})]
= E[Xj(1 -

ltT<JJ)(l - ltT<kJ)]EXk = 0.

Similarly the terms in the right side of (10) vanish when j > k. When
j = k we obtain
The random variable (1 - I{T<j}) depends only on x1, x2, .. . 'xj-1, and
hence is independent of X1. Since this random variable takes on only the
values 0 and I, we see that

Thus
E[X j(l - 1{T<Jl)2 ] = E[X j(l - l(T<Jl)]

= EXJE(l

- ltT<JJ)

= Var Xj(l - P(T

= u 2P(T

< j))

> j).

We now conclude from (10) and Theorem 5 of Chapter 4 that

00

E(Sr - x) = u

~ P(T

> j) = u 2 ET,

j=1

which proves (9) and hence also (8).

9.2.

Simple random walks

We will assume throughout this section that a < x < b, a < b, and
a, b, and x are integers. The two identities of the previous section are
most easily applied if it is known that

(11)

P(Sr = a or b)

P(Sr

= a) + P(Sr = b) = 1.

220

Random Walks and Pois.son Processes

This will certainly be the case if

P(X" = - I, 0, or I) = I,

(12)

in which case the random walk S 0 , S 1 , . . . is called a simple random walk.

The main property that distinguishes simple random walks from other
random walks is that they do not "jump over" integer points. Set
p
q

= P{X"
= P {x"

r = P{X"

= 1},
= - 1},

0}.

Then p ~ 0, q > 0, r > 0, and p + q + r = 1. The assumption (2)

states that r < 1. It follows from (11) that
(13)

ESr = aP(Sr = a)

a(l - P(Sr

+ bP(Sr = b)
= b)) + bP(Sr =

b).

For simple random walks we can solve explicitly for P(Sr = a),
P(Sr = b), ESr, and ET. Consider first, the case p = q. Then 11 = 0 and
Wald's identity (4) becomes ESr = x. Thus by (13)
x = a(l - P(Sr = b))

+ bP(Sr = b).

It follows immediately that

P(Sr

(14)

b) =

x - a
b- a

and
P(Sr =a)=

(15)

b ~ x

b- a

Wald's identity does not give any information about ET when 11 = 0.

The identity (8) is applicable in this case, however, and we find that
u 2 = p + q = 1 - r and
Var Sr = u 2 ET = (1 - r)ET.

Now
Var Sr

= ESf.

- (ESr) 2

= b2 P(Sr = b) + a 2 P(Sr

b2 (x - a) + a 2 (b - x)
-x2
b- a

(ax

bx - ab) - x 2

= (x - a)(b - x).

Thus if p = q,
(16)

= a) - x 2

ET

= (x -

a)(b - x) .
1- r

9.2.

Simple random walks

If r

221

= 0 and p =

1/2,

(17)

ET = (x - a)(b - x).

Two players having respective initial capitals of $5 and

$10 agree to make a series of$1 bets until one of them goes broke. Assume
the outcomes of the bets are independent and both players have probability
1/2 of winning any given be:t. Find the probability that the player with the
initial capital of $10 goes broke. Find the expected number of bets.
Example 1.

The problem fits into our scheme with S" denoting the capital of the less
wealthy player after n bets if we choose p = q = 1/2, x = 5, a = 0, and
b = 15. The answer to the first part is given by

P(Sr

b)

5
- O
15 - 0

=!.
3

The answer to the second part is

ET

(5 - 0)(15 - 5)

50.

Suppose next that p i= q. To avoid trivial exceptions we will also assume

that p > 0 and q > 0. Wald's identity cannot be used to find P(Sr = b)
if p i= q; therefore another approach is required.
Define f(x) for x an integer in [a, b] by letting f(x) denote the probability that Sr = b when S 0 = x. We observe first that f satisfies the
difference equation
(18)

f(x) = pf(x

1)

+ qf(x -

1)

+ rf(x),

a< x <b.

This is true because

f(x) = P(ST = b)

= p. P(Sr = b I xl =
+ r. P(ST = b I xl

1)

+ q. P(Sr =

I xl =

-1)

= 0)

and
P(Sr = b

I xl =

i)

= f(x + i),

1, -1, 0.

In addition to (18) we have :the obvious boundary conditions

f(a) = 0

(19)

From (18) and (1 - r) = p

(20)

f(x

Set c

= f(a +

and

f(b) = 1.

q, we see that

1) - f(x) = q_ (f(x) - f(x - 1)),

p

1)

f(x

= f(a +

a< x <b.

1) - f(a). Then (20) implies that

I) - f(x) = c

(;r-.

a<

< b.

Random Walks and Poisson Processes

222

Using the formula for the sum of a geometrical progression we obtain

~-1

f(x)

= f(x)

- f(a)

L -(f( y + 1) -

= ~-1
~ C
y=a

(
I

f(y))

9_) ' -
p

1 - (qfpyx-

=c

1 - (q/p)

'

a~ x ~b.

From the special casef(b) = 1, we now have that

c - _1_
-~(c.:..:.
q/~
p):...__
- 1 _ (q/p)b-a '

We substitute this in our expression for f(x) and obtain

1 - (q jpy:-a
f(x) = 1 _ (q/p)b-a

We have shown, in other words, that if p ::;. q and p > 0, then

P(S

(21)

= b)

= 1 - (qfpy:-a

1 _ (q/p)b-a '

a< x

~b.

It follows immediately from (21) that under the same conditions

= a) = (q /p)"-a - (q /p)b-a
a ~ x < b.
T
1 _ (q/p)b - a
'
From (13) and (21)
ES = (b - a) 1 - (q/p)x - a + a.
(23)
T
1 _ (q/p)b - a

( 22)

P(S

Since J.L = p - q, it now follows from Wald's identity that

(24)

ET = (b - a) 1 -

(qfp)x - a _ x - a
P - q 1 - ( q I P)"-a
P- q,

a~

x ~b.

Let us modify the previous example by supposing that the

wealthier player becallse of his greater skill has probability .6 of winning
any given bet, with the other player having probability .4 of winning the
bet. Find the probability that the wealthier player goes broke, the
expected gain to that player, and the expected number of bets.
Example 2.

Here we take p = .4 and q

player goes broke is
= 15) =

P(S
T

= .6.

The probability that the wealthier

1 - (.6/.4f' = .0151.
1 - (.6/.4) 1 '

9.2.

Simple random walks

223

In order to find the expected gain to the wealthier player we first note that
the expected capital of the poorer player after they stop playing is
EST

15P(ST = 15)

15(.0151)

= .23.

Thus the expected gain to the wealthier player or the expected loss to the
other player is $5.00 - $.23 = $4.77, a good percentage of the poorer
player's initial capital. The expected number of bets is
ET = EST-

X=

-.2

J1

Let b

-4.77 ~ 24.

oo in (21). It can be shown that the left side of (21) converges

to P(S, > a for all n > 0}. If q < p, the right side of (21) converges to
1 - (qfp)x-a. If q > p, the right side of (21) converges to 0. If q = p,
the right side of (14) converges to 0. Thus for a < x = S 0 ,
(25)

--+

P(S,

> a for all n > 0) =

Similarly for b > x

P(S, <

t-(!('

for q < p,
for q

p.

= So>

b for all n > 0) =

{~ - (~r-

for p < q,
for

> q.

A gambling house has a capital of one hundred thousand

dollars. An infinitely rich gambler tries to break the house. He is allowed
to try, and decides to bet $1000 at a time. If the gambler has the probabilities .49 of winning each bet and .51 of losing, what is the probability
he will ever break the house?
Example 3.

Let S, denote the capital of the house (in multiples of $1000) after n
games. Then p = .51, q = .49, x = 100, and a = 0. By (25) the probability that the house will go broke is
I - P(S, > 0 for all n ;:: 0) =

(;r-

(::~ro

= .018.

Let A be a subset of the integers (in applications, A will have 0, I, or 2

points). For x j A and y A, let P ..(x, y) denote the probability that a
simple random walk beginning at x will hit y at some positive time before
hitting A. For x e A or y e A set P ..(x, y) = 0. These probabilities can be
computed in terms of the formulas of this section.
Example 4.

Suppose p

q. Find Pfa,b>(y, y), where a

< y < b.

After one step, the random walk is at y - I, y, or y + 1 with respective

probabilities p, 1 ~ 2p, and p. From y - 1, the probability of returning

224

Random Walks and Poisson Processes

to y before a is (y - a - 1)/(y - a). From y + 1, the probability of

returning to y before b is (b - y - 1)/(b - y). Thus the probability of
returning toy before hitting a orb is given by
Pca,b}(y, Y) = P

y-a-1
y-a

1 - 2p

b-y-1

P ---'--

b-y

or
p

(26)

(
{a, b}

) _
p(b - a)
_
y, y - 1
(y- a)(b - y)

For x ~A andy~ A, let G ...(x, y) denote the expected number of visits

toy (for positive values of n) before hitting A for a random walk starting
at x. Set G A(x, y) = 0 if x e A or y e A. It is not hard to show that the
number of returns from y toy before hitting A has a geometric distribution
with parameter p = 1 - P ...(y, y). Thus by Example 3 of Chapter 4,
G...(y, y) =

(27)
If x

::1=

P..t(Y, y) .
1 - P..t(Y, y)

y, then

(28)

GA(x, y) ~ PA(x, y)(l

G...(y, y)).

For to have any positive number of visits to y before hitting A, we must

first get toy before. hitting A . This has probability P...(x, y). If we do get
toy before hitting A, then the total number of visits toy before hitting A is
I plus the total number of returns from y to y before hitting A . This
explains (28). From (27) and (28) we have that
(29)

G A(x, y) =

p A(x, y)
1 - P"..t(y, y)

for all x andy.

Let us return to the first example of this section. Find the

expected number of times n > 1 that two players will be back to their
initial capital before one of them goes broke.
Example 5.

We recall that in this example p = q = 1/2, a = 0, x = 5, and b = 15.

The probability of returning back to the original capitals before one of
the players goes broke is, by (26),
P ,5 .
p
(5 5) = 1 - (l/2)(15) = .85.
{0, 15} '
5. 10 '
s- .
) ~
Thus by (27) the expected number of times both players will be back to
their initial capital before one of them goes broke is
G

(5 5) {0, 15}

'

Pco,t5>(5, 5)
1 - p
(5 5)
{0,15}

85
= = 5.67 .
.15

'

9.3.

225

Construction of a Poisson process

9.3.

Construction of a Poisson process

In the remaining sections of this chapter we will consider a probabilistic

model for the random distribution of particles in space or events in time.
Such models find applications in a variety of different fields . As an
example, suppose we have a piece of radioactive material and let an
experiment consist of observing the times when disintegrations occur.
Suppose the experiment starts at time 0 and let Dm denote the time of the
mth disintegration. As discussed in Example 2 of Chapter I, the laws of
physics tell us that the times Dm must be considered random variables.
The collection of points {D 1 , D 2 , .. } can be considered as a random
countable subset of [0, oo ) ..
As another illustration of essentially the same phenomena, consider
calls coming into a telephone exchange. Let D'" denote the time when the
mth call begins. There is no known way of predicting the Dm exactly, but
they can usefully be treated as random variables.
Consider an experiment of the following type. A swab is dipped into a
test tube containing a suspension of bacteria and is then smeared uniformly across the surface of a disk containing nutrients upon which the
bacteria can multiply. After a few days, wherever a bacterium was dropped
there appears a visible colony of bacteria. The locations of these spots as
well as their total number are random. This situation is illustrated in
Figure 2.

Figure 2

The location of the spots can be viewed as a random subset of points of

the disk.
In the examples of radioactive particles and bacteria, we were led to
consider a random collection of points in a certain subset S of Euclidean
space. In these examples both the location of the "particles" and their
total number is random. Associated with such a random collection are
various random v~riables such as the total number N of particles in S, the
number of particles N 8 in a specified subset B c S, and the distance D'"
from a specified point in S to the mth closest particle. In Figure 3 the

Random Walks and Poisson Processes

226

Figure 3

particles are denoted by dots. In that figure N = Ns = 4 and N 8 = 2.

It is the distributions and joint distributions of random variables of this
type that we will be studying in the rest of this chapter.
Naturally, there are many different mathematical models that consider
the random distribution of particles. We will consider one of the most
elementary and most important such models, called the Poisson process.
Such a process is closely related to the uniform distribution of particles
which we will discuss first.
Consider then a system in which the total number of particles n is fixed,
but for which the locations of the particles inS are random. The model
we want is one for which these n particles are independently and uniformly
distributed over a set S having finite volume. Denote the volume of a
subset B of S by IBI. Then each particle has probability p = IBI/ISI of
landing in B. Consequently, the number of particles N 8 that land in the
set B has the binomial distribution with parameters n and p. More
generally, let B 1 , B2 , , B~c be k disjoint subsets of S whose union isS
and let Pi = IBJI/ISI. Then the random variables N 81 , , N 8 " have the
multinomial distribution with parameters n and p 1 , , P~c Hence if
n 1, . , n" are nonnegative integers with sum n,
P(NB1 -- n h

'

N Bk --

nJ --

n'

111

(n 1 !) ( n~c !)

p1

l'k
"""

n!
IB I"J
=-nElL.
lSI" J=t n !
k

The Poisson process on Sis a modification of the above. We suppose

now that the total number of particles N = Ns in S is a random variable
having a Poisson distribution with parameter A.ISI. Moreover, we assume
that given N = n, the n particles are independently and uniformly dis-

9.3.

227

Construction of s Poisson process

tributed over S. Let B1 , .. , B" be as before. Then our assumptions are

that if n 1 , , n" are k nonnegative integers with sum n, then
P(NBt

n1, ... ' N Bk

" IB I"J
n
lSI"
n !

= n" I N = n) = -n!

Sl!_

1=1

Hence
(30) P(N

"
= n, N 8 , = n 1 , . . , N 8 = n") = P(N = n )n!- fl
k

IS I" J= 1

IBI"1
n1 !

Sl!_

A."JSI" e-AISI n!
IBii"J
n!
lSI" 1=1 n1 !

= A."e-liSI

J= 1

IBJI'J.
n1 !

Since the sets B1 are disjoint with union S,

lSI

IB11 + ... +

IB~cl

'

and thus we can write the right-hand side of (30) as

n" (A.IBJI)"J
n !

~--"-'.:.-e

J=1

Now the event {N = n, NB,

event {N81 = n1 , . .. , N 8 ,. =
N 81 + + N 8 k. Thus
P(N

-liB11.

n 1 , , N 8 k = n"} is the same as the

n~c}, because n = n 1 + + n" and N =
=

= n, N 8 , = n 1 , , N 8 = n~c) = P(N8 , = ntt ... , N 8 ,. = n~c).

k

We have therefore proved the important fact:

(31)

P(N 8, = nh . . . , N 8k = n") =

n"

(A.IBJI)"J -AIB1l
e
.
i= 1
n1 !

In other words, the random variables N 81 , , N 8 " are independent and

Poisson distributed with parameters A.IB11 respectively.
It is not too surprising that the random variables N 81, 1 < j < k, are
Poisson distributed; but it is surprising that they are mutually independent
because in the case where the number of particles is fixed the corresponding
quantities are dependent. It is thi:s independence property that makes the
Poisson process easy to work with in applications.
In the preceding models the total number of particles, whether random
or fixed, was always finite, and they were distributed over sets having
finite total volume. For some purposes, however, it is theoretically simpler
to consider an infinite number of particles distributed over a set having
infinite volume. Thus we might want to distribute particles over all of
R' or over [0, oo ), etc. To cover such cases, we require only a slight
extension of the previous model.

Random Walks and Poisson Processes

228

The basic assumption of a Poisson process on a set S having finite or

infinite volume is that if B 1 , B2 , . , Bk are disjoint subsets of S each having
finite volume, then the random variables N 81 , , N 8 ,. are independent and
Poisson distributed with parameters A.IB1 1, . .. , .AIBkl respectively. The
constant A. is called the parameter of the process.
Let B be a subset of S having finite volume. As a consequence of the
definition of a Poisson process, it follows that if B 1 , B2 , , Bk are disjoint subsets of B whose union is B, and n 1 , n 2 , . . . , nk are nonnegative
integers with sum n, then

(32)

P(NBI = nl, ... ' NBk = nk I NB

n! k IB-1"1
~-.
IBI" i=t ni!

n) = -

To verify (32) note that the desired probability is just

P(NBt

nl, ... ' N Bk = nk) - n~= 1 e-).IBJI (A.IBji)"J /nj!

P(N8 = n)
e-J.IBI (A.IBI)"/n!

which reduces to the right-hand side of (32).

Another way of looking at Equation (32) is as follows: Given that there
are n particles in B, the joint distribution of the random variables
N 81 , , N 8 k is the same as that obtained by distributing n particles
independently and uniformly over B. This fact is very useful for solving
some problems in which the Poisson process acts as an input to a more
complicated system. We will not pursue this aspect of a Poisson process
any further in the text. (See, however, Exercises 21 and 31 for simple
illustrations of its use.)
9.4.

Distance to particles

Suppose we have a Poisson process on a subset S of Euclidean space.

If S has finite volume, then the number N of particles in S is finite. Let
D 1 < D 2 < < DN denote the distance to these particles from the
origin arranged in nondecreasing order. If S has infinite volume, then
the number of particles inS is infinite, and we let D 1 < D2 < < Dm
denote the distances to the particles arranged in non decreasing order.
Such an arrangement is possible because for any positive number r only
finitely many particles are at distance less than r from the origin. In this
section we will compute the distribution of Dm for various choices of the
setS.
We first give an example where these distances enter in a natural way.
Suppose that stars in a certain set S of 3-dimensional Euclidean space are
distributed according to a Poisson process on S with parameter A.. Suppose
further that these stars are equally bright. The amount of light reaching
the origin from a star is proportional to the inverse square of the distance

9.4.

Distnce to particles

229

from the origin to the star. Thus the amount of light received from a star
of distance r from the origin is K/r 2 for some positive constant K. The
amount of light received from the nearest (and hence apparently brightest)
star is Kf
The total amount of light is

n:.

By advanced probability techniques it can be shown that if S is all of

three-dimensional space, then the sum of the above series is infinite with
probability one. This fact has interesting implications in cosmology.
We will now compute the distribution of Dm, assuming for simplicity
that S has infinite volume. Let S,. = S n {x : lxl < r} (that is, let S,. be the
set of points in Sat distance at most r from the origin), and let lfJ(r) denote
the volume of S,.. The number Ns,. of particles in S,. has a Poisson distribution with parameter A<p(r). The event {Dm < r} is the same as the event
{Ns,. ~ m}. Thus by Equations (39) and (40) of Chapter 5,
(33)

P(Dm < r) = P(Ns..

l(ll(r)

J.

m)

tm-1e-t
(m - 1)!

dt.

It follows from (33) that if <p(r) is differentiable, then Dm has a density

function fm given by
.,tm<p(r }m-1 <p'(r )e-l(ll(r)
r > 0.
(34)
fm(r) =
(m _ )!
,
1

If <p(r) is strictly increasing it has a continuous in't'~rse function <p - 1 (r ).

It follows from (33) that the random variable q>(Dm) has the gamma
distribution f(m, A.).
In several important cases q>(r) is of the form <p(r) = cr 4, where c is a
positive numerical constant (this is true, for example, if S = R 4 or if
S = [0, oo)). In this case (34) becomes
(35)

( )
/,mr

d(c.A.)m
md-1 -cJ.rd
r
e
'
- 1)!

= (m

> 0.

Various special cases of this formula will be considered in the exercises.

We will use Formula (35) to compute the moments ED/, in these cases.
Thus

f
f.

00

ED!,.

riJ,ir) dr

d(cl)m
r md+j-1 e -c).rd d r.
o (m - 1)!
oo

---

230

Random Walks and Poisson Processes

To compute this, we make the change of variables

to obtain
EDi = (J..c)- 114r(m + (jfd))
m
(m - 1)!
9.5.

= J..cr 4 and integrate

Waiting times

So far we have been visualizing a Poisson process as a model for the

distribution of particles in space. If we think of the set [0, oo) as the time
axis, then we may consider a Poisson process on [0, oo) as the distribution
of times at which certain events occur. In the beginning of Section 3 we
mentioned some examples of using a Poisson process on [0, oo) in this
manner.
In the switch from thinking of a Poisson process on [0, oo) as a distribution of particles in the set [0, oo) to thinking of the Poisson process as
the distribution of events in time, a new set of terms is introduced. Instead
of speaking <?f " particles," we now speak of "events", and the distance Dm
to the mth particle now becomes the time when the mth event takes place.
Let N(t) = N[o,r1 denote the number of events that occur during the
time span [0, t]. Then N(t) is a Poisson distributed random variable with
parameter J..t. If 0 < s ~ t, then N(t) - N(s) represents the number of
events taking place during the time span (s, t], and it has a Poisson distribution with parameter J..(t - s). More generally, ifO < ! 1 < t 2 < < t,.,
then N(t 1), N(t 2 ) - N(t 1), . , N(tlt) - N(tlt_ 1 ) are independent random
variables having Poisson distributions with parameters
A/1, J..(t2 - 11), ... , J..(tlt - tlt-1)

respectively. These facts are immediate from our definition of the Poisson
process and its translation into the time language.
As mentioned above, Dm is the time of the mth event. From our results
in Section 9.4 we know that Dm has the gamma distribution r(m, J..). In
particular, D 1 is exponentially distributed with parameter J... Recall
from Chapter 6 that the sum of m independent, identically distributed
exponential variables has the gamma distribution r(m, J..). Define
random variables W1 , W2 , , Wit,... as follows : W1 = D 1 , Wit =
Dlt - Dlt_ 1 , n > 2. Then clearly Dm = W1 + + Wm. The discussion
just given makes it plausible that wl, w2, ... , wm are independent
exponentially distributed random variables with the common parameter J...
This is in fact true and is a very interesting and useful property of the
Poisson process on [0, oo ). The random variable Wm is, of course, nothing
more than the time between the (m - I)st event and the mth, so that the
times W., W2 , are the waiting times between successive events in a
Poisson process.

9.5.

231

Waiting times

Theorem 1 Let W1 , W2 , , W11, be the waiting times

between successive events in a Poisson process on [0, oo) with parameter A.. Then W11, n ~ I, are mutually independent exponentially
distributed random variables with the common mean A. - l .
Proof

Let f,. be the n-dimensional density given by

for 0 < t 1 ~ t 2 ~
elsewhere.

< t

11 ,

From Example 13 of Chapter 6, we see that the theorem is true if and only
if the random variables D 1 , , D, have joint density /,. This is true
for n = I, since D 1 is exponentially distributed with parameter A.
A rigorous proof in general is more complicated. Before giving this
proof we will first give a heuristic way of seeing this fact.
LetO = t 0 < t 1 < < t11 andchooseh > Oso small that t,~_ 1 + h < t.,
i = I, 2, ... , n. Then (see Figure 4)

(36)

+ h, 1 < i < n)
= 0, N{t 1 + h) - N(t 1 ) = 1, ... ,

P(t1 < D1 < t 1

= P(N(t 1 )

N(t11 ) - N(t,._ 1

+ h) = 0, N (t,. + h)

- N {t11)

>

1)

= e-Att(A_h)e-).h . . e-A(tn-tn-t-h)[l _ e-).h]

= A.n-1h11-le-At"(1
0

_ e-Ah).
1

Figure 4

If we knew that the random variables D 1, D 2 ,

g 11 that was continuous at the point {t 1 , t 2 ,

D 11 had a joint density

t,.), we could conclude that

P(t1 < D 1 :s;; t 1 + h for 1 < i < n) = g11(t 1 , . ,

111)h"

where e(h) is some function of h such that

lim e(h)
11
h lO h

0.

It would then follow from (36) that

g(t 1 ,

t11 ) = lim h-"P(t1 < D 1 < t 1 + h)

h10

= lim
h

10

= A."e-M"

as desired.

-).h

A."- 1 e-;.'" ~e_ _

h

e(h),

232

Random Walks and Poisson Processes

We will now give an elementary but rigorous proof of Theorem 1.

Although this proof is not difficult, it is rather long, and the reader may
wish to omit it.
Let F,. be the distribution function having density/,.. It follows at once
from the definition of/,. that for n > 2,

/,.(s1 ,

.. ,

s,.) = / 1(s 1 )f,._ 1 (s2

s1 , , s,. - s 1).

Integrating both sides over the set s 1 < t 1 ,

(37)

F.(t 1 ,

t,.) =

J~

s,. < t,., we see that

1
/

1 (s 1)F,._ 1(t 2 -

sH .. . , t,. - s 1 ) ds 1

Let G,. denote the joint distribution of D 11 D 2 , , D,.. From Example 10

of Chapter 6 we see that the theorem is true if and only if the random
variables D 1 , , D,. have joint density/,., and hence joint distribution
function F,.. Consequently, to prove our theorem we must show that
F,. = Gn.
Now F 1 is just the exponential distribution with parameter A. As
pointed out before, G 1 , the distribution of D 1 , is also exponential with
parameter A. Thus F 1 = G 1 .
Suppose we can show that for n > 2,
(38)

Gn{t 1 ,

. ,

t,.)

= J~'

1 (s 1 )G,._ 1 (t2 -

s1 ,

tn - s 1 ) ds 1

Then, as G1 = F 1 it would follow from (37) and (38) that G2 = F2

Using the fact that G 2 = F 2 , another application of (37) and (38) would
show G3 = F 3 , etc. In other words, if we knew that (38) held, then our
theorem would follow from (37) and (38) by induction. In establishing
(38) we can assume that 0 < t 1 < < t,., since otherwise both sides of
(38) are zero.
To start on the proof; observe that the event {Di < tJ is the same as the
event {N(t1) > i} and thus

,.

{D1 < t 1, 1 < i < n} = () {D1 ~ t1}

i= 1

,.

n {N(t,>

~ i}

i= 1

We can therefore write

G,.(t 1 , t 2 ,

t,.)

= P(N(tJ >

i, 1

< i < n).

Consequently (38) is the same as

(39)

P(N(t1) > i, 1 < i < n)

=

J~'

(s 1 )P(N(t; - s 1) > i - 1, 2 < .i < n) ds 1

9.5.

233

Waiting times

To establish (39), note first that for any k > 1

J.t ,e-.u [A.(t -

(A.tt
e -J.t =

(40)

k!

s)]"- e -J.(t-s) ds.

(k - 1)!

Indeed,
t

J.

5)]k-1

Ut

A.e-;., ""- -

e-.t(r-s) ds

(k - 1)!

(k - 1)!
(k -

Now let 0 < t 1 < t 2

Next we claim that

P(N(t 1 )
=

k 1,

~ ..

J~' Ae-J. P(N(t 1

5

(At)" e-J.t.
k!

k,)

s) = k 1

fr s"-1 ds =
1)! Jo

< t, and let 1 < k 1 < k 2 < .. < k,.

N(t,)

s)"- 1 ds

e-J.tA_k

(41)

J.t (t -

-J.nk
"'

1, ... , N(t, - s) = k, - 1) ds.

To see this observe that by (40)

(42)

P(N(t 1 ) = k 1 ,

N(t,)

k,)

=P (N(t1) = k1)

f.

, _ [A.(t

P(N(t1)

N(t1 _t) = ki - k1 _t)

_ s)]"'-te-.t<r,-s) ds

Ae As ~~1~--=---------(k1 - 1)!

fi [A(t, -

}=2

1'=2

t,_l)]"r"J-e-..l<trtJ-> .
(kj - kj-1)!

On the other hand,

(43)

J:'

A.e-..l5 P(N(t 1

s)

k1

= J~' A.e-.A.sP(N(t 1
,
=

r,

J.o
X

A.e- As

s) - N(t1 _ 1

1 -

}=2

s) = k 1

n P(N(t

1, . .. , N(t,. - s)

= k,.

- 1) d$

1)
-

s)

= k1

k 1 _ 1 ) ds

[A(t _ s)]"-1e-.t(r,-s)
"'--'---"1'---_....:_=---------

(kl - 1)!
, [A.(t _
n
j=2
J

t _ )]"1 -k1 -e-JJ.<t1 -t1 -1)

j

ds.

(kj - kj-1)!

Comparing the right-hand side of (42) with that of (43), we see that (41)
holds. The desired equality (39) now follows from (41) by summing both
sides of (41) over all values of k 1 , . , k, such that k 1 < k 2 < < k,.
and k 1 > I, k 2 ~ 2, . .. , k, > n.
I

234

Random Walks and Poisson Processes

Exercises

1 Let

S,. be a random walk with Jl = 0 and S0

P(a -

< Sr

~ b

+ d)

= x.
= I,

Suppose that

where a < x < b, c ;;;::: 0, and d > 0.

(a) Show that
(a - c)P(Sr

< a)

+ bP(Sr >

b)

< x < aP(Sr < a) + (b

+ d)P(Sr >

b).

(b) Show that

< P(Sr ;;;::: b) ~ x - a + c.

b-a+d
b-a+c
x - a

2 A gambler makes a series of bets of $1 . He decides to quit betting as

soon as his net winnings reach $25 or his net losses reach $50. Suppose
the probabilities of his winning and losing each bet are both equal to
1/2.
(a) Find the probability that when he quits he will have lost $50.
(b) Find his expected loss.
(c) Find the expected number of bets he will make before quitting.
3 Suppose the gambler described in Exercise 2 is playing roulette and his

true probabilities of winning and losing each bet are 9/19 and 10/ 19
respectively. Solve (a), (b), and (c) of Exercise 2 using the true
probabilities.
4 A gambler makes a series of bets with probability p of winning and

probability q > p of losing each bet. He decides to play until he has

either won M 1 dollars or lost M 2 dollars, where M 1 and M 2 are
positive integers. He has a choice of betting 1 dollar at a time or of
betting 1/2 dollar at a time. Show that he is more likely to win M 1
dollars before losing M 2 dollars if he bets 1 dollar at a time than if he
bets I /2 dollar at a time. What generalization of this result seems
plausible?
5 Derive (14) by solving the appropriate difference equation.
6

Let Sn denote a simple random walk with p = q = 1/2 and let a < b.
Find P1a,b}(x, y) and G1a,blx, y) for a < x < b and a < y < b.

7 Let Sn denote a simple random walk withp = q = 1/2. Find P 101 (x, y)

and G101 (x, y) for x > 0 andy > 0.

8

Let Sn denote a simple random walk with 0 < q < p. Find P 0 (x, y)
and G0(x, y).

9 Let Sn denote a simple random walk with 0 < q < p. Find P<01( -1 , y)
and G101( -1 , y) for y < 0.
10

Suppose points are distributed in 3-dimensional space according to a

Poisson process with parameter }. = 1. Each point of the process is

Exercises

235

taken as the center of a sphere of radius r. Let X denote the number

of spheres that contain the origin. Show that X is Poisson distributed
with mean (4/3}rrr 3 .
11 A point is chosen at random in a circle with center at the origin and
radius R. That point is taken as the center of a circle with radius X
where X is a random variable having density f. Find the probability
p that the random circle contains the origin.
12 Suppose n points are independently and uniformly chosen in the circle
with center at the origin and radius R. Each point is taken as the
center of a random circle whose radius has density f Find, in terms of
p of Exercise II, the probability that exactly k circles contain the origin.
13 Find the answer to Exercise 12 if the n points are replaced with a
random number N of points having a Poisson distribution with mean
nR2
14 Suppose N balls are distributed at random into r boxes, where N is
Poisson distributed with mean A.. Let Y denote the number of empty
boxes. Show that Y is binomially distributed with parameters r and
p = e-J.tr. Hint: If X 1 is the number of balls in box i, then Xh ... , X,.
are independent Poisson distributed random variables each having
mean ).jr.
15 Using the result of Exercise 14 we may easily derive the probability
Pk(r, n) that exactly k boxes are empty when n balls are distributed at
random into r boxes. To do this first observe that
<Xl

P(Y = k) =

L P(N = n)P(Y =
n=O

IN =

n)

and

So

Now equate coefficients of A." to rederive Equation (16) of Chapter 2.

18 Suppose the times of successive failures of a machine form a Poisson
process on [0, oo) with parameter A..
(a) What is the probability of at least one failure during the time period
(t, t + h), h > 0?
(b) What is the conditional probability of at least one failure by time
t + h, given that there is no failure by time t?
17 Suppose we have a Poisson process on (0, oo) with parameter A.. Let
Z , denote the distance from t to the nearest particle to the right.
Compute the distribution function of Z,.

Rsndom Walks and Poisson Processes

236

A.. Let
Y, denote the distance from t to the nearest particle to the left. Take
Y, = t if there are no particles to the left. Compute the distribution
function of Y,.
For Zt and Y, as in Exercises 17 and 18,
(a) show tha.t Yt and Zr are independent,
(b) compute the distribution of Z, + Y,.
Particles arrive at a counter according to a Poisson process with
parameter A.. Each particle gives rise to a pulse of unit duration. The
particle is counted by the counter if and only if it arrives when no pulses
are present. Find the probability that a particle is counted between time
t and timet + I. Assume t > I.
Consider a Poisson process on (0, oo) with parameter A. and let T be a
random variable independent of the process. Assume T has an
exponential distribution with parameter v. Let Nr denote the number
of particles in the interval [0, T]. Compute the discrete density of Nr.
Do Exercise 21 if T has the uniform distribution on [0, a], a > 0.
Consider two independent Poisson processes on [0, oo) having parameters A. 1 and A.2 respectively. What is the probability that the first
process has an event before the second process does?
Suppose n particles are distributed independently and uniformly on a
disk of radius r. Let D 1 denote the distance from the center of the disk
to the nearest particle. Compute the density of D 1
For D 1 as in Exercise 24 compute the moments of D 1 Hint: Obtain a
Beta integral by a change of variables.
Consider a Poisson process on R' having parameter A.. For a set A
having finite volume, let N ... denote the number of particles in A.
(a) Compute ENl .
(b) If A and Bare two sets having finite volume, compute E(N,.N8 ).
Let A1 , A 2 , , A 11 ben disjoint sets having finite volume, and similarly
let B 1 , B 2 , , B11 be n disjoint sets having finite volume. For real
numbers cx 1, , CX11 and P1, , p11 , set

18 Suppose we have a Poisson process on [0, oo) with parameter

19

20

21

22
23

24

25
26

27

11

f(x) = ~ cx11,.,(x)
i=l

and
II

g(x) = ~ P,ls,(x).
l= 1

Show that for a Poisson process having parameter A.

E

(t

~~.,) (t

CXtN

= A.

p,N

B,)

2(JR. f(x) dx) (JR. g(x) dx)

A.

JR. f(x)g(x) dx.

237

Exercises

28

In Exercise 27 show that

29 Consider a Poisson process with parameter A on R 3 , and let D,. denote

the distance from the origin to the mth nearest particle.

(a) Find the density of D,..
(b) Find the density of D!.
30 Suppose we have a Poisson process with ,parameter A. on the upper
half-plane of R 2 , i.e., the Poisson process is on the subset S =
{(x, y): y > 0} of R 2
(a) What is the density of the distance D,. from 0 to the mth nearest
particle?
(b) Find the first two moments of D,..
31 Consider the following system. The times when particles arrive into
the system constitute a Poisson process on [0, oo) with parameter A..
Each particle then lives for a certain length of time independent of the
arrival times of the particles in the process and independent of the
lives of the other particles. Suppose the lengths of life of the particles
are exponentially distributed with common parameter p,. Let M(t)
denote the number of particles that are alive at time t. Compute the
distribution of M(t) by carrying out the following steps.
(a) Suppose a particle arrives according to the uniform distribution on
[0, t] and lives for a random length of time that is exponentially
distributed with parameter p,. Find the probability p, that the
particle is alive at time t.
(b) Using the fact that given N(t) = n the particles are independently
and uniformly distributed over [0, t], show that
P(M(t) = k I N(t)

= n) = (~) p~(l

- p,)"-"

(c) Now show that M(t) is Poisson distributed with parameter ltp,.

Answers

CHAPTER 1

2. 18/37.

3. 1/2.

6. 3/10.
9. 5/12.

4. 1/8.

8.

7. 1/2.
10. 5/8, 3/8.

12. 1/2.

3/10.

11. 4/5.

13. (a) 1/2, (b) 1/2.

15. 5/29.

14. 2/5.

19. 0.

18. (a) 19/25, (b) 35/38.

20. (a)
(r

(c)
(r

r(r - 1)
b)(r + b -

br
b)(r + b -

16. 10/19.

(b)
(r

1)

rb
b)(r + b -

1)

b(b - 1)
b)(r + b -

1)

(d)

(r

1)

21. (a) 98/153, (b) 55/153.

22. (b) 13/25, (c) 21/25.

23. (b) 2/5, (c) 9/10.

25. (a) 1/12, (b) 17/36,

26.

(a) 1/4, (b) 2/5,

27. 14/23.

(c) 1/2.

28. 4/9.

29. 2/13.

31. (a) (r

+ c)/(b + r +

(c) 6/17.

c), (b) b/(b

30. 1/3.

+r+

c).

37. 1 - (4/5)51 56/5.

38 ' (a)

tto (~) (if (lfo-t,

39..9976.

(b) 1 -

6 122
134 - 12

c:) (~r (~fO,

41. 75/91.

40. 2.
44. 1 - (1/10) 12

42. - - -4
45' (a)

(~f

(b)

ktO

c:) (~f (~rl-t.

46. 1 - (11/4)(3/4)7
239

Answers

240

(12) (.!)k (_!_)12-k

k
10
10 .
.
~ (12) (.!)k (_!_)12-k
k
10
10
~

47.

t=2

k=t

CHAPTER 2

3. 1/n.

2. (a) 2", (b) 2(2" - 1).

4. (10)6/106

5. (n - l)(r),.- 1 /r".

6. [('

1. 64.

~ N) (~))'-1

7. 2(n - k - 1)/n(n - 1).

8. n(n + 1)/2.
9. (a) (n - 2)/n(n - 1), (b) (n - 2)/(n - 1)2

(:

~ ~) r +

b - n

14. (a) 4q, (b) 4 10q, (c) 13 48q,

(d) 13.
(g) 13.

12. 4.

e:)

Here q =

6q,
3

4 q.

c:)
e:) (~) (~).

(e) 4.

(h)

q,

(f) 10 4 5 q,

11 4q.

(i) 13

c (~)
2
)
3

52)-l
.
( 5

15.

sj c3) .

11.

<a> <r - k>,.t<r>,.. <h> ( 1 - ;)".

18.

(~ =~)/(:).

20. Expand out the terms.

16. 4. (48),._ t/(52),..

4 q.

Answers

. 241

CHAPTER 3

1 _/(x) = {o1/10, x

o, 1, ... , 9,

elsewhere.

2. P(X

+ r = n) = pr (

- r

n -

)< -1)"-r(l -

(!) (n ~ k).

3. (a) P(X = k) =

p)"-r,

s k s

6,

n = r, r + 1, ... .

C~)

= k) = (;)

(b) P(X

4. 1/(2N +1

Gr or-"

n.

2).

5. (a) .3, (b) .3, (c) .55, (d) 2/9, (e) 5/11.

+ (1 _ p)to,
(1 _ p)' _ (1 _ p)u.

&. (a) (1 - p)4 , (b) (1 - p) _ (1 _ p)s

(c) (1 _ p)3 _ ( 1 _ p)6

7. (a) 3/4, (b) 1/10, (c) 1/25, (d) 3/50.

8. P(X

= k)

9. P(X

= k) = (k

= (2k - 1)/144,

IC~),

12.

k = 2, 3, .. .. 12.

- 1)

0
- {p(1 - p)",
1 . P( Y - x) - (1 _ p)M

= 0, 1, ... M -

x = M.

1,

P(X 2 ;;: k2) - p(l - p)", k = 0,1 , 2, ... ,

(b) P(X + 3 = k) = p(l - p}"- 3 , k = 3, 4, ... .

11. (a)

12. (a) P(Y

(b) P(Z

(:)I(~)

s y) =

z) = ('

~-

y = n, n

z) I (~), z =

I, ... , r,

1, 2, ... , r - n

13. (a) 2/3, (b) 2/9, (c) 13/27.

14 ( ) N + 2
a 2(N + 1) '

(b)

15. (a) P(min (X, Y)

1
N + 1

= z) =

2
(N - z)

+ 1)
2z + 1
z
,
(N + 1) 2
(N

(b) P(max (X, Y) = z)=

(c) P(IY -

XI = 0) =

P(lY - X I'

1
N

'!' 1 ,

z) = 2(N

= 0, ... , N,

+ 1-

(N

z = 0, .. . , N,

1)2

z)

z =

... '

+ 1.

242

Answers

16. (a)

(b)

P2
,
P2 - PtP2

P1P2
P2 - P1P2

Pt +
Pt +
17. (a) geometric with parameter p 1 + P2 - PtP2

(b)

18.

z = 0, 1~ 2~ . . ..

PtP2 [(1- pz)H1- (1- Ptt+t],

P1- P2

(a) g(x)

I:, h(y),

(b) h(y) L:x g(x).

20. 5/72.
2
( r)! 2 ,
x 1 ! . .. x,.! r ,.
{2r)!
(b) 2"r2".

21. (a)

where x 1 are nonnegative integers whose sum is 2r,

22. (a) binomial with parameters nand p 1

(b)

(z) (

P1

)z-y (

Pt
P2

P2

P1

),.

P2

24. {17/2)e- 3
26. p"(l - p):x,.-r.

23.
25. {a) 1 - {5/6)6 , {b) 4.
(53/8)e- S/ 2

(;

~ 11) (~ =; ~ 11) (: =;)

30. ~--~~------~~--~

(;)
z - 1

2 :s; z :s; N,

Jil'
31. P(X + Y =

z)

= ------=--2N + 1 - z , N + 1 ::::; z ::::; 2N,

N2

32.

<l>x{t)

1 (1 -

35

=
{

(z) (
y

).1

{x/2)!
0

).2

x! y! z!

37. (a)

elsewhere.

).1

)>'

).2

).1

3 6 (x + Y + z)! (
'

= 1.

, x a nonnegative even mteger,

)%-y (

).1

t :1: 1, and <l>x(l)

1-t

N+l

).:xf2e-A

33 fx(x)

elsewhere.
1
tN+ ) '

).1
).2

).2

):x (
).3

).1

).2
).2

)' (

).3

l1

).3
l2

eAp(t-O,

{b) Poisson with parameter lp.

CHAPTER 4

1. {2N + 1)/3.

3. ;.-t(l -

e-A).

2. 4p{l - p)(l - 2p).

4. 17.

)z
).3

Answers

243

6. p-1(1 - p)[1 - (1 - p)M].

7. M + p-1(1 - p)M+1.

+ 2N)/12.

8. EX = N/2 and Var X = (N 2

10. 2.
14. E(2X + 3Y) = 2EX + 3EY,
Var (2X + 3 Y) = 4 Var X + 9 Var Y.
16. (a)

(1 - ;1)",

(d) r ( 1 -

(b)

r[

(1 - 2)"
; ,

(c) r

~)"]

1 - (1 -

(1 - t),
~

r(r - 1)[ ( 1 -

~r

-(

1-

~)

2
"]

17. (a) 1, (b) 1.

k-1

18. ~

2.

r(1 - i/r)

t= 1

20.

-C1z

v'(af + ai)(ai + a~)

21. 9- 2.Ji

22. -1.

23. (a) -1/3, (b) -1/2.

25. (c) EXY

= n(n -

Var X= n ('1 )
r

n( n- 1)

1)

'

1 2
'

r(r - 1)

(1 - 'r r - 1n ,
1

r1r2

) '

- n2

r(r - 1)

(d)

Var Y

'1'2
2

= n ('

(1 - ~)r r - 1
r -

n ;

nG=;) J';~z ( r:) ( r:)

1_

26. 0

1_

= 1.

27. Chebyshev's inequality shows that a = 718 will suffice (see also the answer to
Exercise 46 of Chapter 7).

32. z/2.
CHAPTER 5

1. Fx(- 1) + 1 2. F(x) = 0, x <

3. F(x) = 0, x <
4. F(x) = 0, x <

Fx(3).
0; F(x) = x/R 2 , 0:::;; x :::;; R 2 ; and F(x) = 1, x > R 2

0; F(x) = x 3 /R3 , 0 :::;; x :::;; R ; and F(x) = 1, x > R.

0; F(x)

= x/a, 0

:::;; x :::;; a; and F(x) = 1 for x > a.

= (2hx - x 2 )/h2 , 0 :::;; x :::;; h; and F(x) = 1, x

sJJj 2; F(x) = .J4x 2 - 3s 2 js, sJJ/2 :::;; x :::;; s;

5. F(x) = 0, x < 0; F(x)

> h.

= 0, x <
and F(x) = 1, x > s.

6.

F(x)

7.

F(x) = O,x < O;F(x) = x 2 /2, 0:::;; x < 1;F(x) = -1

1 :::;; x :::;; 2; and F(x) = 1, x > 2.

+ 2x

-(l/2)x2 ,

244

Answers

8. m = A. -

Jo~e 2.

9. t = - Jog .9/100 Jog 2 = 1.52

10- 3

10. F(x) = 0, x < 0; F(x) = xfa, 0 s x < a/2; and F(x) = 1, x ~ a/2.
11. (a) 7/12, (b) 1/3, (c) 1/6, (d) 5/12, (e) 1/2.

12. (a) (iv) F(x -) = F(x) for all x;

(b) (ii) F is a nonincreasing function of x and (iii) F(- oo) = 1 and F( + oo) = 0;
(c) (ii) F is a nonincreasing function of x, (iii) F(- oo) = 1 and F( + oo) = 0, and
(iv) F(x -) = F(x) for all x.

13. F(x) = O,..x < -5;F(x)

14. e- 1 - e- 2

= (x + 10)/20, -5 s

x < 5;andF(x)

= 1,x ~

5.

15. f(x) = 1/ 2(jxj + 1)2 = F'(x) for all x.

16./(x) = 3x2/R3 ,0 < x < R;and{(x) = OeJsewhere..

17. /(x) = x, 0 < x < 1;((x) = 2- x, 1 < x < 2; and/(x)

18. / 1 (y) = f(y) + f(-y), y > 0;/y(y) = 0, y s 0.

= Oelsewhere.

19. /(x) = 2xg(x 2 ), x > 0; and g(y) = f(-.ly)/2 ../y, y > 0.

20. If P > 0 then/1 (y) = pyl-1, 0 < y < 1, and/1 (y) = 0 elsewhere.
If P < 0, then/y(Y) = -pyl- 1, y > 1 ; andfr(Y) = 0 elsewhere.
21. fr(Y) = y- 2 (((1 - y)fy), 0 < y < 1, andfr(y) = 0 elsewhere.
23. 9'(x) = (x - a)f(b - a), - oo < x < oo.
24. Y has an exponential .density with parameter Afc.

25. Multiply g by 12.

= lbl/n(b 2

26. fr(Y)

(y -

a)2 ), - oo < y < oo.

27. F(x) = 0, x < -1; F(x) = 1/2 + 1/n arcsin x, -1 s x s 1; F(x) = 1, x > 1.
/(x) = ljn../1 - xz, -l < x < 1, andf(x) == 0 elsewhere.
28. f(x) = A.lxle-Ax2 , - oo < x < oo .
29. X - a and a - X have the same distribution. F(a - x) = 1 - F (a + x) for all x.
30. <I(x)

= 1/2 +
2

31. fr(Y) =

1/2 erf (xf ../2), - oo < x < oo.

_ e- 7212 (1\

0 < y < oo, and fr(Y) = 0 elsewhere.

a v'2n

32. f 1 (y)

_ exp [-(logy - p) 2/2u 2],

0 < y < oo, and/y(y) = 0 elsewhere.

ayV2n

33 . .6826.
34. (X - #)/a has the standard normal distribution.
35. fr(- 6) = .0030,
/ 1( - 5) = .0092,
fr(- 4) = .0279,
/ 1 (-2)

.1210,
/y(2) = .1210,
/ 1 (6) = .0030,

/ 1 ( - 1) = .1747,
[ 1 (0) = .1974,
/ 1 (3) = .0655,
/y(4) = .0279,
fr(Y) = 0 elsewhere.

/y(- 3) = .0655,
/ 1 (1)

= .1747,

/y{5) = .0092,

36. # = 160, (1 = 29.6, P(X ~ 200) = .0885, P(X ~ 220 I X~ 200)

38. 2 seconds.
39. Geometric with parameter 1 - e- J..

.244 (24.4%).

245

AnswtJts

40. (e)

A, t > 0, where A is the parameter of the exponential distribution;

(f) improves for a < 1, deteriorates for a > 1, stays the same for a = 1.
g(t) =

41. Gamma density r(a, A/c).

( ) _- -2AYl-1 e -A>'l,
43 JirY

y > 0, and/r(Y)

r(a)

= v'y, y

44.

tp(y)

45.

tp(x) = [~- 1 (x)) 2 , -1

<

46.

~- 1 (.1) = -1.282,
~- 1 (.4) = - .253,
~- 1 (.7) = .524,

<>- 1 (.2)

47.

p.

= 0 elsewhere.

;;:: 0.

+ .615G.

< 1.

~-1(.5)

= -.842,

t>- 1(.3) = - .524,

~- 1 (.6) = .253,

0,

~- 1 (.9)

t>- 1 (.8) = .842,

= 1.282.

49 . .82.

48. 1.

CHAPTER 6

1. Fw z(W, z)

(w -b a' z -d c) .

fw z(w, z) = _!_ I

bd

(w -b a' z -d c) .

F(v'w, v'z) - F(- v'w, v'z) - F(v'w, - v'z) + F(- v'w, - v'z)
1
_ (/(V;, v'~) + /( -v';, V~) + /(v';, -v;)
andfw,z(w, z) =
4V wz
+ f( -Vw, -v;))
for w, z > 0 and Fw.z(w, z) and.fw.z(w, z) equal zero elsewhere.

2. Fw.z<w, z)

3. (a) 3/4, (b) 5/12, (c) 3/4; these results are easily obtained by finding the areas of
the appropriate unit square.
4. 1 - e-t/2(12.

6. 1/3.

5. 3/8.

7. X is exponentially distributed with parameter l. Y has the gamma density r(2, .t).
Fx.r(x, y) = 1 - e-J.x - lxe-A)', 0 :S x :S y;
Fx.r(x, y) = 1 - e- 1Y(1 + Ay), 0 :S y < x; and Fx.r(x, y) = 0 elsewhere.
a > -1, (b) c = (a + 1)(a + 2),
(c) fx(x) =(a+ 2)(1 - x)+t, 0 < x < 1, andfx(x) = 0 elsewhere;

8. (a)

/y(y)

9.

=(a+ 2)y+t, 0

c = ../15/4n.

10. fr-x(z) =

:S y :S

1, and/y(Y) = 0 elsewhere.

X is distributed as

n(O, 16/15) and Y is distributed as n(O, 4/15).

s:oo fx(x)fr(z + x)dx.

=

A1 A2

(e-J.2 z - e-J.z), z > 0, and/x+y(z)

lt - .t2
(b) fx+r(z) = 0, z :S 0;/x+r(z) = 1 - e-Az, 0 :S z :S 1;
/x+r(z) = e-Az(e1 - 1), 1 < z < oo .

11. (a) fx+y(z)

r (Z )
12 JX+Y

a +-2 Z + 1 ,
= -

2
/x+r(z) = 0 elsewhere.

0 :S z :S 1, fx + r(z)

= a+
- - 2 (2 2

= 0, z

z )41

+t

:S 0.

1 < z :S 2;

246

Answers

13. fir-x 1(z)

14.

fz(Z) =

15. (a 1

17. fR(r)

2- (1 - _z_),
b-a
b-a

= -

_!_ Joo

lbl -oo
1)/(a1 + a 2

= ' 2 e-r212

(1

(x, z - ax) dx,

2
,

0 < z
-00

b - a, and fir-x 1(z) = 0 elsewhere.

< z <

00.

2).

16. n(Jl2 - Jltt uf + ui).

r > 0; and /R(r)

= 0,

s 0.

(]

18. /xr(z)

oo

f(x, zfx)dx.

-oo lxl

20. /z(z) = 2/n(l + z 2 ), z > 0, and/z(z)

0, z ~ 0.

21. lr1x(z) = 1/(1 + z) 2 , z > 0, andfr1x(z)

22. Beta density with parameters ct 1 and a 2

= 0, z

0.

23. (a) /r 1x(x) = k-l(y-.x), 0 s x s y, andfr1x(Yix) = 0 elsewhere.

(b) /rlx(Y I x) = (a + 1)(y - xt/(1 - xt+t, 0 ::; x < y < 1,
and lr 1x(Y I x) = 0 elsewhere.
(c) lr,x(Y I x) = n(y; x/8, 1/4).

24. cl>(3/2) = .933.

26. Beta density with parameters ct 1 + y and a 2

+n

- y.

27. fr(Y) = apf(y + p)+ 1 , y > 0, and /y(y) = 0, y s 0. The conditional density
of A given Y = y is the gamma density r(a + 1, p + y).

fi (

28 . fr(Y) = V2/n
- 3- y 2 e -y2; 262,

y 2:: 0, and r y)

0,

y < 0.

(]

30. /y(y) = y 2 j2, 0 ~ y < 1;/y(y)

fr(Y) = y/2 - 3y + 9/2, 2 ~ y
P(X1 + X2 + x3 s 2) = 5/6.

-y2 + 3y- 3/2, 1 s y < 2;

~ 3, and /y(y) = 0 elsewhere.

31. /x 1.x2 .x3 (Xtt x2, x3) = 1fxtx2, 0 < x 3 < x 2 < x 1 < 1, and equals zero elsewhere.
/x3(x) = (loge x)2 /2. 0 < x < 1, and equals zero elsewhere.
1)(y - x)"- 2 0 < x s y < 1, and equals zero elsewhere;
(b) /R(r) = n(n - 1)(1 - r)r"- 2, 0 < r < 1, and zero elsewhere.
(c) Beta density with parameters k and n - k + 1.

32. (a) /x 1 x"(x)

= n(n -

33. Exponential with parameter nl.

34.

x("/ 2 )-l e-.x/2 {2"12

r(n/2),

> 0, and 0 elsewhere.

35. Beta density with parameters m/2 and n/2.

36. aX + bY and bX - a Y are jointly distributed as independent random variables
each having the normal density n(O, a 2

+ b 2 ).

37. fx,x+r(x, z) = /(x)f(z- x).

38. Uniform on (0, z) for z > 0.

39. Uniform on (0, z) for 0 < z ~ c, and uniform on (z - c, c) for c < z < 2c.

Answers

247

41 . fw,z(w, z) = ( z ) f ( z , wz ) .
w+1
w+1 w+1

CHAPTER 7

1. cxtf(cx 1 + rx 2 ).
2. Z will have finite expectation when cx 1 > 1 and cx2 > 0. In this case EZ
rx2 /(rx 1

1).

3. uJifn.
4. Xefe has a geometric distribution with parameter (1 - e- 1 e).
EXe

5.
6.

ee-lt/(1 - e-lt). limt.... 0 EXt

1/)..

EX"' = f(cx 1 + cx2 ) f(cx 1 + m)/r(cx 1 ) r(cx 1

Var X = cx 1rx2 /(rx 1 + cx 2 + 1)(cx 1 + cx2 ) 2

+ cx2 +

m).

vir (n; )/r (~) .

1

8. a2 (cx 1 + a2 - l)/(rx 1 - 1)2 (a 1

9. EY = 3/2).. Var Y = 5/4).2

2) for cx 1 > 2.

10. EX= 2R/3, Var X= R 2 /18.

11. EX= 0, VarX = R 2 /4.

12. EZ

= a .J n/2, Var Z = u 2 (2 - n/2).

13. EY = m .J2jn, Var Y = u 2 (3 - 8/7r).

14. EX= 0, Var X= 1/2.

15. (a) E I X

I=

u .J2fn, Var I X I = u 2 (1 - 2/n);

(b) EX 2 = u 2 , Var X 2 = 2u4 ;
(c) Ee'X = ealtl/2, Var e'X = e2alt2 - ea2t2.

16. Ee'x = ( - ).- )" for t < )..

). -

17. EX'

+ r )/f(cx)).' for r > -ex.

k /(n + 1), Var Xk = k(n - k + 1)/n + 1) 2 (n + 2).

= f(cx

19. EXk =

248

Answers

20. ER = (n - 1)/(n + 1), Var R

= 2(n -

1)/(n

+ 1)2 (n + 2).

21. p = 1/4.

22. EZ = JJ.a/A., Var Z = a(u 2a + u 2 + JJ. 2)fA.2.

25. P3 ~ .458.
26. E [Y I X = x] = x, 0 < x < 1; E [Y I X= x] = 2 E[Y I X= x] = 0 elsewhere.

27. E[X I Z

+ a 2 ) for z > 0 and 0 elsewhere.

+ y)f(a 1 + a 2 + n), y = 0, 1, 2, . . . , n, and 0 elsewhere.

= z] = a 1 z/(a 1

28. E[ll I Y = y] = (a 1

33.

x, 1 ~ x ~ 2; and

P(X ~ x) ~ 4>((A.x - a)/v'~).

34. (a) EXf = u 2 and Var Xf = 2u 4

+ + x;

{b) P(Xt

:$

x) ~ <l>((x - nu 2 )/a 2 V2n).

35. (a) .921. (b) .842. (c) 23.26. (d) 27.71.

37. .02.

36 . .9773.

38. .0415.

39. .0053.
40. (a) fx(x) ~

). -t/ 2 q~((x - A.)/.J~),

(b) fx(x) ~ 4>((x

+ 1/2 - A.)/.J~) -

4>((x -

1/2 - A.)fv'l).

41. lfv'mr.
42. 1/v'me. Approximation (15) is not directly applicable because the greatest common
divisor of the set {x - 1 I x is a possible value of S 1 } is two rather than one.
43 . .133.

45. n ~ 6700.

44. .523.

46. 551.

CHAPTER I
1 . Mx(t) =

(ebt - e11')/(b - a)t, t

=F 0, and Mx(O)

= 1.

2. e11' M x(bt ).
4. (a) Mx(t) = [pf(l - e'(l - p))]", - oo < t < log (1 /(1 - p)).
5. (b) (2n)!

6. (a) dMx(t)

= npe'(pe'

dt
d2Mx(t)

dt 2

= npe'(pe' +

1 - p)"-

1 - p)"-t

10. el(e't-1).
'Px(t) =

.l.-+ 00

n(n - 1)p2e2'(pe'

13. A.t<A. - ir> r.

<l>x(e1').

21. (a) lPx+r(t) = e- 2 1'1 and

23. (b) lim

and
1 - p)"-2.

11. p/(1 - e 1'(1 - p)) .

12. [p/(1 - e"(1 - p)) ]".

14.

'P(x+Y)/ 2 (1) = e-1'1 .

P(X.l.v';.- A. ::;; x) =

~(x),

- oo <

x< oo .

Answers

249

CHAPTER 9

2. (a) 1/3, (b) 0, (c) 1250.

3. (a) ((10/9)50 - (10/9)75)/(1 - (10/9)' 5 )
(b) ~ $44.75,
~

(c)

6. For x

850.
= y

p{ b}(y y) = 1 - - a,

~ .93,

b-a

- ---2(y - a)(b - y)

'

and
G

{a,b}

) = 2(y -

y, Y

a)(b - y) _

-a

For x < y

x-a

P<a,b}(x, y) = - y- a

and

_ 2(x - aXb - y)
G{a bl(X , Y)
'

b-a

For x > y

b-x

P{a,b}(x, y) = -b -

-y

and

_ 2(y - a)(b - x)
G{a b)(X, Y)
b- a

'

7. For x = y
P{o}(y, y) = 1 - 1/2y and G{o}(y, y) = 2y For x < y
P{o}(x, y) = x/y and G{o}(x, y) = 2x.

1.

For x > y
P{o}(x, y)

1 and G{o}(x, y)

2y.

8. For x = y

Pll'(y, y)

q - p and Gll'(y, y) =

q - P
p-q

For x < y

= 1 and G0 (x, y) = 1/{p -

Pll'(x, y)

q).

For x > y

P (x, y)
0

(tj_)x-y and G (x, y) =

0

9. P<o>(-1, -1)

(qfp)x-y.

p-q

= qandG<o>(-1, -1) =

!!.
p

For y < -1
P ( 1 y) {O} -

11. P =

;:z

'

P - q
q((qfp)Y -

11

and

00

foR x[(x .+ z)dx)dz.

G ( 1 y) {O} -

1
q(qfp)"

250

Answers

12. (;) p''(l 13.

15.

(nR 2p)"

k!

p)"-".

-JCR2p

P~c(r, n) =

r) '" (-1i (' ~ k) (t - + k)"

J=O

16. (a) 1 - e-J.h, (b) 1 - e-J.h.

17.

Fzc(x) = 0, x < 0; and Fzc(x) = 1 - e-).x, x ;;::; 0.

18. Fyt(x) = 0, x < 0; Fylx) = 1 - e-.tx, 0 ~ x < t; and Fyt(x) = 1, x ~ t.

= 1 - e-J.x(1 + A.x), 0 ~ x < t;
A.t)~ t !5 x < oo .

19. (b) Frc+z,(x) = 0, x < 0; Frc+zt(x)

and Fr,+zc(x) = 1 - e-lx(l

20. A.e-J..
21. IN/k) = vA."/0. + v)"+ 1 , k = 0, 1, 2, ... , and zero elsewhere.

22.

fNT(k) = -

[1 - e-la

A.a

(A.~)J]

i=O

, k

1!

= 0, 1, 2, ... , and zero elsewhere.

23. A.tf(A.l + A.2).

2nx ( 1 - x
24. ID 1(X) = ~

72

25. EDT= r'"n! r (; +

1
)"- ,

0 =::;; x ~ r, and 0 elsewhere.

1) jr (; +

1).

26. (a) A.2 IAI2 + A.IAI, (b) A.2IAIIBI + A.IA n Bl.

29. (a) ID (r) = 3(4nA./3rr 3 m-Ie- 4 f().'313/(m - 1)!, r > 0, and 0 elsewhere.
'"
(b) Gamma
density r(m, 4nA./3).
30. (a) fD (r) = (nA.)'"r 2 m-I e-dr2 / 2 /2'"- 1(m - 1)!, r > 0, and 0 elsewhere.
'"
1/2
(b) ED.,. = (A.rr/2)r(m + 1/2) and ED~ = 2m.
(m - 1)!
nA.

31.

(a) Pr = -1

i'
0

e-p<t-s> ds = 1 - e -pt .
Jl.t

Table I

Values of the stanclanl normal distribution fUDCtion

ct>(z) =

J
-

I
v2,;

- - e-" 2 12 du = P(Z S z)
>

-3.

.0013 .OOJO .0007 .0005 .0003 .0002 .0002 .0001 .0001 .0000

-2.9
- 2.8
-2.7
-2.6
-2.5
-2.4
-2.3
-2.2
-2.1
-2.0
-1.9
-1.8
-1.7
-1.6
-1.5
-1.4
-1.3
- 1.2
-1.1
-1.0
- .9
- .8
- .7
- .6
- .5
- .4
- .3
- .2
- .1
- .0

.0019
.0026
.0035
.0047
.0062
.0082
.0107
.0139
.0179
.0228
.0287
.0359
.0446

.0548
.0668
.0808
.0968
.1151
.1357
.1587
.1841
.2119
.2420
.2743
.3085
.3446
.3821
.4207
.4602
.5000

.0018
.0025
.0034
.0045
.0060
.0080
.0104
.0136
.0174
.0222
.0281
.0352
.0436
.0537
.0655
.0793
.0951
.1131
.1335
.1562
.1814
.2090
.2389
.2709
.3050
.3409
.3783
.4168
.4562
.4960

.0017
.0024
.0033
.0044
.0059
.0078
.0102
.0132
.0170
.0217
.0274
.0344
.0427
.0526
.0643
.0778
.0934
.1 112
.1314
.1539
.1788
.2061
.2358
.2676
.3015
.3372
.3745
.4129
.4522
.4920

.0017
.0023
.0032
.0043
.0057
.0075
.0099
.0129
.0166
.0212
.0268
.0336
.0418
.0516
.0630
.0764
.0918
. 1093
.1292
.1515
.1762
.2033
.2327
.2643
.2981
.3336
.3707

.0016
.0023
.0031
.0041
.0055
.0073
.0096
.0126
.0162
.0207
.0262
.0329
.0409
.0505

.0618

.0749
.0901
.1075
. 1271
. 1492
.1736
.2005.2297
.2611
.2946
.3300
.3669
.4090 .4052
.4483 .4443
.4880 .4840

.0016 .0015 .0015 .0014

.0022 .0021 .0020 .0020
.0030 .0029 .0028 .0027
.0040 .0039 .0038 .0037
.0054 .0052 .0051 .0049
.0071 .0069 .0068 .0066
.0094 .0091 .0089 .0087
.0122 .0119 .0116 .0113
.0158 .0154 .0150 .0146
.0202 .0197 .0192 .0188
.0256 .0250 .0244 .0238
.0322 .0314 .0307 .0300
.0401 .0392 .0384 .0375
.0495 .0485 .0475 .0465
.0606 .0594 .0582 .0570
.0735 .0722 .0708 .0694
.0885 .0869 .0853 .0838
.1056 .1038 .1020 .1003
.1251 .1230 .1210 .1190
.1469 .1446 .1423 .1401
.1711 .1685 . 1660 .1635
.1977 .1949 .1922 .1894
.2266 .2236 .2206 .2177
.2578 .2546 .2514 .2483
.2912 .2877 .2843 .2810
.3264 .3228 .3192 .3516
.3632 .3594 .3557 .3520
.4013 .3974 .3936 .3897
.4404 .4364 .4325 .4286
.4801 .4761 .4721 .4681

.0014
.0019
.0026
.0036
.0048
.0064

.0084
.0110
.0143
.0183
.0233
.0294
.0367
.0455
.0559
.0681
.0823
.0985
.1170
.1379
.1611
.1867
.2148
.2451
.2776
.3121
.3483
.3859
.4247
.4641

Reprinted with permission of The Macmillan Company from INTRODUcnON TO

PROBABILITY AND STATISTICS, second edition, by B. W. Lindgren and G. W.
McElrath. Copyright 1966 by B. W. Lindgren and G. W. McElrath.

262

Values of the standard normal distribution fuoctioo

Table I

.0

.5000
.5398
.5793
.6179

---

1.9
2.0
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9

.5040
.5438
.5832
.6217
.6554 .6591
.6915 .6950
.7257 .7291
.7580 .7611
.7881 .7910
.8159 .8186
.8413 .8438
.8643 .8665
.8849 .8869
.9032 .9049
.9192 .9207
.9332 .9345
.9452 .9463
.9554 .9564
.9641 .9648
.9713 .9719
.9772 .9778
.9821 .9826
.9861 .9864
.9893 .9896
.9918 .9920
.9938 .9940
.9953 .9955
.9965 .9966
.9974 .9975
.9981 .9982

3.

.9987 .9990 .9993 .9995

.1
.2
.3
.4

.5
.6
.7
.8
.9
1.0
1.1
1.2
1.3
1.4
1.5
L6
1.7

1.8

.5080
.5478
.5871
.6255
.6628
.6985
.7324
.7642
.79j9
.8212
.8461
.8686
.8888
.9066
.9222
.9357
.9474
.9573

.9656
.9726
.9783
.9830
.9868
.9898
.9922
.9941
.9956
.9967
.9976
.9982

.5120 .5160
.5~17 .5557
.5910 .5948
.6293 .6331
.6664 .6700
.7019 .1054
.7357 .7389
.7673 .7703
.7967 .7995
.8238 .8264
.8485 .8508
.8708 .8729
.8907 .8925
.9082 .9099
.9236 .9251
.9370 .9382
.9484 .9495
.9582 .9591
.9664 .9671
.9732 .9738
.9788 .9793
.9834 .9838
.9871 .9874
.9901 .9904
.9925 .9927
.9943 .9945
.9957 .9959
.9968 .9969
.9911 .9977
.9983 .9984
.9997

.5199
.5596
.5987
.6368
.6736
.7088
.7422
.7734
.8023
.8289
.8531
.8749
.8944
.9115
.9265
.9394
.9505
.9599
.9678
.9744
.9798
.9842
.9878
.9906
.9929
.9946
.9960
.9970
.9978
.9984

.5239
.5363
.6026
.6406
.6772
.7123
.7454
.7764
.8051
.8315
.8554
.8770
.8962
.9131
.9278
.9406
.9515
.9608
.9686
.9750
.9803
.9846
.9881
.9909
.9931
.9948
.9961
.9971
.9979
.9985

.5279
.5675
.6064
.6443
.6808
.7157
.7486
.7974
.8078
.8340
.8577
.8790
.8980
.9147
.9292
.9418
.9525
.9616
.9693
.9756
.9808
.9850
.9884
.9911
.9932
.9949
.9962
.9972
.9979
.9985

.5319
.5714
.6103
.6480
.6844
.7190
.7517
.7823
.8106
.8365
.8599
.8810
.8997
.9162
. 9306
.9430
.9535
.9625
.9700
.9762
.9812
.9854
.9887
.9913
.9934
.9951
.9963
.9973
.9980
.9986

.5359
.5753
.6141
.6517
.6879
.7224
.7549
.7852
.8133
.8389
.8621
.8830
.901S
.9 177
.9319
.9441
.9545
.9633
.9706
.9767
.9817
.9857
.9890
.9916
.9936
.9952
.9964
.9974
.9981
.9986

.9998 .9998 .9999 . 9999 1.0000

Note I : If a normal variable X is not "standard,.. its values must be "standardized": Z "' (X is, P(X

x)

(X-"')

~~>)fq,

That

(I - t 1 -

Noto 2: For "two-tail" probabilities, sec Table lb.

Note 3: For z :1!:: 4, ~x) - ~ to four decimal places; for z ~ - ,4,
Note 4: The entries opposite z 3 are for 3.0, 3. I, 3.2, etc:.

~z)

.. _0 to four decimal places.

253

Index

Absolutely continuous distribution function,

115

sum of independent random variables,

204
Uniqueness Theorem, 208
Chebyshev's Inequality, 101
Chi-square (x2 ) distribution, 164
mean, 177
moments, 177
variance, 177
Combinations, 31-34
Committee membership, 32
Complement of an event, 3, 6

Bayes' rule, 17, 155

Bernoulli distribution, 66. See also Binomial
distribution
Bernoulli trial~ 66
infinite sequences of, 70
Beta distribution, 148
Beta function, 149

Complex numbers, 2~202

Complex-valued random variable, 202
Conditional density. discrete, 107
in Bayes rule, 155
with respect to integration, 153, 160
Conditional expectation, continuous random
variable, 182
discrete random variable, 108

Binomial coefficients, 31
Binomial distribution, 51
application of Chebyshev's Inequality,
102
Bernoulli trials, 66
mean, 83, 89
moment generating function, 198
normal approximation, 188, 190
Poisson approximation, 69
probability generating function, 73
sums of binomial random variables, 75
variance, 97
Birthday problem. 29
Bivariate distribution, 143
normal, 172
standard normal, 144

Conditional probability. 14
involving random variables, 57
Constant random variable, 52
characteristic function, 202
Continuity Theorem, 208
Continuous random variable, 109, 113
Convolution, 146

Correlation coefficient, 99, 176

Cauchy distribution, 122
sum of Cauchy random variables, 21 S

Coupon problem, 46
Covariance,96, 105,176,178

Central Limit Theorem. 185, 212

application to sampling, 190
local form, 187-189
normal approximation, l86
Change of variable formula, multidimensional, 166-168
one-dimensional, 119

Decay, exponential, 5, 11 I
Deciles, 133
DeMoivre-Laplace Limit Theorem, 184
De Morgan's laws, 10

Characteristic function, 200

Continuity Theorem, 208 .
inversion formula, 205-207

Density. See Discrete density; Density with

respect to integration

255

256
Density with respect to integration, 115
beta, 148
bivariate, 140, 143
chi-square (X 2 ), 164
conditional, 107, 153, 160
exponential, 119
F, 164
gamma, 129
joint, 140, 143, 157, 158
marginal, 141 , 158
Maxwell, 171
normal, 125
Rayleigh, 170
symmetric, 123
t, 165
Discrete density function, 50, 54
Bernoulli, 66
binomial, 51
conditional, 107
geometric, 55
hypergeometric, 52
joint, 62
marginal, 62
multinomial, 68
negative binomial, 55
Poisson, 56
symmetric, 123
Discrete random variable, 50
Discrete random vector, 61
Distribution, 51
Distribution function, 110, 115
absolutely continuous, 11 S
Cauchy, 122
discrete random variable, 57-58
gamma, 130
geometric, 59
inverse, 131
joint, 139, 157
marginal, 140, 157
normal, 125
properties, 112
symmetric density, 124
transformations involving, 131
uniform, 118

Error function, 136

Events, 3, 6
complement, 3, 6
independent, 19, 20
intersection, 3, 6
union, 3, 6, 38
Expectation, complex-valued random variable, 202
conditional, 108, 182
continuous random variable, 173
discrete random variable, 84
function of random variables, 86, 176

Index

general definition, 176

properties, 85, 176
Exponential decay, 5, 111
Exponential distribution, 119, 126
characteristic function, 203-204
mean, 174
moment generating function, 198
moments, 177
special property, 127
sums of exponential random variables,
146, 159, 168
variance, 177
waiting times for Poisson process, 230

F distribution, 164
Failure rate, 137
Field of sets, 7
sigma field, 7

Gamma distribution, 129

distance to particles in Poisson process,
229
mean, 174
moment generating function, 198
moments, 177
normal approximation, 185-186
quotients of gamma random variables,
152
sums of gamma random variables, 148,
159
variance, 177
waiting times in Poisson process, 230
Gamma function, 129
Geometric distribution, 55, 56
distribution function, 59
mean, 84-85,91,96
probability generating function, 73
special property, 59-60
sums of geometric random variables,
72, 75-76
variance, 96
waiting times in Bernoulli trials, 70

Half-life, 133
Hypergeometric distribution, 52
mean, 90
variance, 98

Independent events, 19, 20

Independent random variables, 63, 64, 66,
142, 143, 154, 158, 159
quotients, 151
sums, 72, 146

257

Index

Indicator random variable, 52. See also

Bernoulli distribution
Intersection of events, 3, 6
Inversion formulas involving characteristic
functions, 205-207
Jacobians, 167
Joint density, discrete, 61, 62
with respect to integration, 140, 143,
157, 158
Joint distribution function, 1.39, 157
Lognormal distribution, 136
Lower decile, 1. 33
Lower quartile, 133
Marginal density, discrete, 62
with respect to integration, 141, 158
Marginal distribution function, 140, 157
Matching prol)lems, 31, 40
Maxwell distribution, 171
Maxwell's law, 126
Mean, 83, 176
Median, 133
Moment generating function, 197
computation of moments, 199
sum of independent random variables,
199
Moments, 92, 176-177
central, 92, 176-177
Multidimensional change of variables, 166168
Multinomial distribution, 68
application to order statistics, 163
connection with Poisson process, 226
Mutually independent events, 19-20
Mutually independent random variables.
See Independent random variables
Negative binomial distribution, 55
mean, 95-96
normal approximation, 185-186
probability generating function, 73
sums of negative binomial random variables, 75
variance, 95-96
waiting times in Bernoulli trials, 71
Normal approximation, 186
Normal distribution, 124-126
bivariate, 172

Central Limit Theorem, 184-185, 212

characteristic function, 204-205
inversion formula, 207
mean, 178
moment generating function, 197-198
moments, 178, 199-200
normal approximation, 186
sampling distributions, 163
standard, 125
standard bivariate, 143-144
sums of normal random variables, 149,
159
transformations involving, 132
variance, 178
Occupancy problems, 43
Order statistics, 160
Ordered samples, 27-30
Pairwise independent events, 19
Partitions, 34-38
Percentiles, 133
Permutations, 29-31
Poisson approximation to binomial distribution, 69
Poisson distribution, 56
approximation to binomial distribution,
69,
mean, 84
moment generating function, 198
normal approximation, 185
probability generating function, 74
relation to gamma distribution, 130
sums of Poisson variables, 75
variance, 96
Poisson process, 228
distance to mth nearest particle, 228
waiting times, 230
Poker hand, 47
Polya's urn scheme, 18
Possible value, 50
Probability generating function, 73
computation of moments, 94
sums of independent random variables,
74
Probability measure, 8
Probability space, 8
Probability theory, 1
Probable error, 134
Quartiles, 133
Quotients of random variables, 150

258

Index

Random sample, 101

Symmetric density, 123

median, 133
moments, 178
Symmetric probability space, 9, 27
Symmetric random variable, 123
median, 133
moments, 178

Random variable, 110

complex-valued, 202
continuous, 109, 113
discrete, 50
symmetric, 123
Random walk, 216
simple, 220
Range, 160
Rayleigh distribution, 170
Regression function, 182

t distribution, 165

Relative frequency interpretation, 1-3

conditional probability, 14
expectation, 82

Uncorrelated random variables, 99

Uniform distribution, discrete, 55
mean, 82-83

Sampling distributions, 163

Sampling with replacement, 28. See also
Binomial distribution

Uniform distribution on an interval, 118

characteristic function, 203
mean, 173
transformations involving, 131-132

Sampling without replacement, 29, 31, 37- 38,

Uniform probability space, 9-10

Union of events, 3, 6, 38
Uniqueness theorem involving characteristic
functions, 208

52
Schwarz inequality, 99
Sigma field (a-field) of subsets, 7
Simple random walk, 220
Standard bivariate normal
143-144
Standard deviation, 94, 176

distribution,

Standard normal distribution, 125

Statistical regularity, 1
Stochastic process, 216
Sums of independent random variables
characteristic function, 204
continuous, 145
discrete, 72
moment generating function, 199
probability generating function, 74
variance, 97

Unordered samples, 31-34

Upper decile, 133
Upper k-percentile, 133
Upper quartile, 133

Variance, 93, 176, 177

Waiting times, Bernoulli trials, 70

Poisson process, 230
Wald's identities, 217-218
Weak Law of Large Numbers, 102, 211

Introduction
to Probability
Theory