Real Analysis

July 10, 2006

1 Introduction
These notes are intended for use in the warm-up camp for incoming Berkeley Statistics
graduate students. Welcome to Cal! The real analysis review presented here is intended to
prepare you for Stat 204 and occasional topics in other statistics courses. We will not cover
measure theory topics and some other material that you should be very familiar with if you
intend to take Stat 205. If you have never taken a real analysis course, you are strongly
encouraged to do so by taking math 104 (or the honors version of it). Math 105 (usually of-
fered in the spring) will provide you with necessary measure-theoretical background essential
for Stat 205. The presentation follows closely and borrows heavily from Real Mathematical
Analysis by C.C. Pugh, the standard textbook for honors version of math 104. The empha-
sis is on metric space concepts and the pertinent results on the reals are presented as specic
cases of more general results, and a lot of them are presented together as exersises in section
3.8. We do not expect you to be familiar with the metric space concepts but we do expect
you to be familiar with specic results on real line, as that is usually the approach taken in
1
most real analysis courses. We hope that you nd these notes helpful! Go Bears!
2 Some Denitions
We will denote by R, Q, Z, and N the sets of all real numbers, rational numbers, integers
and positive integers, respectively. We will take for granted the familiarity with notions
of nite, countably innite and uncountably innite sets. Given a set S R, M R is an
upper bound for S if s S it is true that s M. S is said to be bounded above by M.
M

is said to be the least upper bound (or l.u.b) for S if for all upper bounds M it is
true that M

M (note that it implies that l.u.b. is unique). The lower bounds and the
greatest lower bound (g.l.b) are dened similarly. For example, if S=[0,1], then 1 and 4 are
upper bounds, and 0, -4 are lower bounds with 0 and 1 being g.l.b and l.u.b, respectively.
A set is said to be bounded if it is bounded from above and below. A set S is said to be
unbounded from above if N s S s.t. s > N. The denition of a set unbounded
below is similar. A set is unbounded if its either unbounded from above or below. The
supremum (sup) of a set S R is dened to be l.u.b for S if S is bounded from above,
and to be + otherwise. The innum (inf) is dened to be g.l.b for a set bounded form
below and to be otherwise.The following results will be assumed about R (proofs can
be looked up in any analysis textbook):
1. If S R is bounded from above/below then the l.u.b/g.l.b for S exists and is unique
2. Triangle Inequality: [x + y[ [x[ +[y[
3. -principle: If x,y R , and > 0, x y + then x y. Also if > 0, [x y[ ,
2
then x=y.
4. Every interval (a,b) contains countably innitely many rationals and uncountably in-
nitely many irrationals.
3 Metric Spaces
3.1 Denition
A metric space M is a set of elements together with a function d:M M R (known as
metric) that satises the following 3 properties. For all x, y, z M:
1. d(x,y) 0 and d(x,y)=0 i x=y
2. d(x,y) = d(y,x)
3. d(x,y) d(x,z) + d(z,y)
When metric d is understood, we refer to M as the metric space. When we want to specify
that the metric is in M , we might use notation d
M
(x, y). It also helps sometimes to think of
d as the distance function, since it makes the 3 properties more intuitive (we usually think
of distance as being nonnegative, and the distance from A to B should be the same as the
distance from B to A). Here are some examples:
1. R with usual distance function: d(x,y) = [x y[
2. Q with the same metric
3. R
n
with Euclidean distance d(x,y) = |x y|
3
4. Any metric space (for ex. R or N) with the discrete metric: d(x,y)=1 if x ,= y,
d(x,y)=0 otherwise. This metric makes the distance from a point to itself be 0 and the
distance between any two distinct points be 1.
You should check for yourself that the metrics above satisfy the 3 conditions.
3.2 Sequences
We will use the notation (x
n
) for the sequence of points x
1
, x
2
, . . . , x
n
, . . . in metric space
M. The members of a sequence are not assumed to be distinct, thus 1,1,1,1,. . . is a legit-
imate sequence of points in Q. A sequence (y
k
) is a subsequence of (x
n
) if there exists
sequence 1 n
1
< n
2
< n
3
< . . . s.t. y
k
= x
n
k
. Some subsequences of the sequence
1,1,2,1,2,3,1,2,3,4,1,2,3,4,5,. . . are:
1. twos: 2,2,2,2,2,. . .
2. odds: 1,3,5,7,9,. . .
3. primes: 2,3,5,7,11,. . .
4. original sequence with duplicates removed: 1,2,3,4,5,6,7,. . .
5. the previous subsequence with rst 3 elements removed: 4,5,6,7,. . .
A sequence (x
n
) of points in M is said to converge to the limit x in M if > 0 N
s.t. n N = d(x
n
, x) < . We then say that x
n
x. Notice that if our metric space
is R, then replacing d(x
n
, x) by the usual metric [x
n
x[ gives the familiar denition of a limit.
4
Theorem: The limit of a sequence, if it exists, is unique
Proof: Let (x
n
) be a sequence in M that converges and suppose its limit is not unique.
Let x,y denote two (of possibly even more) limits. Let > 0 be given. Then N
1
s.t. n N
1
= d(x
n
, x) < /2. Similarly N
2
s.t. n N
2
= d(x
n
, y) < /2. Let N = max(N
1
, N
2
),
and let n N. Then by the 3rd property of metric function:
d(x, y) d(x, x
n
) + d(x
n
, y) < /2 + /2 = .
Since this is true for every we have x=y by the -principle
Theorem: Every subsequence of a convergent sequence converges, and it converges to
the same limit as the original sequence.
Proof: Easy.
A sequence (x
n
) in M is said to be Cauchy if > 0 N s.t. n, m N = d(x
n
, x
m
) < .
In other words a sequence is Cauchy if eventually all the terms are all very close to each other.
Theorem: Every convergent sequence is Cauchy
Proof: Suppose x
n
x in M. Let > 0 be given. Then N s.t. n N = d(x
n
, x)
< /2. Let n, m N. Then
5
d(x
n
, x
m
) d(x
n
, x) + d(x, x
m
) < /2 + /2 < = (x
n
) is Cauchy.
Does every Cauchy sequence converge to a limit? Consider the sequence 3, 3.14, 3.141,
3.1415, . . . This sequence is clearly Cauchy. When considered as a sequence in R, it does
converge to . However, we can also think of it as a sequence in Q, in which case it doesnt
converge, since / Q. The following denition formalizes the dierence between Q and R:
the metric space M is said to be complete if all Cauchy sequences in M converge to a limit
in M.
Theorem: R is complete.
Proof: Let (a
n
) be a Cauchy sequence in R and let A be the set of elements of the
sequence, i.e.
A=x R : n N and a
n
= x
Let =1. Then since (a
n
) is Cauchy, N
1
s.t. n, m N
1
, [a
n
a
m
[ < 1. Therefore, n N
1
we have [a
n
a
N
1
[ < 1 and thus n N
1
= a
n
[a
N
1
1, a
N
1
+ 1].
The nite set a
1
, a
2
, . . . , a
N
1
, a
N
1
1, a
N
1
+1 is bounded (as is every nite subset of R), and
therefore all of its elements belong to some interval [L, L]. Since both a
N
1
1, a
N
1
+ 1
[L, L], we conclude that [a
N
1
1, a
N
1
+ 1] [L, L] and therefore a
n
[L, L] n, and
A is bounded. Now, consider set
6
S = s [L, L] : innitely many n N, for which a
n
s
Obviously, L S and S is bounded from above by L. We then know that the l.u.b for S
exists (call it b). We will show that a
n
b.
Let > 0 be given. Then since (a
n
) is Cauchy, N
2
s.t. m, n N
2
= [a
n
a
m
[ < /2.
Since s S, s b, we have that b + /2 / S. That means that a
n
exceeds b + /2 only
nitely often and N
3
N
2
s.t. n N
3
= a
n
b + /2. Since b is the least upper bound
for S we have that b /2 is not an upper bound for S and therefore s S s.t. s > b /2
and a
n
s > b /2 innitely often. In particular that gurantees that there exists some
N
4
N
3
s.t. a
N
4
> b /2. Moreover, since N
4
N
3
we have a
N
4
(b /2, b +/2]. And
since N
4
> N
2
, we get n N
4
=
[a
n
b[ [a
n
a
N
4
[ +[a
N
4
b[ < /2 + /2 <
and therefore a
n
b.
It is an easy exersise to show that every discrete metric space (for example, N with dis-
crete metric) is complete.
3.3 Open and Closed Sets
Let M be a metric space and let S be a subset of M. We say that x M is a limit(point) of
S (note that S is not a sequence here, but a subset of M) if there exists a sequence (x
n
) in S
s.t. x
n
x. For example, let M = R and S=Q. Then 2 is an example of a limit point of S
7
((x
n
) = 2, 2, 2, 2, 2, . . . or (x
n
) = 1, 1 +1/2, 1 +2/3, 1 +3/4, . . .) that belongs to S, and is a
an example of a limit point of S ((x
n
) = 3, 3.1, 3.14, 3.141, 3.1415, . . .) that does not belong
to S (of course, it still belongs to M). A set is said to be closed (in underlying metric space
M) if it contains all of its limits. For example, a singleton set x is closed in M (assuming
M is non-empty), since the only possible sequence is (x
n
) = x, x, x, . . . x. Clearly, M
is a closed subset of itself. A set S is said to be open (in underlying mertric space M) if
x S r > 0 s.t. d(x, y) < r = y S. The set of points y M : d(x, y) < r is
called (open) r-neighborhood of x and is denoted B
r
(x) (for example, B
2
(5) in R is simply
the open interval (3, 7)). Thus a set S is open (in M) if for every point in S there exists
some small neighborhood of that point in M contained entirely in S. Its easy to see that
interval (a,b) is an open set in R, and clearly every metric space M is an open subset of itself
(since every r-neighborhood of x is still in M). The following theorem provides a connection
between open and closed sets:
Theorem: The compliment of an open set is closed and the compliment of a closed set
is open.
Proof:Suppose S M is open. Suppose x
n
x in M, and moreover x
n
S
c
n. We
need to show that x S
c
. Suppose not, then x S. Since S is open, r > 0 s.t. d(x, y) < r
= y S. Now, since x
n
x N s.t. n N = d(x
n
, x) < r = x
n
S. That is clearly
impossible since no point could be in both S and S
c
, and we have reached a condtradiction.
Therefore S
c
is closed.
Now suppose that S M is closed. Suppose S
c
is not open, then x S
c
s.t. r
n
> 0 x
n
8
s.t. d(x, x
n
) < r
n
but x
n
S. Now let r
n
= 1/n and pick x
n
as above. Then x
n
S n and
x
n
x S
c
= S is not closed (since it fails to contain all of its limits) = contradiction.
Therefore S
c
is open.
Notice that some sets (like the space M itself) are both closed and open, they are referred
to as clopen sets. By theorem above, M
c
= is clopen. Now consider interval [0, 1) R.
It is neither open (every r-neighborhood of 0 includes points in [0, 1)
c
R) nor closed (it
fails to include 1, which is the limit of the sequence x
n
= 1 1/n in [0, 1)). Thus subsets of
a metric space can be open, closed, both, or neither.
Theorem: Arbitrary union of open sets is open.
Proof: Suppose U

is a collection of open sets in M, and let U = U

. Then x U
= x U

for some and r > 0 s.t. d(x, y) < r = y U

= y U. Therefore U is
open.
Theorem: Intersection of nitely many open sets is open.
Proof: Suppose U
1
, U
2
, . . . , U
n
are open sets in M. Let U = U
k
. If U = then U is
open. Now suppose x U, then x U
k
for k = 1, 2, 3, . . . , n. Since each U
k
is open, r
k
> 0
s.t. d(x, y) < r
k
= y U
k
. Let r = min(r
1
, r
2
, . . . , r
n
), then d(x, y) < r = y U
k
k
= y U. Therefore U is open.
9
Notice that the innite intersection of open sets is not necessarily open. For example,
its easy to see that U
k
= (1/k, 1/k) is an open subset of R, but U
k
= 0 is clearly not
open in R.
Theorem:Arbitrary intersection of closed sets is closed. Also, nite union of closed sets
is closed.
Proof: Well use DeMorgans Laws: (U
k
)
c
=(U
k
c
). Let K

be a collection of closed
sets in M. Then K

= (K
c

)
c
and since each K

c
is open, their union is open and its
complement is closed. The proof of the second part of the theorem is similar.
Notice that the innite union of closed sets is not guaranteed to be closed. For example,
even though each K
k
= [0, 1 1/k] is closed in R, the intersection K
k
= [0, 1) is not.
Theorem:Let M be a complete metric space, N M be closed. Then N is complete as
a metric space in its own right.
Proof: Let (x
n
) be a Cauchy sequence in N. Since (x
n
) is also a Cauchy sequence in
M, and M is complete, we have x
n
x M. But N is closed in M, therefore x N and
we conclude that N is complete.
Let lim S denote the set of all limit points of S in M. It is pretty clear that x lim S
r > 0, B
r
(x) S ,= (you can easily prove it as an exersise). We can also show that
lim S is a closed set. Let > 0 be given and let (y
n
) be a sequence of points in lim S s.t.
10
y
n
y in M. Then N s.t. n N = d(y
n
, y) < /2. Since each y
n
lim S, x
n
S s.t.
d(y
n
, x
n
) < /2 n. Then we have n N =
d(x
n
, y) d(x
n
, y
n
) + d(y
n
, y) < /2 + /2 =
Therefore x
n
y and y lim S. We conclude that lim S is closed.
We can also show that B
r
(x) is an open subset of M as follows: let y B
r
(x). Let s =
r d(x, y) > 0. Then d(y, z) < s =
d(z, x) d(x, y) + d(y, z) < d(x, y) + (r d(x, y)) = r
= z B
r
(x). Thus if y B
r
(x) s s.t. B
s
(y) B
r
(x), and therefore B
r
(x) is open.
Theorem: Every open set U R can be expressed as a countable disjoint union of open
intervals of the form (a, b), where a is allowed to take on the value and b is allowed to
take on the vaule +.
Proof:If U = = (0, 0), then the statement is vacuously true. If U is not empty,
x U dene a
x
= infa : (a, x) U, b
x
= supb : (x, b) U. Then I
x
= (a
x
, b
x
) is
a (possibly unbounded) open interval, containing x. It is maximal in the following sense:
suppose b
x
U, then by construction above J U, an open interval s.t. b
x
J. You
can prove for yourself that the union of two open intervals with non-empty intersection is in
fact, an open interval (you just have to take care of a number of base cases for endpoints).
11
We then have that I
x
J is an open interval containing x, and moreover, since J is open,
and b
x
J, b

J s.t b

> b
x
. But then b

b : (x, b) U, which contradicts b

x
being
the supremum of such a set. Thereofre b
x
/ U, and similarly a
x
/ U. Now let x, y U, and
suppose I
x
I
y
,= . Then once again, I
x
I
y
is an open interval containing both x and y and
by maximality we have I
x
= I
x
I
y
= I
y
. Thus we conclude that x, y U either I
x
= I
y
or
the two intervals are disjoint. So, U is a disjoint union of open intervals. To show that the
union is countable, pick a rational number in each interval. Since the intervals are disjoint,
the numbers are distinct, and their collection is countable (since rationals are countable).
A few more denitions are in order. As before, S is some subset of a metric space M.
The closure of S is

S = K

where K

is the collection of all closed sets that contain S.

The interior of S is int(S) = U

, where U

is the collection of all open sets contained

in S. Finally, the boundary of S is S =

S

S
c
. For example, if M=R and S=[a, b], then

S = [a, b], int(S) = (a, b), and S = a b. If S=Q, then

S = R, int(S) = , and S = R.
Notice that the closure of a closed set S is S istelf, and so is the interior of an open set S.
Also notice that both

S and S are closed (as intersections of closed sets), and int(S) is
open as a union of open sets.
Theorem:

S=lim S.
Proof: We have shown before that lim S is a closed set, also since each point of S is
also a limit of point of S (its the limit of a sequence s, s, s, . . .), we have S lim S, and we
conclude that

S lim S. Also, since S

S, and

S is closed, it must contain all the limit
12
points of S, thus lim S

S. We conclude that

S = lim S.
You may check for yourself at this point that every subset of a discrete metric space M
(for example, N with discrete metric) is clopen (why would it suce to show that a singleton
x is open?) and that therefore S M, int(S) = S =

S and S = .
A subset S of a metric space M is said to cluster at point x M if r > 0 B
r
(x)
contains innitely many points of S. S is said to condense at x, if each B
r
(x) contains
uncountably many points of S. For example if M=R, then every point of S = [a, b] is a
condensation point (also a cluster point), and no other point is a cluster point. If S=1/n :
n Nthen the only cluster point is 0 / S, and no point is a condensation point (no point
could have uncountably many members of S in its r-neighborhood, since S itself is count-
able). If S=Q, then x is a cluster point of S x R. An open interval (a, b) is an example
of a subset of R that contains some but not all of its cluster/condensation points. Finally,
N with either discrete metric or with the metric it inherits as subset of R has no cluster
points. We emphasize that if x is a cluster point of S, each r-neighborhood of x must contain
innitely many distinct points of S. It is easy to see that x is a cluster point of S i (x
n
), a
sequence of distinct points in S, s.t. x
n
x. We denote the set of all cluster points of S by S

.
Theorem: S S

=

S. S is closed i S

S.
Proof: We already know that S

S. Moreover, by above we have that a cluster point
is a limit point and therefore S

lim S =

S. Thus S S

S. Also, if x

S = lim S, then
13
either x S or (x
n
), a sequence of distinct points in S, s.t. x
n
x, but that would make
x a cluster point. So we have

S S S

and we conclude that S S

=

S. Now, we know
that S is closed i S =

S = S S

i S

S.
We notice here that if S R is a bounded from above/below, then its l.u.b/g.l.b are its
cluster points (you can check it for yourself), and therefore by above theorem belong to the
closure of S. Thus a closed and bounded subset of R contains both its l.u.b and g.l.b.
Up to this point, instead of just saying S is open or S is closed, we often said S is open
in M or S is closed in M. We can drop the mention of M, when it is understood what the
underlying metric space is, but we point out that it is essential to the openness/closedness
of S. For example, both Q and the half-open interval [a, b) are clopen when considered as
metric spaces in their own right. Neither one, of course, is either open or closed when treated
as a subset of R. Another example is a set S = Q (, ), a set of all rational numbers in
the interval (, ). As a subset of metric space Q S is both closed (if (x
n
) is a sequence
in S, and x
n
x Q then x S) and open (check for yourself). As a subset of R, how-
ever, it is neither open (if x S then every neighborhood of x contains some y / Q) nor
closed (there are sequences in S converging to R). The following few theorems establish
the relationship between being open/closed in metric space M and some metric subspace N
of M that inherits its metric from M (i.e. d
N
(x, y) = d
M
(x, y). We will denote the clousre
of set S in M by cl
M
(S), and the closure of S in N by cl
N
(S). Note that cl
N
(S) = cl
M
(S)N.
Theorem: If S N M, then S is closed in N i L M s.t. L is closed in M and
14
S = L N.
Proof: Suppose S is closed in N, then let L = cl
M
(S). Clearly L is closed in M, and
L N = cl
N
(S) = S (since S is closed in N). Now suppose that L as in the statement of
theorem exists. Since L is closed, it contains all of its limit points and S = L N contains
all of its limit points in N, therefore S is closed in N.
Theorem If S N M, then S is open in N i L M s.t. L is open in M and
S = L N.
Proof:Notice that the complement of LN in N is L
c
N, where L
c
si the complement
of L in M. Now take complements and apply previous theorem.
A popular way to summarize the preceding two theorems is to say that metric subspace
N inherits its opens and closeds from M.
We also introduce here the notion of boundedness. S M is bounded if x M,
r > 0 s.t. S B
r
(x), i.e S is bounded if theres some point x M s.t. S is contained in
some neighborhood of x. For example, [1, 1] is bounded in R since its contained in B
5
(2)
or B
2
(0). On the other hand, the graph of function f(x) = sin(x) is an unbounded subset
of R
2
, although the range of f is a bounded subset of R (range = [1, 1]). In general, we
say that f is a bounded function if its range is a bounded subset of the target space.
15
Theorem: Let (x
n
) be a Cauchy sequence in M. Then S = x M : x = x
n
for some
n is bounded. In other words, Cauchy sequences are bounded.
Proof: Let = 1. Then N s.t. n, m N = d(x
n
, x
m
< 1, in particular n N =
d(x
n
, x
N
) < 1. Now, let r = 1+maxd(x
1
, x
2
), d(x
1
, x
3
), . . . , d(x
1
, x
N
). Clearly 1 k N
d(x
1
, x
k
) < r 1 < r. For k N, d(x
1
, x
k
) d(x
1
, x
N
) + d(x
N
, x
k
) < r 1 + 1 = r. Thus
S B
r
(x
1
) and the sequence is bounded.
As a consequence of preceding theorem, all convergent sequences are bounded (since
convergence implies Cauchy).
3.4 Continuous Functions
Let M, N be two metric spaces. A function f : M N is continous at x M if > 0
> 0 s.t. y M and d
M
(x, y) < = d
N
(f(x), f(y)) < . We say that f is continuous
on M if its continuous at every x M. Notice that a specic depends on both x and .
If the choice of does not depend on x, then we have the following denition: a function
f : M N is uniformly continuous if > 0, > 0s.t. y M and d
M
(x, y) < =
d
N
(f(x), f(y)) < .
For example, function 1/x on the interval (0, 1) is continuous (note that 0 is outside of the
domain), but is not uniformly continuous (given , no matter how small we choose to be,
there are always points x, y in the interval (0, ) s.t. [f(x) f(y)[ > ).
The following theorem provides some alternative characterizations of continous functions:
16
Theorem: The following denitions of continuity are equivalent:
1. , denition
2. f : M N is continuous if for each convergent sequence x
n
x in M, we have
f(x
n
) f(x) in N. Thus continous functions send convergent sequences to convergent
sequences, preserving the limits.
3. f : M N is continuous if closed S N, f
1
(S) is closed in M. (Here f
1
is the
notation for preimage of a set in a target space).
4. f : M N is continuous if open S N, f
1
(S) is open in M.
Proof: 1 = 2: Suppose f is continuous, (x
n
) is a sequence in M s.t. x
n
x. Let > 0
be given. By 1, we know that > 0 s.t. d
M
(x, y) < 0 =d
N
(f(x), f(y)) < . Since x
n
x
L s.t. n L = d
M
(x
n
, x) < . But that means that n L = d
N
(f(x
n
), f(x)) < , and
therefore f(x
n
) f(x).
2 = 3: Let S N be closed in N, and let (x
n
) be a sequence in S
1
= f
1
(S), s.t.
x
n
x M. We need to show that x S
1
. By 2 and by the fact that x
n
x, we know
that f(x
n
) f(x). Since (f(x
n
)) is a sequence in S and S is closed, we have f(x) S, and
therefore x S
1
.
3 = 4: Let S N be open in N. Then f
1
(S) = (f
1
(S
c
))
c
, and since the latter is the
complement of a closed set (by 3), it is open in M.
4 = 1: Let x M and > 0 be given. Then we know that B

(f(x)) is open in N and

17
therefore f
1
(B

(f(x)) is open in M (by 4). Since x f

1
(B

(f(x)) and f
1
(B

(f(x)) is
open, > 0 s.t. B

(x) f
1
(B

(f(x)), i.e. d
M
(x, y) < = d
N
(f(x), f(y)) < .
Theorem: Composite of continuous functions is continuous.
Proof: Let f : M N and g : N L be continuous, and let U L be open in L, and
denote h = g f M L. Then g
1
(U) is open in N and f
1
(g
1
(U)) = h
1
(U) is open in
M by denition 4 above. We conclude that h is continuous by denition 4.
It is worth pointing out that while continuous functions preserve the convergent se-
quences, they in general do not preserve the non-convergent Cauchy sequences. For exam-
ple, the continuous funcion f : (0, 1] R given by f(x) = 1/x maps the Cauchy sequence
1,1/2,1/3,1/4,. . . in (0, 1] to non-Cauchy sequence 1,2,3,4,. . . in R. Uniform continuity en-
sures the preservation of Cauchy sequences.
Theorem: Let M, N be metric spaces, f : M N a uniformly continuous function,
and (x
n
) a Cauchy sequence in M. Then (f(x
n
)) is a Cauchy sequence in N.
Proof: Let > 0 be given. Then since f is uniformly continuous, > 0 s.t. d
M
(x, y) <
= d
N
(f(x), f(y)) < . Since (x
n
) is Cauchy, L s.t. m, n > L = d
M
(x
m
, x
n
) < =
d(f(x
n
), f(x
m
)) < = (f(x
n
)) is Cauchy.
You can show pretty easily that every function dened on a discrete metric space is
18
uniformly continuous (why?).
Example: Consistent Estimates
In statistics, we usually estimate parameters of interest from the sample we have at
hand. Suppose that

n
is the estimate based on sample of size n, then we say that the
estimate is consistent in probability if > 0, P([

n
[ > ) 0 as n .
For example, the Weak Law of Large Numbers states that if we have a sequence of IID
random variables X
1
, X
2
, . . . , X
n
with expected value and variance
2
, then the sam-
ple mean
1
=

X =
1
n
n

i=1
X
i
is a consistent estimate of . Moreover, if we consider
the sequence X
1
k
, X
2
k
, . . . , X
n
k
, then by applying the Weak Law again, we conclude that

k
= X
k
=
1
n
n

i=1
X
i
k
is a consistent estimate of k-th moment of X,
k
= E(X
k
).
Theorem: Suppose

n
is a consistent estimate of , and suppose that f is continuous.
Then
n
= f(

n
) is a consistent estimate of = f().
Proof: Suppose not. Then for some

> 0, P([f(

n
) f()[ >

) 0, i.e. for some

L > 0 subsequence (

n
k
) of (

n
) s.t. P([f(

n
k
) f()[ >

) > L. Now, since f is con-

tinuous, > 0 s.t. [

n
[ = [f(

n
) f()[

, and therefore we have that

[f(

n
) f()[ >

= [

n
[ > . Also, since

n
is a consistent estimate of , we know
that N s.t. n N = P([

n
[ > ) < L, in particular this holds n
k
N. Combining
the last two results with the fact that (A = B) = (P(A) P(B)), we have n
k
N
= L < P([f(

n
k
) f()[ >

) P([

n
k
[ > ) < L, which leads to a contradiction.
19
Therefore we conclude that
n
= f(

n
) is indeed a consistent estimate of = f().
This result is very useful in general and is the foundation of estimation technique known
as Method of Moments. Once again, suppose we have a sequence of IID random variables
X
1
, X
2
, . . . , X
n
with expected value and variance
2
. We know that

X is a consistent
estimate of and by above result we have

X
2
is a consistent estimate of
2
. Then you
can prove yourself that X
2

X
2
is a consistent estimate of
2
= V ar(X) (do it!). In
general, if there is a continuous function = f(
1
,
2
, . . . ,
k
), then

= f(
1
,
2
, . . . ,
k
) is
a consistent estimate of . Therefore if we can express a parameter of interest as a continuous
function of the moments of distribution, then applying function to sample moments will give
us a consistent estimate of the parameter. For example, if we are sampling from N(,
2
)
distribution and we have a sample of size n, then =
1
and
2
= E(X
2
)
2
=
2

1
2
,
and therefore =

X and
2
=

1
n
n

i=1
X
2
i

X
2
=
1
n
n

i=1
(X
i

X)
2
are consistent estimates
of and
2
, respectively.
3.5 Product Metrics
Let M = M
1
M
2
be the Cartesian product of metric spaces M
1
, M
2
, i.e. M is the set of
all points x = (x
1
, x
2
) s.t. x
1
M
1
and x
2
M
2
. For example, R
2
= R R is the set of all
points in the plane. How would one dene a useful metric on this product space? Three of
the possibilities are listed below:
1. d
M
(x, y) =

d
M
1
(x
1
, y
1
)
2
+ d
M
2
(x
2
, y
2
)
2
= d
E
(x, y), a Euclidean metric
2. d
M
(x, y) = maxd
M
1
(x
1
, y
1
), d
M
2
(x
2
, y
2
) = d
max
(x, y)
20
3. d
M
(x, y) = d
M
1
(x
1
, y
1
) + d
M
2
(x
2
, y
2
) = d
sum
(x, y)
Continuing with our example, if we took points x = (5, 2) and y = (7, 11) in R
2
, then
d
E
(x, y) =

12
2
+ 9
2
= 15, d
max
(x, y) = max12, 9 = 12, and d
sum
(x, y) = 12 + 9 = 21,
where we use the usual distance metric on R. It turns out that in a way, all these metrics
are equivalent.
Theorem: d
max
(x, y) d
E
(x, y) d
sum
(x, y) 2d
max
(x, y).
Proof Some basic arithmetic.
Theorem: Let M = M
1
M
2
, and let (x
n
) = ((x
1,n
, x
2,n
)) be a sequence in M. Then
(x
n
) converges with respect to (wrt) d
max
i it converges wrt d
E
i it converges wrt d
sum
i
both (x
1,n
) and (x
2,n
) converge in M
1
and M
2
, respectively.
Proof: The equivalence of the rst 3 convergences is obvious from previous theorem.
For example, suppose (x
n
) converges to some x Mwrt d
max
. Let > 0 be given, then N
s.t. n N = d
max
(x
n
, x) < /2 = d
E
(x
n
, x) d
sum
(x
n
, x) . The convergence of
component sequences is obviously equivalent to convergence wrt d
max
.
In particular, if x
n
x and y
n
y in R , then z
n
= (x
n
, y
n
) z = (x, y) in R
2
.
Theorem: The above results extend to the Cartesian product of n > 2 metric spaces.
21
Proof: By induction.
Theorem: R
m
is complete.
Proof: Let (x
n
) be a Cauchy sequence in R
m
wrt any of the above metrics. Then
each component sequence (x
k,n
) for k = 1, 2, 3, . . . , m is Cauchy in R (by same reasoning as
convergence equivalence theorem above), and since R is complete, all component sequences
converge, x
k,n
converges to some x
k
k, therefore x
n
(x
1
, x
2
, . . . , x
m
).
3.6 Connectedness
A metric space M is said to be disconnected if it can be written as a disjoint union of
two non-empty clopen sets. Note that it is sucient to discover one proper (i.e ,= or
M itself) clopen subset S of M, since its complement would also have to be clopen and
proper (hence non-empty), and wed have M = S . S
c
, where symbol . indicates disjoint
union. M is said to be connected if it is not disconnected. S M is said to be connected
if its connected when considered as metric space in its own right (with metric inherited
from M), and is disconnected otherwise. For example, interval (2, 5) is a connected subset
of R, but (2, 5)(5, 8) is not. Q is also disconnected, since Q = (Q(, )) . (Q(, )).
Theorem: Suppose M is connected, and f : M N is a continuous function onto N.
Then N is connected.
22
Proof: We note here rst that f : M N is onto if y N x M s.t. y = f(x),
i.e. every point in N is image under f of some point in M (there could be many points
in M all mapping to same point in N). Now, suppose N is not connected, and let S N
be a proper clopen subset, then f
1
(S) is a non-empty (since f is onto), clopen (since f
is continuous) subset of M and so is f
1
(S
c
), resulting in M = (f
1
(S)) . (f
1
(S
c
)) is a
disjoint union of proper clopen subsets, contradicting the connectedness of M. We conclude
that N is connected.
Thus a continuous image of a connected set is connected. A discontinuous image of a
connected set need not be connected, however. For example if f : R R sends all negative
numbers to 0 and all non-negative numbers to 1, then the image is 0, 1 a nite set, which
is clearly disconnected. Of course this example only works if R was connected to begin with.
Theorem: R is connected.
Proof: Suppose R is disconnected. Then S R s.t. S is proper and clopen. Since S is
opein in R we know that S is a countable disjoint union of open intervals. Let (a, b) be one
of these intervals (we know that S is non-empty since its proper, so (a, b) exists). We have
seen in the proof of the theorem about every open set U in R being the countable disjoint
union of open intervals that the endpoints of hte intervals do not belong to U, thus a, b / S.
Suppose b < , then b is clearly a limit point of of S and b / S contradicts S being closed.
Therefore b = , and similarly a = . But then S = R and S is not proper and we arrive
at contradiction. Thus, R is connected.
23
Theorem: Open and closed intervals in R are connected.
Proof: Let (a, b) R. We know that f : R (/2, /2) given by f(x) = tan
1
(x) is
continuous (you can verify that for yourself). You are also invited to nd continuous func-
tion g : (/2, /2) (a, b) (which is easy). Then g f is continuous as a composition of
continuous functions and therefore (a, b) is connected as a continuous image of the connected
set R.
Now, let [a, b] R. Then dene f : R [a, b] as follows:
f(x) =

a if x < a
x if a x b
b if x > b
Clearly f is continuous and therefore [a, b] is connected.
Theorem: Suppose that S

is a collection of connected sets and that S

,= . Then
S = S

is connected.
Proof: Let x S

. Now suppose that S is disconnected, S = A . A

c
, where A and
A
c
are disjoint non-empty clopen subsets of S. Now, x S

= x S

for some =
x S = S

= x A or x A
c
. Without loss of generality, assume x A. Since each S

is a subset of S, S

inherits its closed and open sets from S by the inheritance theorem,
and since x S

we have A S

is a non-empty clopen subset of S

. Since each S

24
is connected, we conclude that A S

= S

and therefore A = (A S

) = S

= S,
contradicting A being a proper subset of S. We conclude that S is connected.
Theorem: Let S M be connected. Then S T

S = T is connected. In particu-
lar, the closure of a connected set is connected.
Proof: Suppose T is disconnected, T = A . A
c
, where A and A
c
are proper clopen
subsets of T. By intheritance principle, AS is a clopen subset of S. Since S is connected,
B = A S cannot be proper, therefore either B = S or B = . Without loss of generality,
suppose B = (if B = S, just look at C = A
c
S, the complement of B in S, which leads
to C = ). Since AS = , we have S A
c
in T. But A ,= and A T

S = x A
s.t. x is a limit point of S. Since A is open, r > 0, s.t. B
r
(x) A. But x is a limit point
of S and therefore B
r
(x) S ,= = A A
c
,= , and we arrive at a contradiction. We
conclude that T is connected.
Now we introduce the intermediate value property. Let f : M R be a func-
tion. Then f is said to have intermediate value property if x, y M it is true that if
f(x) = a < b = f(y) then c (a, b) z M s.t. f(z) = c. In other words, if function
assumes two distinct values in R then it also has to assume all the values in-between to have
the intermediate value property.
Theorem: Let M be connected, and f : M R be continuous. Then f has the in-
termediate value property. In particular, every continuous function f : R R has the
25
intermediate value property.
Proof: Suppose not. Then x, y M s.t. f(x) = a < b = f(y) and c (a, b)
s.t z M f(z) ,= c. Let A = (, c), and A

= (c, ). Then M = B . B
c
, where
B = f
1
(A) and B
c
= f
1
(A

) is the complement of B in M. Note that x B and y B

c
,
so B ,= ,= B
c
. Since f is continuous, the preimage of an open set is open and since both
A and A

are open subsets of R, we have that B and B

c
are both open, and therefore also
both closed as complements of each other. We conclude that M is a disjoint union of non-
empty clopen sets and is therefore disconnected, which contradicts our assumption. Thus,
we conclude that f has the intermediate value property.
3.7 Compactness
We say that S M is sequentially compact if every sequence (x
n
) in S, has a conver-
gent subsequence (x
n
k
) s.t. x
n
k
x for some x S. For example, every nite subset
S = x
1
, x
2
, . . . , x
n
of a metric space M is compact, since any sequence in S has to repeat
at least one x
k
innitely many times, and that results in existence of at least one convergent
subsequence x
k
, x
k
, x
k
, . . . Notice also that the convergent subsequence or indeed the limit
need not be unique, as the example of a sequence 1,2,1,2,1,2,1,2, . . . in S = 1, 2 N shows.
Also, N is not a compact subset of R since the sequence 1,2,3,4,5,. . . does not converge to
a point in N. Other non-compact subsets of R include (0, 1] (sequence 1, 1/2, 1/3, . . . does
not have a subsequence that would converge to a point in (0, 1]) and Q (every subsequence
of the sequence 3, 3.1, 3.14, 3.141, . . . converges to / Q).
26
We say that a collection U

of open subsets of M is an (open) cover for S M, if

x S s.t x U

. We say that a collection V

is a subcover of U

if V

= U

for some , i.e. V

. We then also say that U

reduces to V

. We say
that S M is covering compact if every open cover U

reduces to nite subcover. In

other words, if were given any open cover for S whatsoever, and we can always throw away
enough members of it, so that were left with only nitely many and they still cover all of
S then S is covering compact. Note that every subset S of M has at least one open cover,
namely M, since M is open and denitely covers S (this particular open cover is already
nite). Also, in order to see that not all subsets are covering compact, consider the cover of
(0, 1] R by open intervals U
n
= (1/n, 1 + 1/n). Clearly, U
n
: n N is an open cover for
(0, 1], but you can easily see that it cant be reduced to a nite subcover, since in that case
wed be left with U
n
1
, U
n
2
, . . . , U
n
k
and letting N = maxn
1
, n
2
, . . . , n
k
, we observe that
x (0, 1/N) x is not contained in any of U
n
1
, U
n
2
, . . . , U
n
k
.
Now, let S M and let U

be some open cover for S. Then if > 0 s.t. x S

B

(x) U

for some we say that is a Lebesgue number for U

. In other words,
a Lebesgue number for an open cover of S is some small radius, s.t. neighborhood of every
point in S of that radius is contained in some member of the cover (obviously, which member
it is, depends on the particular point). Notice that a given cover for a given set might not
have a Lebesgue number. For example, (0, 1) R has as one possible cover (0, 1), i.e.
its covered by itself. Suppose it had Lebesgue number , then pick x (0, ). Since
B

(x) = (x , x + ) contains negative points, we have B

(x) (0, 1), and thus this

27
neighborhood is not contained in any member of the cover, leading to contradiction.
Theorem Every open covering of a sequentially compact S M has a Lebesgue number.
Proof: Suppose not. Let U

be an open cover for S s.t.

n
x
n
S s.t. B
n
(x
n
) U

. Let
n
= 1/n, and x
n
be as above. Then since S is sequentially compact, (x
n
) has some
convergent subsequence (x
n
k
) x S. Since U

is a cover for S, x U

for some .
Since U

is open, r > 0 s.t. B

r
(x) U

. Now, since x
n
k
x, N
1
s.t. n
k
N
1
=
d(x, x
n
k
) < r/2. Moreover, N
2
N
1
s.t n
k
N
2
=
n
k
< r/2. Now pick some x
n
N
s.t.
n
N
> N
2
. Let y B
n
N
(x
n
N
), then:
d(x, y) d(x, x
n
N
) + d(x
n
N
, y) < r/2 +
n
N
< r/2 + r/2 = r
We conclude that B
n
N
(x
n
N
) B
r
(x) U

, contrary to our assumption that n B

n
(x
n
)
U

. Thus, every open covering of a sequentically compact set does indeed have a Lebesgue
number.
Theorem: S M is sequentially compact i it is covering compact.
Proof: Suppose S is covering compact, but not sequentially compact. Then let (x
n
) be
a sequence in S s.t. no subsequence of (x
n
) converges to a point in S. That implies that
x S r
x
> 0 s.t. B
rx
(x) contains only nitely many terms of (x
n
) (otherwise thered
be a subsequence converging to x). Now, B
rx
(x) is an open cover for S and since S is
covering compact, it reduces to a nite subcover B
rx
1
(x
1
), B
rx
2
(x
2
), . . . , B
rx
k
(x
k
) and since
28
each member of this subcover contains only nitely many terms of (x
n
), we conclude that S
contains nitely many terms of (x
n
), an obvious contradiction. Thus covering compactness
implies sequential compactness.
Now suppose S is sequentially compact, and let U

be some open cover for S. We know that

U

has some Lebesgue number > 0. Pick x

1
S and some U
1
U

s.t. B

(x
1
) U
1
.
If S U
1
then we have succeded in reducing U

to a nite subcover. If not, then pick

uncovered point x
2
S (x S U
1
c
) and U
2
s.t. B

(x
2
) U
2
. If S U
1
U
2
, were done,
if not we continue picking uncovered points to obtain a sequence of points (x
n
) in S and a
sequence (U
n
) of members of U

s.t. B

(x
n
) U
n
and x
n+1
AU
1
c
U
2
c
. . . U
n
c
. If
at some point the sequences terminate (if for some N, A = (U
1
U
2
. . . U
N
)) then we have
reduced U

to a nite subcover. Now suppose, the sequences never terminate. Then since
S is sequentially compact, there is some subsequence (x
n
k
) of (x
n
) s.t. x
n
k
x S. There-
fore, N s.t n
k
N = d(x
n
k
, x) < . In particular, d(x
N
, x) < = x B

(x
N
) U
N
.
Since U
N
is open, r > 0, s.t. B
r
(x) U
N
. But n
k
> N = x
n
k
/ U
N
, and therefore
B
r
(x) contains only nitely many terms of (x
n
k
), a clear contradiction to convergence. We
conclude that in fact, U

reduces to a nite subcover, and sequential compactness implies

covering compactness.
S M is said to be compact if it is sequentially/covering compact. In the proofs of the
theorems about compactness that follow, we will use whichever denition of compactness
leads to a more direct proof and we will supply two proofs for some of the theorems to give
you better understanding of both kinds of compactness.
29
Theorem: Let S
1
M
1
and S
2
M
2
be compact. Then S = S
1
S
2
M
1
M
2
= M
is compact.
Proof: Let (x
n
, y
n
) be a sequence in S (we avoid here cumbersome notation ((x
n
, y
n
))).
Then (x
n
) is a sequence in S
1
and therefore has some subsequence (x
n
k
), that converges in S
1
.
Now (y
n
k
) is then a sequence in S
2
and therefore has some subsequence (y
n
k
l
) that converges
in S
2
. Moreover, since (x
n
k
) was a convergent sequence in S
1
, the subsequence (x
n
k
l
) is also
a convergent sequence in S
1
. Therefore, the sequence (x
n
, y
n
) has a convergent subsequence
(x
n
k
l
, y
n
k
l
) with respect to all product space metrics discussed above (since both component
sequences converge), and we conclude that Cartesian product of two compact sets is compact.
Theorem: The Cartesian product of n compact sets is compact.
Proof: By induction and above theorem.
Theorem: Closed interval [a, b] R is compact.
Proof 1: Let (x
n
) be a sequence in [a, b]. Consider set C = x [a, b] : x
n
< x only
nitely often . Clearly, a C, and b is an upper bound for C, therefore C has a least
upper bound x

. We will show that there is some subsequence (x

n
k
) of (x
n
) converging to
x

. Suppose x

= b, then N s.t n N = x
n
= b, and clearly we have a subsequence
converging to b = x

. Now, suppose x

< b and no subsequence converges to x

, then
b x

> r > 0 s.t. x

n
B
r
(x

) = (x

r, x

+ r) only nitely often. But then x

+ r C
30
(note that x

+r [a, b]), contradiction to x

being the l.u.b for C. We conclude that in fact

there is some subsequence converging to x

and therefore [a, b] is a compact subset of R.

Proof 2: Let U

be some open cover for [a, b] and consider set C = x [a, b]: nitely
many U

would suce to cover the interval [a, x]. Clearly a C and b is an upper bound,
therefore C has the l.u.b. x

. Suppose x

< b, and let U

1
, U

2
, . . . , U
n
be those nitely
many members of U

that suce to cover [a, x

]. Then x

k
for some k (not neces-
sarily unique), and since U

k
is open, r > 0 s.t. B
r
(x

) = (x

r, x

+ r) U

k
. Now
y (x

, x

+r), y is covered by same U

1
, . . . , U
n
as x

, and therefore picking some specic

y we have that nitely many members of U
alpha
suce to cover [a, y], which contradicts x

being the l.u.b. for C. We conclude that x

= b and therefore nitely many U

cover [a, b],

and thus [a, b] is compact.
Theorem: Closed box [a
1
, b
1
] [a
2
, b
2
] . . . [a
n
, b
n
] R
n
is compact.
Proof: It is a nite Cartesian product of compact sets.
Theorem (Bolzano-Weierstrass): Any bounded sequence in R
n
has a convergent sub-
sequence.
Proof: Any bounded sequence in R
n
is contained in some closed box, and therefore has
a subsequence that converges to some point in that box.
31
Theorem: Suppose M is compact and S M is closed. Then S is compact.
Proof 1: Let (x
n
) be a sequence in S, then (x
n
) is also a sequence in M and therefore
has a convergent subsequence x
n
k
x M. Since S is closed in M, we have x S, and
therefore (x
n
) has a subsequence that converges in S. We conclude that S is compact.
Proof 2: Let U

be some open cover for S, then since S

c
is open, U

S
c
is an
open cover for M and since M is compact, it reduces to a nite subcover for M and therefore
it is a nite cover for S. Now there are two possibilities. First, S
c
might not be a member
of this nite subcover, then we have reduced U

to a nite subcover for S. Now, suppose

S
c
is a member of the nite subcover we found ourselves with, then it is obvious that the
rest of the nite subcover still covers S (since no element of S is contained in S
c
) and by
excluding S
c
we are left with the nite subcover for S to which the original cover U

is
now reduced. We conclude that S is compact.
Theorem: Compact set S M is closed and bounded.
Proof: Suppose S is not closed, then (x
n
), a sequence in S s.t. x
n
x / S. Since S is
compact, some subsequence x
n
k
x

S, but since (x
n
) is a convergent sequence in M,
all of its subsequences converge to the same limit in M, and we have that x
n
k
x, and by
uniqueness of limits we have x = x

S, a contradiction to x not being in S. So S is closed.

Now let x M and suppose S is not bounded. Then n N x
n
S s.t. d(x, x
n
) > n.
Since S is compact, some subsequence (x
n
k
) of a sequence (x
n
) in S, s.t. x
n
k
x

S.
Since all convergent sequences are bounded, for some r > 0 and for some y M (for example
32
x

), we have x
n
k
B
r
(y) n
k
. Now, let r

= d(x, y). We observe that by the property 3 of

the metric, we have d(x, x
n
k
) d(x, y) + d(y, x
n
k
) r

+ r, contradictory to the fact that

d(x, x
n
k
) as n
k
. We conclude that S is in fact bounded.
The converse to that statement (every closed and bounded subset of a metric space is
compact) is not true in general. For example, let M = N with discrete metric. Then M is
a closed and bounded subset of itself (x B
2
(1)x M), but the sequence 1,2,3,4,. . . in M
has no convergent subsequence, and therefore M is not compact. The converse is, however,
true in R
n
and a more general converse will be provided again later.
Theorem (Heine-Borel): Every closed and bounded S R
n
is compact.
Proof: Since S is bounded, it is contained in some closed box in R
n
, and is therefore a
closed subset of a compact set. We conclude that S is compact.
We will now proceed to construct a more general converse statement. We start by intro-
ducing a new notion: S M is said to be totally bounded if > 0 there exists a nite
covering of S by -neighborhoods. Notice that this denition is dierent from the denition
of (non-total) boundedness given earlier.
Theorem: Let M be a complete metric space. Then S M is compact i S is closed
and totally bounded.
Proof: Suppose S M is compact. We have already shown that S has to be closed.
33
Now let > 0 be given, and consider the following open cover for S: B

(x) : x S. Since
S is compact, the cover reduces to a nite subcover and we have a nite covering of S by
-neighborhoods, and we conclude that S is totally bounded.
Now suppose S M is closed and totally bounded. Let (x
n
) be a sequence in S. Let
n
= 1/n n. Since S is totally bounded, we have a nite covering for S by
1
-neighborhoods
B

1
(y
1,1
), B

1
(y
1,2
), . . . , B

1
(y
1,m
1
) for some y
1,1
, y
1,2
, . . . , y
1,m
1
M. Then at least one of
these neighborhoods contains innitely many terms of the sequence (x
n
), suppose B

1
(y
1,k
1
),
and let N
1
be s.t. x
N
1
B

1
(y
1,k
1
). Now, since every subset of a totally bounded set
is totally bounded (you can easily show it), we have a nite covering of B

1
(y
1,k
1
) by
2
-
neighborhoods B

2
(y
2,1
), B

2
(y
2,2
), . . . , B

2
(y
2,m
2
) and once again one of these neighborhoods
contains innitely many terms of the sequence (x
n
), suppose B

2
(y
2,k
2
), and let N
2
> N
1
be
s.t. x
N
2
B

2
(y
2,k
2
). Continuing in this manner we obtain a subsequence (x
N
k
) which is
Cauchy, since n, m > N

= d(x
Nn
, x
Nn
) <
N
= 1/N

. Since M is complete, x
N
k
x
M. Since S is closed in M, x S. Thus (x
n
) has a subsequence that converges to a limit
in S and we conclude that S is compact.
We note here that the conditions specied are indeed necessary. Take completeness, for
example. Consider S = Q [, ], a subset of a metric space Q with the usual distance
metric. Then S is closed and totally bounded subset of Q but its clearly not compact since
sequence 3, 3.1, 3.14, 3.15, . . . in S doesnt have a subsequence that converges to a point in
S or even in Q for that matter. Also, substituting boundedness for total boundedness in the
theorem above would not suce, since we have already seen that a complete metric space N
with discrete metric is a closed and bounded non-compact subset of itself.
34
Below are some important results on the behavior of continuous functions on compact
sets.
Theorem: The continuous image of a compact set is compact.
Proof 1: Let f : M N be continuous, let S M be compact and let f(S) N denote
the image of S under f. Let (y
n
) be a sequence in f(S), then n x
n
S s.t. f(x
n
) = y
n
.
Since S is compact, the sequence (x
n
) in S has some convergent subsequence x
n
k
x S.
By continuity of f, we then have that y
n
k
= f(x
n
k
) f(x) f(S), and therefore (y
n
) has
a subsequence that converges to the limit in f(S), and we conclude that f(S) is compact.
Proof 2: Let f : M N be continuous, let S M be compact and let f(S) N denote
the image of S under f. Let U

be an open cover for f(S) in N. Then since f is continuous,

f
1
(U

) is an open set in M , and therefore f

1
(U

) is an open cover for S in M. Since

S is compact, f
1
(U

) reduces to a nite subscover f

1
(U
1
), f
1
(U
2
), . . . , f
1
(U
n
), and
clearly U
1
, U
2
, . . . , U
n
is an open cover for f(S) (you can easily see it yourself), which is a
nite subcover of the original cover. We conclude that f(S) is compact.
Theorem: Continuous real-valued funciton dened on a compact set assumes its maxi-
mum and minimum.
Proof: Let f : M R be continuous and let S M be compact. Then f(S) R is
closed and bounded, and we have seen that a closed and bounded subset of R contains its
35
g.l.b and its l.u.b.
Theorem: Every continous function dened on a compact set is uniformly continuous.
Proof: Let M be compact and f : M N continuous. Suppose f is not uniformly con-
tinuous. Then > 0 s.t.
n
> 0 x
n
, y
n
M s.t. d(x
n
, y
n
) <
n
but d(f(x
n
), f(y
n
)) > .
Let
n
= 1/n and for each n, let x
n
, y
n
M be as above. Since M is compact, the se-
quence (x
n
) in M has some convergent subsequence (x
n
k
). Moreover, the sequence (y
n
k
)
also has some convergent subsequence y
n
k
l
y M. Since (x
n
k
l
) is a subsequence of a
convergernt sequence, it also converges and it converges to the same limit as the mother
sequence, and since d(x
n
k
l
, y
n
k
l
) < 1/n
k
l
, we can easily see that it also converges to y. The
continuity of f implies that f(x
n
k
l
) y and f(y
n
k
l
) y. Now, let N be s.t. n
k
l
N =
d(f(x
n
k
l
), f(y)) < /2 and d(f(y
n
k
l
), f(y)) < /2, then we have n
k
l
N =
d(f(x
n
k
l
), f(y
n
k
l
)) d(f(x
n
k
l
), f(y)) + d(f(y), f(y
n
k
l
) < /2 + /2 =
which contradicts our assumption that d(f(x
n
), f(y
n
)) > n. We then conclude that f
indeed is uniformly continuous.
3.8 More Results on R
We introduce some notation rst. Given sequence (x
n
) in R we say that lim(x
n
) = x R
if x
n
x in R. We say that lim(x
n
) = if M > 0 N s.t. n N = x
n
> M. And
similarly, lim(x
n
) = if M < 0 N s.t. n N = x
n
< M. Here are some examples:
36
1. (x
n
) = 3, 3.1, 3.14, 3.141, . . . Then lim(x
n
) =
2. (x
n
) = 2, 3, 5, 7, 11, . . . Then lim(x
n
) =
3. (x
n
) = -1, -4, -9, -16, . . . Then lim(x
n
) =
4. (x
n
) = -1, 1, -2, 2, -3, 3, . . . Then lim(x
n
) does not exist.
Given a sequence (x
n
) in R, we say that M =lim sup(x
n
) if M = lim
n
(sup
kn
x
k
). We dene
m = lim inf(x
n
) similarly. Here are some examples:
1. (x
n
) = 1, 2, 3, 4, 5, . . ., then lim sup(x
n
) = lim inf (x
n
) =
2. (x
n
) = 1, 1, 2, 1, 2, 3, . . ., then lim sup(x
n
) = , lim inf(x
n
= 1
3. (x
n
) = 3, 3.1, 3.14, 3.141, . . ., then lim sup(x
n
) = lim inf(x
n
)=
We say that sequence (x
n
) is monotone increasing if n > m = x
n
> x
m
. The
sequence is monotone non-decreasing if n > m = x
n
x
m
. The monotone decreas-
ing and monotone non-increasing sequences are dened similarly. A sequence is said to be
monotone if its monotone non-increasing or monotone non-decreasing (note that every
monotone increasing sequence is also monotone non-decreasing, and every monotone de-
creasing sequences is monotone non-increasing, so these two classes are also covered by the
denition).
1. (x
n
) = 3, 3.1, 3.14, 3.141, . . . is monotone increasing
2. (x
n
) = -1, -4, -9, -16, . . . is monotone decreasing
3. (x
n
) = 1, 2, 2, 3, 3, 3, 4, 4, 4, 4, . . . is monotone non-decreasing
37
4. (x
n
) = -1, -1, -1, -1, . . . is monotone non-decreasing and monotone non-increasing
5. (x
n
) = 1, 1, 2, 1, 2, 3, 1, 2, 3, 4, . . . is not monotone but it has many possible monotone
subsequences
The following are some basic results about sequences in R :
1. Let x
n
x R, and let k R. Then y
n
= kx
n
kx.
2. Let x
n
x and y
n
y in R. Then z
n
= x
n
+ y
n
x + y.
3. Let x
n
x and y
n
y in R. Then z
n
= x
n
y
n
xy.
4. Let x
n
x ,= 0 in R, s.t. x
n
,= 0 n. Then y
n
= 1/x
n
1/x.
5. Let x
n
x ,= 0 in R, s.t. x
n
,= 0 n, and let y
n
y in R. Then z
n
= y
n
/x
n
y/x
6. Let lim(x
n
) = , lim (y
n
) > 0. Then lim(z
n
= x
n
y
n
) =
7. Let (x
n
) be monotone and bounded. Then (x
n
) converges in R.
8. Let (x
n
) be any sequence in R. Then (x
n
) has a monotone subsequence. Together
this and the previous result imply that every bounded sequence in R has a convergent
subsequence.
9. Let (x
n
) be a monotone sequence. Then lim(x
n
) exists (it could of course take on
values of and )
10. Let f, g be real-valued funcitons M R, continuous at some x M. Then f +g, fg,
and f/g (asusming g(x) ,= 0) are all continuous at x.
38
11. Given a sequence (x
n
), lim sup(x
n
) and lim inf(x
n
) are well dened (either they belong
to R or take values of
12. x
n
x i lim sup(x
n
) = lim inf(x
n
) = x.
The proofs of these results are left as exersises.
3.9 Calculus and Function Convergence
. We say that a real-valued funciton f is dierentiable at x, if one of the following equivalent
conditions holds:
1. lim
tx
f(t) f(x)
t x
= L exists
2. lim
x0
f(x + x) f(x)
x
= L exists
3. > 0 > 0 s.t. [t x[ < = [
f(t)f(x)
tx
L[ <
We call L the derivative of f at x and write f

(x) = L. We say that f is dierentiable on

the interval (a, b) if it is dierentialbe at every x (a, b). The following are familiar results
from calculus:
1. If f is dierentiable at x, then f is continuous at x.
2. If f and g are dierentiable at x, then so is f + g and (f + g)

(x) = f

(x) + g

(x)
3. If f and g are dierentiable at x, then so is (f g), and (f g)

(x) = f

(x)g(x)+f(x)g

(x)
4. If f(x) = c x, then f

(x) = 0 x
5. If f and g are dierentiable at x and g(x) ,= 0, then f/g is dierentiable at x and
(f/g)

(x) =
f

(x)g(x)f(x)g

(x)
g(x)
2
39
6. Chain Rule: If f is dierentiable at x, and g is dierentiable at f(x), then g f is
dierentiable at x and (g f)

(x) = g

(f(x))f

(x)
Theorem Let f be dierentiable on (a, b) and suppose it achieves a maximum or mini-
mum at some c (a, b). Then f

(c) = 0.
Proof: We will prove the theorem for maximum, and proof for minimum is analogous.
Let t approach c from above. Then
f(t)f(c)
tc
0 t. If we let t approach c from below,
then
f(t)f(c)
tc
0 t. Since both expressions have to tend to the same limit L = f

(c), we
conclude that f

(c) = 0.
Theorem (Mean Value): Suppose f is continuous on [a, b] and dierentiable on (a, b).
Then c (a, b) s.t. f(b) f(a) = f

(c)(b a). In particular, if [f

(x)[ M x (a, b),

then t, x (a, b) we have [f(t) f(x)[ M[t x[.
Proof: Let S =
f(b)f(a)
ba
and let g(x) = f(x) Sx. Clearly g is dierentiable on (a, b)
and continuous on [a, b] (why?). Moreover, g(a) = g(b). Since g is continuous on a compact
set, it achieves maximum and minimum and since g(a) = g(b) it achieves at least one of
them at some c (a, b). Then 0 = g

(c) = f

(c) S = f(b) f(a) = f

(c)(b a).
We state the following result without proof (which can be found, for example in Pugh,
p.145): If f is dierentiable on (a, b), then f

has the intermediate value property. Also, all

familiar results from calculus (derivative and integral formulas, LHopitals Rule, results on
series convergence, etc...) hold.
40
We say that a sequence of functions f
n
: [a, b] R converges pointwise to f (f
n
f),
if x [a, b] f
n
(x) f(x) as n . We say that a sequence f
n
: [a, b] R converges
uniformly to f (f
n
f) if > 0, N s.t. n N = [f
n
(x) f(x)[ < x [a, b]. You
should convince yourself that the sequence of functions f
n
(x) = x
n
on open interval [0, 1]
converges pointwise to the function
f(x) =

0 if 0 x < 1
1 if x = 1
but does not converge uniformly.
Theorem: If f
n
f and each f
n
is continuous at x, then f is continuous at x.
Proof: Let > 0 be given, and let N be s.t. n N = [f
n
(y) f(y)[ < /3 y. Since
f
N
is continuous at x, > 0 s.t. [x y[ < = [f
N
(x) f
N
(y)[ < /3. Then [x y[ <
= [f(x) f(y)[ [f(x) f
N
(x)[ +[f
N
(x) f
N
(y)[ +[f
N
(y) f(y)[ < /3 +/3 +/3 = ,
and we conclude that f is continuous at x.
Note that our example above shows that pointwise convergence of continuous functions
is not sucient to ensure continuity of the limit.
We state two more results without proving them. The proofs can be found in Pugh.
1. Suppose f
n
f, then

b
a
f
n
(x)dx

b
a
f(x)dx as n .
41
2. Suppose f
n
f and f

n
g, then g = f

.
4 References
1. Pugh, C.C., Real Mathematical Analysis, Springer, 2002
2. Rice, John A., Mathematical Statistics and Data Analysis, 2nd ed., Duxbury Press,
1995
3. Ross, Kenneth A., Elementary Analysis: The Theory of Calculus, Springer, 2000
42