Mathematical Toolkit Autumn 2021

Lecture 6: October 14, 2021

Lecturer: Madhur Tulsiani

1 Rayleigh quotients: eigenvalues as optimization

Definition 1.1 Let ϕ : V → V be a self-adjoint linear operator and v ∈ V \ {0V }. The Rayleigh
quotient of ϕ at v is defined as
hv, ϕ(v)i
R ϕ (v) := .
k v k2

Proposition 1.2 Let dim(V ) = n and let ϕ : V → V be a self-adjoint operator with eigenvalues
λ1 ≥ λ2 ≥ · · · ≥ λn . Then,

λ1 = max R ϕ (v) and λn = min R ϕ (v)

v∈V \{0V } v∈V \{0V }

Using the above, Rayleigh quotients can be used to prove the spectral theorem for Hilbert
spaces, by showing that the above maximum1 is attained at a point in the space, and de-
fines an eigenvalue if the operator ϕ is “compact”. A proof can be found in these notes by
Paul Garrett [?].

Proposition 1.3 (Courant-Fischer theorem) Let dim(V ) = n and let ϕ : V → V be a self-

adjoint operator with eigenvalues λ1 ≥ λ2 ≥ · · · ≥ λn . Then,

λk = max min R ϕ (v) = min max R ϕ (v) .

S ⊆V v∈S\{0V } S ⊆V v∈S\{0V }
dim(S)=k dim(S)=n−k+1

Definition 1.4 Let ϕ : V → V be a self-adjoint operator. Φ is said to be positive semidefinite if

R ϕ (v) ≥ 0 for all v 6= 0. Φ is said to be positive definite if R ϕ (v) > 0 for all v 6= 0.
1 Strictly speaking, we should write sup and inf instead of max and min until we can justify that max and

min are well defined. The difference is that sup and inf are defined as limits while max and min are defined
as actual maximum and minimum values in a space, and these may not always exist while we are at looking
infinitely many values. Thus, while supx∈(0,1) x = 1, the quantity maxx∈(0,1) x does not exist. However, in
the cases we consider, the max and min will always exist (since our spaces are closed under limits) and we
will use max and min in the class to simplify things.

1
Proposition 1.5 Let ϕ : V → V be a self-adjoint linear operator. Then the following are equiva-
lent:

1. R ϕ (v) ≥ 0 for all v 6= 0.

2. All eigenvalues of ϕ are non-negative.

3. There exists α : V → V such that ϕ = α∗ α.

The decomposition of a positive semidefinite operator in the form ϕ = α∗ α is known as

the Cholesky decomposition of the operator. Note that if we can write ϕ as α∗ α for any
α : V → W, then this in fact also shows that ϕ is self-adjoint and positive semidefinite.

2 Singular Value Decomposition

Let V, W be finite-dimensional inner product spaces and let ϕ : V → W be a linear

transformation. Since the domain and range of ϕ are different, we cannot analyze it in
terms of eigenvectors. However, we can use the spectral theorem to analyze the operators
ϕ∗ ϕ : V → V and ϕϕ∗ : W → W and use their eigenvectors to derive a nice decomposition
of ϕ. This is known as the singular value decomposition (SVD) of ϕ.

Proposition 2.1 Let ϕ : V → W be a linear transformation. Then ϕ∗ ϕ : V → V and ϕϕ∗ :

W → W are self-adjoint positive semidefinite linear operators with the same non-zero eigenvalues.

Proof: The self-adjointness and positive semidefiniteness of the operators ϕϕ∗ and ϕ∗ ϕ
follows from the exercise characterizing positive semidefinite operators in the previous
lecture. Specifically, we can see that for any w1 , w2 ∈ W,

hw1 , ϕϕ∗ (w2 )i = hw1 , ϕ( ϕ∗ (w2 ))i = h ϕ∗ (w1 ), ϕ∗ (w2 )i = h ϕϕ∗ (w1 ), w2 i .

This gives that ϕϕ∗ is self-adjoint. Similarly, we get that for any w ∈ W

hw, ϕϕ∗ (w)i = hw, ϕ( ϕ∗ (w))i = h ϕ∗ (w), ϕ∗ (w)i ≥ 0 .

This implies that the Rayleigh quotient R ϕϕ∗ is non-negative for any w 6= 0 which implies
that ϕϕ∗ is positive semidefinite. The proof for ϕ∗ ϕ is identical (using the fact that ( ϕ∗ )∗ =
ϕ).
Let λ 6= 0 be an eigenvalue of ϕ∗ ϕ. Then there exists v 6= 0 such that ϕ∗ ϕ(v) = λ · v.
Applying ϕ on both sides, we get ϕϕ∗ ( ϕ(v)) = λ · ϕ(v). However, note that if λ 6= 0 then
ϕ(v) cannot be zero (why?) Thus ϕ(v) is an eigenvector of ϕϕ∗ with the same eigenvalue
λ.

2
We can notice the following from the proof of the above proposition.

Proposition 2.2 Let v be an eigenvector of ϕ∗ ϕ with eigenvalue λ 6= 0. Then ϕ(v) is an eigen-

vector of ϕϕ∗ with eigenvalue λ. Similarly, if w is an eigenvector of ϕϕ∗ with eigenvalue λ 6= 0,
then ϕ∗ (w) is an eigenvector of ϕ∗ ϕ with eigenvalue λ.

We can also conclude the following.

Proposition 2.3 Let the subspaces Vλ and Wλ be defined as

Vλ := {v ∈ V | ϕ∗ ϕ(v) = λ · v} and Wλ := {w ∈ W | ϕϕ∗ (w) = λ · w} .

Then for any λ 6= 0, dim(Vλ ) = dim(Wλ ).

Using the above properties, we can prove the following useful proposition, which extends
the concept of eigenvectors to cases when we have ϕ : V → W and it might not be possible
to define eigenvectors since V 6= W (also ϕ may not be self-adjoint so we may not get
orthonormal eigenvectors).

Proposition 2.4 Let σ12 ≥ σ22 ≥ · · · ≥ σr2 > 0 be the non-zero eigenvalues of ϕ∗ ϕ, and let
v1 , . . . , vr be a corresponding orthonormal eigenvectors (since ϕ∗ ϕ is self-adjoint, these are a subset
of some orthonormal eigenbasis). For w1 , . . . , wr defined as wi = ϕ(vi )/σi , we have that

1. {w1 , . . . , wr } form an orthonormal set.

2. For all i ∈ [r ]
ϕ(vi ) = σi · wi and ϕ∗ (wi ) = σi · vi .

Proof: For any i, j ∈ [r ], i 6= j, we note that

ϕ ( vi ) ϕ ( v j )
 

 1
 1
∗ 
wi , w j = , = · ϕ ( v i ), ϕ ( v j ) = · ϕ ϕ ( v i ), v j
σi σj σi σj σi σj
σ

= i · vi , v j = 0 .

σj

Thus, the vectors {w1 , . . . , wr } form an orthonormal set. We also get ϕ(vi ) = σi · wi from
the definition of wi . For proving ϕ∗ (wi ) = vi , we note that

σ2
 
∗ ∗ ϕ ( vi 1
ϕ ( wi ) = ϕ = · ϕ∗ ϕ(vi ) = i · vi = σi · vi ,
σi σi σi

which completes the proof.

3
The values σ1 , . . . , σr are known as the (non-zero) singular values of ϕ. For each i ∈ [r ], the
vector vi is known as the right singular vector and wi is known as the left singular vector
corresponding to the singular value σi .

Proposition 2.5 Let r be the number of non-zero eigenvalues of ϕ∗ ϕ. Then,

rank( ϕ) = dim(im( ϕ)) = r .

Using the above, we can write ϕ in a particularly convenient form. We first need the
following definition.

Definition 2.6 Let V, W be inner product spaces and let v ∈ V, w ∈ W be any two vectors. The
outer product of w with v, denoted as |wi hv|, is a linear transformation from V to W such that

|wi hv| (u) := hv, ui · w .

Note that if kvk = 1, then |wi hv| (v) = w and |wi hv| (u) = 0 for all u ⊥ v.

Exercise 2.7 Show that for any v ∈ V and w ∈ W, we have

rank (|wi hv|) = dim (im (|wi hv|)) = 1 .

We can then write ϕ : V → W in terms of outer products of its singular vectors.

Proposition 2.8 Let V, W be finite dimensional inner product spaces and let ϕ : V → W be a
linear transformation with non-zero singular values σ1 , . . . , σr , right singular vectors v1 , . . . , vr
and left singular vectors w1 , . . . , wr . Then,
r
ϕ = ∑ σi · |wi i hvi | .
i =1

Exercise 2.9 If ϕ : V → V is a self-adjoint operator with dim(V ) = n, then the real spectral
theorem proves the existence of an orthonormal basis of eigenvectors, say {v1 , . . . , vn } with corre-
sponding eigenvalues λ1 , . . . , λn . Check that in this case, we can write ϕ as
n
ϕ = ∑ λi · | vi i h vi | .
i =1

Note that while the above decomposition has possibly negative coefficients (the λi s), the singular
value decomposition only has positive coefficients (the σi s). Why is this the case?