Lecture 2.

Orthogonal Vectors and Matrices

Since the 1960s, many of the best algorithms of numerical linear algebra have
been based in one way or another on orthogonality. In this lecture we present
the ingredients: orthogonal vectors and orthogonal (unitary) matrices.

Adjoint
The complex conjugate of a scalar z , written z or z  , is obtained by negating
its imaginary part. For real z , z = z .
The hermitian conjugate or adjoint of an m  n matrix A, written A, is
the n  m matrix whose i; j entry is the complex conjugate of the j; i entry of
A. For example,
2 3 " #
a11 a12 a a a
6 7
A = 4 a21 a22 5 =) A = a a a .  11 21 31
a31 a32 12 22 32

If A = A , A is hermitian . By denition, a hermitian matrix must be square.

For real A, the adjoint simply interchanges the rows and columns of A. In
this case, the adjoint is also known as the transpose , and is written AT . If a
real matrix is hermitian, that is, A = AT , then it is also said to be symmetric .
Most textbooks of numerical linear algebra assume that the matrices under
discussion are real and thus use principally T instead of . Since most of the
ideas to be dealt with are not intrinsically restricted to the reals, however, we
have followed the other course. Thus, for example, in this book a row vector
11
12 Part I Fundamentals
will usually be denoted by, say, a rather than aT . The reader who prefers to
imagine that all quantities are real and that  is a synonym for T will rarely
get into trouble.

Inner Product
The inner product of two column vectors x; y 2 Cm is the product of the
adjoint of x by y:
Xm
xy = xiyi . (2:1)
i=1
The Euclidean length of x may be written kxk (vector norms such as this are
discussed systematically in the next lecture), and can be dened as the square
root of the inner product of x with itself:
p X
m !1=2
kxk = x x = jxij .
2 (2:2)
i=1
The cosine of the angle  between x and y can also be expressed in terms of
the inner product: y
x
cos  = kxk kyk . (2:3)
At various points of this book, as here, we mention geometric interpretations
of algebraic formulas. For these geometric interpretations, the reader should
think of the vectors as real rather than complex, although usually the in-
terpretations can be carried over in one way or another to the complex case
too.
The inner product is bilinear , which means that it is linear in each vector
separately:
(x1 + x2) y = x1y + x2 y,
x(y1 + y2) = xy1 + x y2,
(x) (y) = xy.
We shall also frequently use the easily proved property that for any matrices
or vectors A and B of compatible dimensions,
(AB ) = B A. (2:4)
This is analogous to the equally important formula for products of invertible
square matrices,
(AB ) 1 = B 1A 1: (2:5)
The notation A  is a shorthand for (A ) 1 or (A 1) ; these two are equal, as
can be veried by applying (2.4) with B = A 1.
Lecture 2 Orthogonal Vectors and Matrices 13
Orthogonal Vectors
A pair of vectors x and y are said to be orthogonal if xy = 0. If x and y
are real, this means they lie at right angles to each other in IRm . Two sets of
vectors X and Y are orthogonal (also stated \X is orthogonal to Y ") if every
x 2 X is orthogonal to every y 2 Y .
A set of nonzero vectors S is orthogonal if its elements are pairwise orthog-
onal, i.e., if for x; y 2 S , x 6= y ) xy = 0. A set of vectors is orthonormal if
it is orthogonal and in addition every x 2 S has kxk = 1.
Theorem 2.1. The vectors in an orthogonal set S are linearly independent.
Proof. If the vectors in S are not independent, then some vk 2 S can be
expressed as a linear combination of other members v1 ; : : : ; vn 2 S ,
X
n
vk = c i vi .
i=1
i=k6

Since vk 6= 0, vk vk = kvk k2 > 0. Using the bilinearity of inner products and
the orthogonality of S , we calculate
X
n
vk vk = ci vk vi = 0,
i=1
i= k
6

which contradicts the assumption that the vectors in S are nonzero. 2

As a corollary of Theorem 2.1 it follows that if an orthogonal set S  Cm
contains m vectors, then it is a basis for Cm .

Components of a Vector
The most important idea to draw from the concepts of inner products and or-
thogonality is this: inner products can be used to decompose arbitrary vectors
into orthogonal components.
For example, suppose that fq1; q2; : : : ; qng is an orthonormal set, and let v
be an arbitrary vector. The quantity qjv is a scalar. Utilizing these scalars as
coordinates in an expansion, we nd that the vector
r = v (q1 v)q1 (q2 v)q2


(qn v)qn (2:6)
is orthogonal to fq1; q2 ;


; qn g. This can be veried by computing qir:
qi r = qi v (q1 v)(qi q1 )


(qn v)(qi qn).
This sum collapses, since qiqj = 0 for i 6= j :
qir = qiv (qi v)(qi qi ) = 0.
14 Part I Fundamentals
Thus we see that v can be decomposed into n + 1 orthogonal components:
Xn Xn
v = r + (qi v)qi = r + (qi qi )v. (2:7)
i=1 i=1
In this decomposition, r is the part of v orthogonal to the set of vectors
fq1; q2; : : : ; qng, or equivalently to the subspace spanned by this set of vectors,
and (qi v)qi is the part of v in the direction of qi .
If fqi g is a basis for Cm, then n must be equal to m and r must be the
zero vector, so v is completely decomposed into m orthogonal components in
the directions of the qi:
Xm Xm
v = (qi v)qi = (qi qi )v. (2:8)
i=1 i=1
In both (2.7) and (2.8) we have written the formula in two dierent ways,
once with (qi v)qi and again with (qi qi )v. These expressions are equal, but
they have dierent interpretations. In the rst case, we view v as a sum
of coecients qi v times vectors qi . In the second, we view v as a sum of
orthogonal projections of v onto the various directions qi . The ith projection
operation is achieved by the very special rank-one matrix qi qi. We shall discuss
this and other projection processes in Lecture 6.

Unitary Matrices
A square matrix Q 2 Cmm is unitary (in the real case we also say orthogonal )
if Q = Q 1, i.e, if QQ = I . In terms of the columns of Q, this product can
be written
2  32 3 2 3
66
q 1 77 66 77 66 1 77
66 q2 77 66 77 66 1 7
66 ... 77 66 q1 q2


qm 77 = 66 . . . 777 .
4 54 5 4 5
qm 1
In other words, qiqj = ij , and the columns of a unitary matrix Q form an
orthonormal basis of Cm. The symbol ij is the Kronecker delta , equal to 1 if
i = j and 0 if i 6= j .

Multiplication by a Unitary Matrix

In the last lecture we discussed the interpretation of matrix-vector products
Ax and A 1b. If A is a unitary matrix Q, these products become Qx and
Qb, and the same interpretations are of course still valid. As before, Qx is
the linear combination of the columns of Q with coecients x. Conversely,
 Lecture 2 Orthogonal Vectors and Matrices 15
Qb is the vector of coecients of the expansion of b
in the basis of columns of Q.
Schematically, the situation looks like this:

Multiplication by Q
b: Qb:
coecients of coecients of
the expansion of b the expansion of b
in fe1; : : : ; emg in fq1 ; : : : ; qm g
Multiplication by Q

These processes of multiplication by a unitary matrix or its adjoint pre-

serve geometric structure in the Euclidean sense, because inner products are
preserved. That is, for unitary Q,
(Qx)(Qy) = xy; (2:9)
as is readily veried by (2.4). The invariance of inner products means that
angles between vectors are preserved, and so are their lengths:
kQxk = kxk: (2:10)
In the real case, multiplication by an orthogonal matrix Q corresponds to a
rigid rotation (if det Q = 1) or re ection (if det Q = 1) of the vector space.

Exercises
1. Show that if a matrix A is both triangular and unitary, then it is diagonal.
2. The Pythagorean theorem asserts that for a set of n orthogonal vectors fxi g,
Xn X
n
k xik2 = kxik2.
i=1 i=1

(a) Prove this in the case n = 2 by an explicit computation of kx1 + x2k2 .

(b) Show that this computation also establishes the general case, by induction.
3. Let A 2 Cmm be hermitian. An eigenvector of A is a nonzero vector x 2 Cm
such that Ax = x for some  2 C, the corresponding eigenvalue.
(a) Prove that all eigenvalues of A are real.
 16 Part I Fundamentals
(b) Prove that if x and y are eigenvectors corresponding to distinct eigenvalues,
then x and y are orthogonal.
4. What can be said about the eigenvalues of a unitary matrix?
5. Let S 2 Cmm be skew-hermitian , i.e., S  = S .
(a) Show by using Exercise 1 that the eigenvalues of S are pure imaginary.
(b) Show that I S is nonsingular.
(c) Show that the matrix Q = (I S ) 1(I + S ), known as the Cayley transform
of S , is unitary. (This is a matrix analogue of a linear fractional transformation
(1 + s)=(1 s), which maps the left half of the complex s-plane conformally
onto the unit disk.)
6. If u and v are m-vectors, the matrix A = I + uv is known as a rank-one
perturbation of the identity . Show that if A is nonsingular, then its inverse
has the form A 1 = I + uv for some scalar , and give an expression for .
For what u and v is A singular? If it is singular, what is null(A)?
7. A Hadamard matrix is a matrix whose entries are all 1 and whose transpose
is equal to its inverse times a constant factor. It is known that if A is a
Hadamard matrix of dimension m > 2, then m is a multiple of 4, but it is
an unsolved problem whether there is a Hadamard matrix for every such m,
though examples are known for all cases m  424.
Show that the following recursive description provides a Hadamard matrix of
each dimension m = 2k , k = 0; 1; 2; : : : .
"
h i Hk Hk #
H0 = 1 ; Hk+1 = :
Hk Hk