CH 5

Chapter 5
Inner Product Spaces

5.1 Length and Dot Product in R
n
5.2 Inner Product Spaces
5.3 Orthonormal Bases:Gram-Schmidt Process
5.4 Mathematical Models and Least Square Analysis
Elementary Linear Algebra
R. Larsen et al. (6 Edition)
2/80
5.1 Length and Dot Product in R
n

Length:
The length of a vector in R
n
is given by
2 2
2
2
1
|| ||
n
v v v + + + = v
Notes: Properties of length
( )
( )
( )
( ) v v
v v
v v
v
c c =
= =
=
>
4
0 iff 0 3
1 2
0 1
is called a unit vector.
) , , , (
2 1 n
v v v = v
Notes: The length of a vector is also called its norm.
Elementary Linear Algebra: Section 5.1, p.278
3/80
Ex 1:
(a) In R
5
, the length of is given by

(b) In R
3
the length of is given by
) 2 , 4 , 1 , 2 , 0 ( = v
5 25 ) 2 ( 4 1 ) 2 ( 0 || ||
2 2 2 2 2
= = + + + + = v
1
17
17
17
3
17
2
17
2
|| ||
2 2 2
= = |
.
|
\
|
+ |
.
|
\
|
+ |
.
|
\
|
= v
) , , (
17
3
17
2
17
2
= v
(v is a unit vector)
4/80
A standard unit vector in R
n
:
( )
( ) <
>
=
0 2
0 1

c
c
cv u
u and v have the same direction
u and v have the opposite direction
Notes: (Two nonzero vectors are parallel)
{ } ( ) ( ) ( ) { } 1 , , 0 , 0 , 0 , , 1 , 0 , 0 , , 0 , 1 , , ,
2 1
=
n
e e e
Ex:
the standard unit vector in R
2
:

the standard unit vector in R
3
:
{ } ( ) ( ) { } 1 , 0 , 0 , 1 , = j i
{ } ( ) ( ) ( ) { } 1 , 0 , 0 , 0 , 1 , 0 , 0 , 0 , 1 , , = k j i
5/80
Thm 5.1: (Length of a scalar multiple)
Let v be a vector in R
n
and c be a scalar. Then
|| || | | || || v v c c =
|| || | |
| |
) (
) ( ) ( ) (
|| ) , , , ( || || ||
2 2
2
2
1
2 2
2
2
1
2
2 2
2
2
1
2 1
v
v
c
v v v c
v v v c
cv cv cv
cv cv cv c
n
n
n
n
=
+ + + =
+ + + =
+ + + =
=
Pf:
) , , , (
2 1 n
v v v = v
) , , , (
2 1 n
cv cv cv c = v
6/80
Thm 5.2: (Unit vector in the direction of v)
If v is a nonzero vector in R
n
, then the vector

has length 1 and has the same direction as v. This vector u
is called the unit vector in the direction of v.
|| || v
v
u =
Pf:
v is nonzero 0
1
0 > =
v
v
v
v
1
u =
(u has the same direction as v)
1 || ||
|| ||
1
|| ||
|| || = = = v
v v
v
u
(u has length 1 )
7/80
Notes:
(1) The vector is called the unit vector in the direction of v.

(2) The process of finding the unit vector in the direction of v
is called normalizing the vector v.
|| || v
v
8/80
Ex 2: (Finding a unit vector)
Find the unit vector in the direction of ,
and verify that this vector has length 1.
|
.
|
\
|
= =
+ +

=
14
2
,
14
1
,
14
3
) 2 , 1 , 3 (
14
1
2 ) 1 ( 3
) 2 , 1 , 3 (
|| ||
2 2 2
v
v
1
14
14
14
2
14
1
14
3

2 2 2
= = |
.
|
\
|
+ |
.
|
\
|
+ |
.
|
\
|
v
v

is a unit vector.
) 2 , 1 , 3 ( = v ( ) 14 2 1 3
2
2
2
= + + = v
Sol:
) 2 , 1 , 3 ( = v
9/80
Distance between two vectors:
The distance between two vectors u and v in R
n
is
|| || ) , ( v u v u = d
Notes: (Properties of distance)
(1)
(2) if and only if
(3)
0 ) , ( > v u d
0 ) , ( = v u d
v u =
) , ( ) , ( u v v u d d =
10/80
Ex 3: (Finding the distance between two vectors)
The distance between u=(0, 2, 2) and v=(2, 0, 1) is
3 1 2 ) 2 (
|| ) 1 2 , 0 2 , 2 0 ( || || || ) , (
2 2 2
= + + =
= = v u v u d
11/80
Dot product in R
n
:
The dot product of and
is the scalar quantity
Ex 4: (Finding the dot product of two vectors)
The dot product of u=(1, 2, 0, -3) and v=(3, -2, 4, 2) is
7 ) 2 )( 3 ( ) 4 )( 0 ( ) 2 )( 2 ( ) 3 )( 1 ( = + + + = v u
n n
v u v u v u + + + =
2 2 1 1
v u
) , , , (
2 1 n
u u u = u ) , , , (
2 1 n
v v v = v
12/80
Thm 5.3: (Properties of the dot product)
If u, v, and w are vectors in R
n
and c is a scalar,
then the following properties are true.
(1)
(2)
(3)
(4)
(5) , and if and only if
u v v u =
w u v u w v u + = + ) (
) ( ) ( ) ( v u v u v u c c c = =
2
|| || v v v =
0 > v v 0 = v v 0 = v
13/80
Euclidean n-space:
R
n
was defined to be the set of all order n-tuples of real
numbers. When R
n
is combined with the standard
operations of vector addition, scalar multiplication, vector
length, and the dot product, the resulting vector space is
called Euclidean n-space.
14/80
Sol:
6 ) 8 )( 2 ( ) 5 )( 2 ( ) a ( = + = v u
) 18 , 24 ( ) 3 , 4 ( 6 6 ) ( ) b ( = = = w w v u
12 ) 6 ( 2 ) ( 2 ) 2 ( ) c ( = = = v u v u
25 ) 3 )( 3 ( ) 4 )( 4 ( || || ) d (
2
= + = = w w w
) 2 , 13 ( ) 6 8 , ) 8 ( 5 ( 2 ) e ( = = w v
22 4 26 ) 2 )( 2 ( ) 13 )( 2 ( ) 2 ( = = + = w v u
Ex 5: (Finding dot products)
) 3 , 4 ( ), 8 , 5 ( , ) 2 , 2 ( = = = w v u
(a) (b) (c) (d) (e)
v u w v u ) ( ) 2 ( v u
2
|| || w ) 2 ( w v u
15/80
Ex 6: (Using the properties of the dot product)
Given

39 = u u 3 = v u
79 = v v
) 3 ( ) 2 ( v u v u + +
Sol:
) 3 ( 2 ) 3 ( ) 3 ( ) 2 ( v u v v u u v u v u + + + = + +
Find
254 ) 79 ( 2 ) 3 ( 7 ) 39 ( 3 = + + =
v v u v v u u u + + + = ) 2 ( ) 3 ( ) 2 ( ) 3 (
) ( 2 ) ( 6 ) ( 3 v v u v v u u u + + + =
) ( 2 ) ( 7 ) ( 3 v v v u u u + + =
16/80
Thm 5.4: (The Cauchy - Schwarz inequality)
If u and v are vectors in R
n
, then
( denotes the absolute value of )
|| || || || | | v u v u s
| | v u v u
v u v u
v v u u v u
v u
s
= = =
= =
55 5 11
1 1
5 , 11 , 1 = = = v v u u v u
Ex 7: (An example of the Cauchy - Schwarz inequality)
Verify the Cauchy - Schwarz inequality for u=(1, -1, 3)
and v=(2, 0, -1)
Sol:
Elementary Linear Algebra: Section 5.1, p.285-286
17/80
Note:
The angle between the zero vector and another vector is
not defined.
The angle between two vectors in R
n
:

t u u s s
= 0 ,
|| || || ||
cos
v u
v u
1 cos =
=t u
1 cos
0
=
= u
0 cos
2
<
< < t u
t
0 cos
2
=
=
t
u
0 cos
2
0
>
< <
t
u
u u u u
0 v u < 0 v u = 0 v u >
Opposite
direction
Same
direction
18/80
Ex 8: (Finding the angle between two vectors)
) 2 , 2 , 0 , 4 ( = u ) 1 , 1 , 0 , 2 ( = v
Sol:
( ) ( ) 24 2 2 0 4
2
2 2
2
= + + + = = u u u
1
144
12
6 24
12
|| || || ||
cos = =
=
v u
v u
u
( ) ( ) 6 1 1 0 2
2
2 2
2
= + + + = = v v v
12 ) 1 )( 2 ( ) 1 )( 2 ( ) 0 )( 0 ( ) 2 )( 4 ( = + + + = v u
= t u u and v have opposite directions. ) 2 ( v u =
19/80
Orthogonal vectors:
Two vectors u and v in R
n
are orthogonal if
0 = v u
Note:
The vector 0 is said to be orthogonal to every vector.
20/80
Ex 10: (Finding orthogonal vectors)
Determine all vectors in R
n
that are orthogonal to u=(4, 2).
0
2 4
) , ( ) 2 , 4 (
2 1
2 1
=
+ =
=
v v
v v v u
| |
(
0
2
1
1 0 2 4
t v
t
v =
=
2 1
,
2
R t ,t
t
e
|
.
|
\
|

= ,
2
v
) 2 , 4 ( = u
Let ) , (
2 1
v v = v
Sol:
21/80

Thm 5.5: (The triangle inequality)
n
, then || || || || || || v u v u + s +
Pf:
) ( ) ( || ||
2
v u v u v u + + = +
2 2 2 2
|| || | | 2 || || || || ) ( 2 || ||
) ( 2 ) ( ) (
v v u u v v u u
v v v u u u v u v v u u
+ + s + + =
+ + = + + + =
2
2 2
||) || || (||
|| || || || || || 2 || ||
v u
v v u u
+ =
+ + s
|| || || || || || v u v u + s +
Note:
Equality occurs in the triangle inequality if and only if
the vectors u and v have the same direction.
22/80
Thm 5.6: (The Pythagorean theorem)
n
, then u and v are orthogonal
if and only if
2 2 2
|| || || || || || v u v u + = +
23/80
Dot product and matrix multiplication:
(
(
(
(
=
n
u
u
u
2
1
u
(
(
(
(
=
n
v
v
v
2
1
v
] [ ] [
2 2 1 1
2
1
2 1 n n
n
n
T
v u v u v u
v
v
v
u u u + + + =
(
(
(
(
= =
v u v u
(A vector in R
n

is represented as an n1 column matrix)
) , , , (
2 1 n
u u u = u
24/80
Keywords in Section 5.1:
length:
norm:
unit vector:
standard unit vector :
normalizing:
distance:
dot product:
Euclidean n-space: n
Cauchy-Schwarz inequality: -
angle:
triangle inequality:
Pythagorean theorem:
25/80
5.2 Inner Product Spaces
(1)
(2)
(3)
(4) and if and only if
u v v u , , =
w u v u w v u , , , + = +
v u v u , , c c =
0 , > v v
0 , = v v
0 = v
Inner product:
Let u, v, and w be vectors in a vector space V, and let c be
any scalar. An inner product on V is a function that associates
a real number <u, v> with each pair of vectors u and v and
satisfies the following axioms.
26/80
Note:
V
R
n
space for vector product inner general ,
) for product inner Euclidean ( product dot
>= <
=
v u
v u
Note:
A vector space V with an inner product is called an inner
product space.
( ) - + , , V
Vector space:
Inner product space: ( ) > < - + , , , , V
Elementary Linear Algebra: Section 5.2, Addition
27/80
Ex 1: (The Euclidean inner product for R
n
)
Show that the dot product in R
n
satisfies the four axioms
of an inner product.
n n
v u v u v u + + + = =
2 2 1 1
, v u v u
) , , , ( , ) , , , (
2 1 2 1 n n
v v v u u u = = v u
Sol:
By Theorem 5.3, this dot product satisfies the required four axioms.
Thus it is an inner product on R
n
.
28/80
Ex 2: (A different inner product for R
n
)
Show that the function defines an inner product on R
2
,
where and .
2 2 1 1
2 , v u v u + = v u
) , ( ) , (
2 1 2 1
v v u u = = v u
Sol:
u v v u , 2 2 , ) (
2 2 1 1 2 2 1 1
= + = + = u v u v v u v u a

w u v u
w v u
, ,
) 2 ( ) 2 (
2 2
) ( 2 ) ( ,
2 2 1 1 2 2 1 1
2 2 2 2 1 1 1 1
2 2 2 1 1 1
+ =
+ + + =
+ + + =
+ + + = +
w u w u v u v u
w u v u w u v u
w v u w v u
) , ( ) (
2 1
w w b = w
Elementary Linear Algebra: Section 5.2, pp.293-294
29/80
Note: (An inner product on R
n
)
0 , ,
2 2 2 1 1 1
> + + + =
i n n n
c v u c v u c v u c v u
v u v u , ) ( 2 ) ( ) 2 ( , ) (
2 2 1 1 2 2 1 1
c v cu v cu v u v u c c c = + = + =
0 2 , ) (
2
2
2
1
> + = v v d v v
) 0 ( 0 0 2 0 ,
2 1
2
2
2
1
= = = = + = v v v v v v v
30/80
Ex 3: (A function that is not an inner product)
Show that the following function is not an inner product on R
3
.
3 3 2 2 1 1
2 v u v u v u + = v u
Sol:
Let ) 1 , 2 , 1 ( = v
0 6 ) 1 )( 1 ( ) 2 )( 2 ( 2 ) 1 )( 1 ( , Then < = + >= < v v
Axiom 4 is not satisfied.
Thus this function is not an inner product on R
3
.
31/80
Thm 5.7: (Properties of inner products)
Let u, v, and w be vectors in an inner product space V, and
let c be any real number.
(1)
(2)
(3)
0 , , = = 0 v v 0
w v w u w v u , , , + = +
v u v u , , c c =
Norm (length) of u:
u u u , || || =
u u u , || ||
2
=
Note:
32/80
u and v are orthogonal if .
Distance between u and v:
) ( = = v u v u v u v u , || || ) , ( d
Angle between two nonzero vectors u and v:
t u u s s = 0 ,
|| || || ||
,
cos
v u
v u
Orthogonal:
0 , = v u
) ( v u
33/80
Notes:
(1) If , then v is called a unit vector.

(2)
1 || || = v
0
1
=
=
v
v

g Normalizin
v
v
(the unit vector in the
direction of v)
not a unit vector
34/80
Ex 6: (Finding inner product)
) ( in s polynomial be 2 4 ) ( , 2 1 ) ( Let
2
2 2
x P x x x q x x p + = =
n n
b a b a b a q p + + + =
1 1 0 0
, is an inner product
? , ) ( = q p a ? || || ) ( = q b ? ) , ( ) ( = q p d c
Sol:
2 ) 1 )( 2 ( ) 2 )( 0 ( ) 4 )( 1 ( , ) ( = + + = q p a
21 1 ) 2 ( 4 , || || ) (
2 2 2
= + + = = q q q b
22 ) 3 ( 2 ) 3 (
, || || ) , (
3 2 3 ) (
2 2 2
2
= + + =
) ( = =
+ =
q p q p q p q p d
x x q p c
35/80
Properties of norm:
(1)
(2) if and only if
(3)
Properties of distance:
(1)
(2) if and only if
(3)
0 || || > u
0 || || = u 0 u=
|| || | | || || u u c c =
0 ) , ( > v u d
0 ) , ( = v u d
v u =
) , ( ) , ( u v v u d d =
36/80
Thm 5.8
Let u and v be vectors in an inner product space V.
(1) Cauchy-Schwarz inequality:

(2) Triangle inequality:

(3) Pythagorean theorem :
u and v are orthogonal if and only if
|| || || || || || v u v u + s +
Theorem 5.5
2 2 2
|| || || || || || v u v u + = +
Theorem 5.6
|| || || || | , | v u v u s
Theorem 5.4
37/80
Orthogonal projections in inner product spaces:
Let u and v be two vectors in an inner product space V,
such that . Then the orthogonal projection of u
onto v is given by
0 v =
v
v v
v u
u
v
) (
) (
=
,
,
proj
Note:
If v is a init vector, then .
The formula for the orthogonal projection of u onto v
takes the following simpler form.
1 || || ,
2
= = v v v
v v u u
v
) ( = , proj
38/80
Ex 10: (Finding an orthogonal projection in R
3
)
Use the Euclidean inner product in R
3
to find the
orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).
Sol:
10 ) 0 )( 4 ( ) 2 )( 2 ( ) 1 )( 6 ( , = + + = ) ( v u
5 0 2 1 ,
2 2 2
= + + = ) ( v v
) 0 , 4 , 2 ( ) 0 , 2 , 1 ( proj
5
10
= =
= v
v v
v u
u
v
Note:
). 0 , 2 , 1 ( to orthogonal is 4) 2, (4, 0) 4, (2, 4) 2, (6, proj = = = v u u
v
39/80
Thm 5.9: (Orthogonal projection and distance)
Let u and v be two vectors in an inner product space V,
such that . Then
0 v =
) (
) (
= <
v v
v u
v u u u
v
,
,
, ) , ( ) proj , ( c c d d
40/80
inner product:
inner product space:
norm:
distance:
angle:
orthogonal:
unit vector:
normalizing:
Cauchy Schwarz inequality: -
triangle inequality:
Pythagorean theorem:
orthogonal projection:
41/80
5.3 Orthonormal Bases: Gram-Schmidt Process
Orthogonal:
A set S of vectors in an inner product space V is called an
orthogonal set if every pair of vectors in the set is orthogonal.
Orthonormal:
An orthogonal set in which each vector is a unit vector is
called orthonormal.

{ }
=
=
= ) (
_ =
j i
j i
V S
j i
n
0
1
,
, , ,
2 1
v v
v v v
{ }
0 ,
, , ,
2 1
= ) (
_ =
j i
n
V S
v v
v v v
j i =
Note:
If S is a basis, then it is called an orthogonal basis or an
orthonormal basis.
42/80
Ex 1: (A nonstandard orthonormal basis for R
3
)
Show that the following set is an orthonormal basis.

)
`
|
.
|
\
|

|
|
.
|
\
|
|
.
|
\
|
=
3
1
,
3
2
,
3
2
,
3
2 2
,
6
2
,
6
2
, 0 ,
2
1
,
2
1
3 2 1
S
v v v
Sol:
Show that the three vectors are mutually orthogonal.
0
9
2 2
9
2
9
2
0 0
2 3
2
2 3
2
0 0
3 2
3 1
6
1
6
1
2 1
= + =
= + =
= + + =
v v
v v
v v
43/80
Show that each vector is of length 1.
Thus S is an orthonormal set.
1 || ||
1 || ||
1 0 || ||
9
1
9
4
9
4
3 3 3
9
8
36
2
36
2
2 2 2
2
1
2
1
1 1 1
= + + = =
= + + = =
= + + = =
v v v
v v v
v v v
44/80
The standard basis is orthonormal.
Ex 2: (An orthonormal basis for )
In , with the inner product
) (
3
x P
2 2 1 1 0 0
, b a b a b a q p + + = ) (
} , , 1 {
2
x x B =
) (
3
x P
Sol:
, 0 0 1
2
1
x x + + = v , 0 0
2
2
x x + + = v , 0 0
2
3
x x + + = v
0 ) 1 )( 0 ( ) 0 )( 1 ( ) 0 )( 0 ( ,
, 0 ) 1 )( 0 ( ) 0 )( 0 ( ) 0 )( 1 ( ,
, 0 ) 0 )( 0 ( ) 1 )( 0 ( ) 0 )( 1 ( ,
3 2
3 1
2 1
= + + = ) (
= + + = ) (
= + + = ) (
v v
v v
v v
Then
45/80
( )( ) ( )( ) ( )( )
( )( ) ( )( ) ( )( )
( )( ) ( )( ) ( )( ) 1 1 1 0 0 0 0
, 1 0 0 1 1 0 0
, 1 0 0 0 0 1 1
3 3 3
2 2 2
1 1 1
= + + = ) ( =
= + + = ) ( =
= + + = ) ( =
v , v v
v , v v
v , v v
46/80
Thm 5.10: (Orthogonal sets are linearly independent)
If is an orthogonal set of nonzero vectors
in an inner product space V, then S is linearly independent.
{ }
n
S v , , v , v
2 1
=
Pf:
S is an orthogonal set of nonzero vectors
0 and 0 i.e. > ) ( = = ) (
i i j i
j i v , v v , v
i c c c
c c c
i i n n
n n
= ) ( = ) + + + (
= + + +
0 , 0 ,
0 Let
2 2 1 1
2 2 1 1
v v v v v
v v v
) ( =
) ( + + ) ( + + ) ( + ) (
i i i
i n n i i i i i
c
c c c c
v , v
v , v v , v v , v v , v

2 2 1 1

t. independen linearly is 0 0 S i c
i i i
= = ) ( v , v
47/80
Corollary to Thm 5.10:
If V is an inner product space of dimension n, then any
orthogonal set of n nonzero vectors is a basis for V.
48/80
Ex 4: (Using orthogonality to test for a basis)
Show that the following set is a basis for .
4
R
)} 1 , 1 , 2 , 1 ( , ) 1 , 2 , 0 , 1 ( , ) 1 , 0 , 0 , 1 ( , ) 2 , 2 , 3 , 2 {(
4 3 2 1
= S
v v v v
Sol:

: nonzero vectors
0 2 2 6 2
0 2 4 0 2
0 2 0 0 2
4 1
3 1
2 1
= + =
= + + =
= + + =
v v
v v
v v
4 3 2 1
, , , v v v v
0 1 2 0 1
0 1 0 0 1
0 1 0 0 1
4 3
4 2
3 2
= + + =
= + + + =
= + + + =
v v
v v
v v
. orthogonal is S
4
for basis a is R S (by Corollary to Theorem 5.10)
49/80
Thm 5.11: (Coordinates relative to an orthonormal basis)
If is an orthonormal basis for an inner
product space V, then the coordinate representation of a vector
w with respect to B is
} , , , {
2 1 n
B v v v =
} , , , {
2 1 n
B v v v = is orthonormal
j i
j i
j i
=
=
= ) (
0
1
, v v
V e w
n n
k k k v v v w + + + =
2 2 1 1
(unique representation)
Pf:
is a basis for V
} , , , {
2 1 n
B v v v =
n n
v v w v v w v v w w ) ( + + ) ( + ) ( = , , ,
2 2 1 1

50/80
i k
k k k
k k k
i
i n n i i i i
i n n i
=
+ + + + =
+ + + =
, , ,
, ) ( ,
1 1
2 2 1 1
v v v v v v
v v v v v w

n n
v v w v v w v v w w ) ( + + ) ( + ) ( = , , ,
2 2 1 1

Note:
If is an orthonormal basis for V and , } , , , {
2 1 n
B v v v =
V e w
Then the corresponding coordinate matrix of w relative to B is
| |
(
(
(
(
) (
) (
) (
=
n
B
v w
v w
v w
w
,
,
,
2
1

51/80
Ex 5: (Representing vectors relative to an orthonormal basis)
Find the coordinates of w = (5, -5, 2) relative to the following
orthonormal basis for .
)} 1 , 0 , 0 ( , ) 0 , , ( , ) 0 , , {(
5
3
5
4
5
4
5
3
= B
3
R
Sol:
2 ) 1 , 0 , 0 ( ) 2 , 5 , 5 ( ,
7 ) 0 , , ( ) 2 , 5 , 5 ( ,
1 ) 0 , , ( ) 2 , 5 , 5 ( ,
3 3
5
3
5
4
2 2
5
4
5
3
1 1
= = = ) (
= = = ) (
= = = ) (
v w v w
v w v w
v w v w
(
(
(
=
2
7
1
] [
B
w
52/80
Gram-Schmidt orthonormalization process:
is a basis for an inner product space V } , , , {
2 1 n
B u u u =
1 1
Let u v =
}) ({
1 1
v w span =
}) , ({
2 2 1
v v w span =
} , , , { '
2 1 n
B v v v =
} , , , { ' '
2
2
n
n
B
v
v
v
v
v
v
1
1
=
is an orthogonal basis.
is an orthonormal basis.
=
= =

1
1

proj
1
n
i
i
i i
i n
n n n n
n
v
v , v
v , v
u u u v
W
2
2 2
2 3
1
1 1
1 3
3 3 3 3

proj
2
v
v , v
v , u
v
v , v
v , u
u u u v
W
= =
1
1 1
1 2
2 2 2 2

proj
1
v
v , v
v , u
u u u v
W
= =
53/80
Sol:
) 0 , 1 , 1 (
1 1
= = u v
) 2 , 0 , 0 ( ) 0 ,
2
1
,
2
1
(
2 / 1
2 / 1
) 0 , 1 , 1 (
2
1
) 2 , 1 , 0 (
2
2 2
2 3
1
1 1
1 3
3 3
= =
= v
v v
v u
v
v v
v u
u v
Ex 7: (Applying the Gram-Schmidt orthonormalization process)
Apply the Gram-Schmidt process to the following basis.
)} 2 , 1 , 0 ( , ) 0 , 2 , 1 ( , ) 0 , 1 , 1 {(
3 2 1
= B
u u u
) 0 ,
2
1
,
2
1
( ) 0 , 1 , 1 (
2
3
) 0 , 2 , 1 (
1
1 1
1 2
2 2
= =
= v
v v
v u
u v
54/80
} 2) 0, (0, 0), ,
2
1
,
2
1
( 0), 1, (1, { } , , { '
3 2 1

= = v v v B
Orthogonal basis
} 1) 0, (0, 0), ,
2
1
,
2
1
( 0), ,
2
1
,
2
1
( { } , , { ' '
3
3
2
2

= =
v
v
v
v
v
v
1
1
B
Orthonormal basis
55/80
Ex 10: (Alternative form of Gram-Schmidt orthonormalization process)
Find an orthonormal basis for the solution space of the
homogeneous system of linear equations.
0 6 2 2
0 7
4 3 2 1
4 2 1
= + + +
= + +
x x x x
x x x
Sol:

(
0 8 2 1 0
0 1 2 0 1
0 6 2 1 2
0 7 0 1 1
. . E J G
(
(
(
(
+
(
(
(
(
=
(
(
(
(
+
=
(
(
(
(
1
0
8
1
0
1
2
2
8 2
2

4
3
2
1
t s
t
s
t s
t s
x
x
x
x
56/80
Thus one basis for the solution space is
)} 1 , 0 , 8 , 1 ( , ) 0 , 1 , 2 , 2 {( } , {
2 1
= = u u B
( )
( ) ( )
( ) 1 , 2 , 4 , 3
0 1, 2, , 2
9
18
1 0, 8, 1,
,
,
0 1, 2, , 2
1
1 1
1 2
2 2
1 1
=
=
) (
) (
=
= =
v
v v
v u
u v
u v
( )( ) { } 1 , 2 , 4 , 3 0 , 1 , 2 , 2 ' = B
(orthogonal basis)
)
`
|
.
|
\
|
|
.
|
\
|
=
30
1
,
30
2
,
30
4
,
30
3
, 0 ,
3
1
,
3
2
,
3
2
' ' B
(orthonormal basis)
57/80
orthogonal set:
orthonormal set:
orthogonal basis:
orthonormal basis:
linear independent:
Gram-Schmidt Process: Gram-Schmidt
58/80
. in all and in all for 0 , if
orthogonal are V space product inner an of and subspaces The
2 2 1 1 2 1
2 1
W W
W W
v v v v = ) (
Orthogonal subspaces:
Ex 2: (Orthogonal subspaces)
. zero is in any vector and in any vector for 0 , because orthogonal are
)
1
1
1 -
span( and )
0
1
1
,
1
0
1
span(
subspaces The
2 1 2 1
2 1
W W
W W
= ) (
(
(
(
=
(
(
(
(
(
(
=
v v
5.4 Mathematical Models and Least Squares Analysis
59/80
Let W be a subspace of an inner product space V.
(a) A vector u in V is said to orthogonal to W,
if u is orthogonal to every vector in W.
(b) The set of all vectors in V that are orthogonal to every
vector in W is called the orthogonal complement of W.
(read perp)
} , 0 , | { W V W e = ) ( e =
w w v v
W
W
Orthogonal complement of W:
{ } ( ) { } 0 (2) 0 (1) = =

V V
Notes:
60/80
Notes:

{ }
W W
W W
V W
V W
=
=

) ( (3)
(2)
of subspace a is (1)
of subspace a is
0
{ }
W W
W W
R y W
x W R V
=
=
=
= =

) ( (3)
) 0 , 0 ( (2)
of subspace a is axis - (1) Then
axis , If
2
2
Ex:
Elementary Linear Algebra: Section 5.4, Addition
61/80
Direct sum:
Let and be two subspaces of . If each vector
can be uniquely written as a sum of a vector from
and a vector from , , then is the
direct sum of and , and you can write .
1
W
2
W
n
R
n
R e x
1
W
1
w
2
W
2
w
2 1
w w x + =
n
R
2 1
W W R
n
=
Thm 5.13: (Properties of orthogonal subspaces)
Let W be a subspace of R
n
. Then the following properties
are true.
(1)
(2)
(3)
n W W = +

) dim( ) dim(
= W W R
n
W W =

) (
1
W
2
W
62/80
Thm 5.14: (Projection onto a subspace)
If is an orthonormal basis for the
subspace S of V, and , then
} , , , {
2 1 t
u u u
V e v
t t W
u u v u u v u u v v ) ( + + ) ( + ) ( = , , , proj
2 2 1 1

Pf:
for W basis l orthonorma an is } , , , { and W proj
2 1 t W
u u u v e
t t W W W
u u v u u v v ) ( + + ) ( = , proj , proj proj
1 1

) proj proj (
, proj , proj
1 1
v v v
u u v v u u v v

=
) ( + + ) ( =
W
W
t t
W W
) , 0 , proj ( , ,
1 1
i
i
W
t t
= ) ( ) ( + + ) ( =

u v u u v u u v
63/80
Ex 5: (Projection onto a subspace)
( ) ( ) ( ) 3 , 1 , 1 , 0 , 0 , 2 , 1 , 3 , 0
2 1
= = = v w w
Find the projection of the vector v onto the subspace W.
{ } : ,
2 1
w w
Sol:
an orthogonal basis for W
{ } ( ) : 0 , 0 , 1 ),
10
1
,
10
3
, 0 ( , ,
2
2
1
1
2 1
)
`
=
w
w
w
w
u u
an orthonormal basis for W
}) , ({
2 1
w w span W =
( ) )
5
3
,
5
9
, 1 ( 0 , 0 , 1 )
10
1
,
10
3
, 0 (
10
6
, , proj
2 2 1 1
= + =
) ( + ) ( = u u v u u v v
W
64/80
Find by the other method:
| |
b b
b
b
v b w w
T 1 T
) (
T 1 T
2 1
) ( proj
) (
, ,
A A A A Ax
A A A x
Ax
A
A cs

= =
=
=
= =
65/80
Thm 5.15: (Orthogonal projection and distance)
Let W be a subspace of an inner product space V, and .
Then for all ,
V e v
W e w v w
W
proj =
|| || || proj || w v v v <
W
|| || min || proj || r o
W
w v v v
w
=
e
W
( is the best approximation to v from W) v
W
proj
66/80
Pf:
) proj ( ) proj ( w v v v w v + =
W W
) proj ( ) proj ( w v v v
W W
By the Pythagorean theorem
2 2 2
|| proj || || proj || || || w v v v w v + =
W W
0 proj proj > = w v v w
W W
2 2
|| proj || || || v v w v
W
>
|| || || proj || w v v v <
W
67/80
Notes:
(1) Among all the scalar multiples of a vector u, the
orthogonal projection of v onto u is the one that is
closest to v. (p.302 Thm 5.9)
(2) Among all the vectors in the subspace W, the vector
is the closest vector to v.

v
W
proj
68/80
Thm 5.16: (Fundamental subspaces of a matrix)
If A is an mn matrix, then
(1)

(2)

(3)
(4)

) ( )) ( (
) ( )) ( (
A CS A NS
A NS A CS
=
=
T
T
) ( )) ( (
) ( )) ( (
T
T
=
=
A CS A NS
A NS A CS
m m T
R A NS A CS R A NS A CS = =

)) ( ( ) ( ) ( ) (
n T n T
R A CS A CS R A NS A CS = =
T
)) ( ( ) ( ) ( ) (
69/80
Ex 6: (Fundamental subspaces)
Find the four fundamental subspaces of the matrix.
(
(
(
(
=
0 0 0
0 0 0
1 0 0
0 2 1
A (reduced row-echelon form)
Sol:
( ) ( ) { } ( )
4
of subspace a is 0 , 0 , 1 , 0 0 , 0 , 0 , 1 span ) ( R A CS =
( ) ( ) ( ) { } ( )
3
of subspace a is 1 , 0 , 0 0 , 2 , 1 span ) ( R A RS A CS = =
T
( ) { } ( )
3
of subspace a is 0 , 1 , 2 span ) ( R A NS =
70/80
( ) ( ) { } ( )
4
of subspace a is 1 , 0 , 0 , 0 0 , 1 , 0 , 0 span ) ( R A NS =
T
(
(
(
=
(
(
(
=
T
0 0 0 0
0 0 1 0
0 0 0 1
~
0 0 1 0
0 0 0 2
0 0 0 1
R A
Check:
) ( )) ( (
T
= A NS A CS
) ( )) ( ( A NS A CS =
T
4
) ( ) ( R A NS A CS
T
=
3
) ( ) ( R A NS A CS
T
=
t s
71/80
Ex 3 & Ex 4:

Let W is a subspace of R
4
and .
(a) Find a basis for W
(b) Find a basis for the orthogonal complement of W.
) 1 0, 0, 0, ( ), 0 1, 2, 1, (
2 1
= = w w
Sol:
2 1
0 0
0 0
1 0
0 1
~
1 0
0 1
0 2
0 1
w w
(
(
(
(
=
(
(
(
(
= R A (reduced row-echelon form)

}) , ({
2 1
w w span W =
72/80
( )
( ) ( ) { } 1 , 0 , 0 , 0 , 0 , 1 , 2 , 1
) (
= A CS W a
is a basis for W
( ) ( ) ( )
( ) ( ) { }

T
T

(
(
(
(
+
(
(
(
(
=
(
(
(
(

=
(
(
(
(
=
= =
W
t s
t
s
t s
x
x
x
x
A
A NS A CS W b
for basis a is 0 , 1 , 0 , 1 0 , 0 , 1 , 2
0
1
0
1
0
0
1
2
0
2

1 0 0 0
0 1 2 1
) (
4
3
2
1
Notes:
4
4
(2)
) dim( ) dim( ) dim( (1)
R W W
R W W
=
= +

73/80
Least squares problem:
(A system of linear equations)
(1) When the system is consistent, we can use the Gaussian
elimination with back-substitution to solve for x
b x = A
1 1 m n n m
(2) When the system is inconsistent, how to find the best possible
solution of the system. That is, the value of x for which the
difference between Ax and b is small.
74/80
Notes:
Least squares solution:
Given a system Ax = b of m linear equations in n unknowns,
the least squares problem is to find a vector x in R
n
that
minimizes with respect to the Euclidean inner
product on R
n
. Such a vector is called a least squares
solution of Ax = b.
b x A
x b x b b b
b
x b x
x
x
A A
A CS A A A
R
n
R
A CS
A CS
n
= =
e =
e
min
proj
is, That . to possible as close as is
) ) (
(i.e., of space column in the proj
such that in
vector a find to is problem square least The

) (
) (
75/80
( ) ) of subspace a is ( ) (
m
n
n m
R A CS A CS A
R
M A
e
e
e

x
x
b x
x b
x b
x b
b x
T T
T
T
=
=
= e

=
A A A
A A
A NS A CS A
A CS A
proj A
A CS
i.e.
0 )
(
) ( )) ( (
) ( )
Let
) (
(the normal system associated with Ax = b)
) system nt inconsiste an is b ( ) ( b = e x A A CS
76/80
Note: (Ax = b is an inconsistent system)
The problem of finding the least squares solution of
is equal to he problem of finding an exact solution of the
associated normal system .
b x = A
b x
T T
= A A A
77/80
Ex 7: (Solving the normal equations)
Find the least squares solution of the following system

(this system is inconsistent)

and find the orthogonal projection of b on the column space of A.

(
(
(
=
(
(
(
(
=
3
1
0
3 1
2 1
1 1
1
0
c
c
A b x
78/80
Sol:
(
=
(
(
(
=
(
=
(
(
(
=
11
4
3
1
0
3 2 1
1 1 1
14 6
6 3
3 1
2 1
1 1
3 2 1
1 1 1
b
T
T
A
A A
the associated normal system
(
=
(
=
11
4
14 6
6 3
1
0
c
c
A A A
T T
b x
79/80
the least squares solution of Ax = b
(
=
2
3
3
5
x
the orthogonal projection of b on the column space of A
(
(
(
=
(
(
(
(
= =

6
17
6
8
6
1
2
3
3
5
) (
3 1
2 1
1 1
proj x b A
A CS
80/80
orthogonal to W: W
orthogonal complement:
direct sum:
projection onto a subspace:
fundamental subspaces:
least squares problem:
normal equations:

CH 5

Uploaded by

Copyright:

Available Formats

CH 5

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CH 5

Uploaded by

Copyright:

Available Formats

Chapter 5

Inner Product Spaces

Elementary Linear Algebra: Section 5.3, pp.310-311

= R A (reduced row-echelon form)

Elementary Linear Algebra: Section 5.4, pp.322-323

(i.e., of space column in the proj

vector a find to is problem square least The

You might also like