Math for CS Tutorial 2-3 1

Solution of Systems of Linear Equations

Gaussian Elimination
LU Decomposition
Gram-Schmidt vector orthogonalization

Tutorial 2-3. Contents
Math for CS Tutorial 2-3 2
Linear Systems in Matrix Form
n n nn n n
n n
n n
b x a x a x a
b x a x a x a
b x a x a x a
= + + +
= + + +
= + + +

2 2 1 1
2 2 2 22 1 21
1 1 2 12 1 11
(
(
(
(

=
(
(
(
(

(
(
(
(

n n nn n n
n
n
b
b
b
x
x
x
a a a
a a a
a a a

2
1
2
1
2 1
2 22 21
1 12 11

Linear equation means that every variable is in the power 1 only.
If the variables in the equation are ordered x
1
, x
n
and the missing variables
x
i
are written as + 0x
i
+ , then the linear equation can be written in the
matrix notation, according to the matrix multiplication rules. Solution of linear
equations was one of the reasons, for matrix notation invention.
(1)
Math for CS Tutorial 2-3 3
An inconsistent example:
Geometric interpretation


(

=
(

5
4
4 2
2 1
2
1
x
x
(

0 0
2 1
Rank{A}=1
Rank{A|b}=2 Rank{A}
ERO:Multiply the first row with
-2 and add to the second row
(

3
4
0
2
0
1
Math for CS Tutorial 2-3 4
Full-rank systems
If Rank{A}=n
Det{A} = 0 A
-1
exists Unique solution
(

=
(

2
4
1 1
2 1
2
1
x
x
Math for CS Tutorial 2-3 5
First step of elimination
(
(
(
(
(
(

=
(
(
(
(
(
(

(
(
(
(
(
(

=
=
=
) 2 (
) 2 (
3
) 2 (
2
) 1 (
1
3
2
1
) 2 ( ) 2 (
3
) 2 (
2
) 2 (
3
) 2 (
33
) 2 (
32
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
) 1 (
11
) 1 (
1 1 ,
) 1 (
11
) 1 (
31 1 , 3
) 1 (
11
) 1 (
21 1 , 2
0
0
0
/
/
/
n
n
nn n n
n
n
n
n n
b
b
b
b
x
x
x
x
a a a
a a a
a a a
a a a a
a a m
a a m
a a m

(
(
(
(
(
(

=
(
(
(
(
(
(

(
(
(
(
(
(

) 1 (
) 1 (
3
) 1 (
2
) 1 (
1
3
2
1
) 1 ( ) 1 (
3
) 1 (
2
) 1 (
1
) 1 (
3
) 1 (
33
) 1 (
32
) 1 (
31
) 1 (
2
) 1 (
23
) 1 (
22
) 1 (
21
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
n
n
nn n n n
n
n
n
b
b
b
b
x
x
x
x
a a a a
a a a a
a a a a
a a a a

Pivotal element
Math for CS Tutorial 2-3 6
Second step of elimination
(
(
(
(
(
(

=
(
(
(
(
(
(

(
(
(
(
(
(

=
=
) 3 (
) 3 (
3
) 2 (
2
) 1 (
1
3
2
1
) 3 ( ) 3 (
3
) 3 (
3
) 3 (
33
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
) 2 (
22
) 2 (
2 2 ,
) 2 (
22
) 2 (
32 2 , 3
0 0
0 0
0
/
/
n
n
nn n
n
n
n
n n
b
b
b
b
x
x
x
x
a a
a a
a a a
a a a a
a a m
a a m

(
(
(
(
(
(

=
(
(
(
(
(
(

(
(
(
(
(
(

) 2 (
) 2 (
3
) 2 (
2
) 1 (
1
3
2
1
) 2 ( ) 2 (
3
) 2 (
2
) 2 (
3
) 2 (
33
) 2 (
32
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
0
0
0
n
n
nn n n
n
n
n
b
b
b
b
x
x
x
x
a a a
a a a
a a a
a a a a

Pivotal element
Math for CS Tutorial 2-3 7
Back substitution algorithm
(
(
(
(
(
(
(
(

=
(
(
(
(
(
(
(
(

(
(
(
(
(
(
(
(

) (
) 1 (
1
) 3 (
3
) 2 (
2
) 1 (
1
1
3
2
1
) (
) (
1
) (
1 1
) 3 (
3
) 3 (
33
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
0 0 0 0
0 0 0
0 0
0
n
n
n
n
n
n
n
nn
n
n n
n
n n
n
n
n
b
b
b
b
b
x
x
x
x
x
a
a a
a a
a a a
a a a a

| |
1 , , 2 , 1
1
1
1
) ( ) (
) (
1
1
) 1 (
1
) 1 (
1 1
1
) (
) (
=
(

=
= =

+ =

n n i x a b
a
x
x a b
a
x
a
b
x
n
i k
k
i
ik
i
i
i
ii
i
n
n
n n
n
n
n
n n
n
n
nn
n
n
n
Math for CS Tutorial 2-3 8
LU Decomposition
A=LU
Ax=b LUx=b
Define Ux=y
Ly=b Solve y by forward substitution
EROs must be performed on b as well as A
The information about the EROs are stored in L
Indeed y is obtained by applying EROs to b vector
Ux=y Solve x by backward substitution
Math for CS Tutorial 2-3 9
LU Decomposition by Gaussian
elimination
(
(
(
(
(
(
(
(

(
(
(
(
(
(
(
(

=


) (
) (
1
) (
1 1
) 3 (
3
) 3 (
33
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
4 , 3 , 2 , 1 ,
3 , 1 2 , 1 1 , 1
2 , 3 1 , 3
1 , 2
0 0 0 0
0 0 0
0 0
0
1
1
0
0 0 1
0 0 0 1
0 0 0 0 1
n
nn
n
n n
n
n n
n
n
n
n n n n
n n n
a
a a
a a
a a a
a a a a
m m m m
m m m
m m
m
A

Compact storage: The diagonal entries of L matrix are all 1s,

they dont need to be stored. LU is stored in a single matrix.
There are infinitely many different ways to decompose A.
Most popular one: U=Gaussian eliminated matrix
L=Multipliers used for elimination
Math for CS Tutorial 2-3 10
An example: Linear System
(
(
(

=
(
(
(

(
(
(

29
20
14
6 4 3
4 3 2
3 2 1
3
2
1
x
x
x
Rank{A}=n A
-1
exists Unique solution
Math for CS Tutorial 2-3 11
Gaussian Elimination
(
(
(

(
(
(

(
(
(

3
8
14
1 0 0
2 1 0
3 2 1
13
8
14
3 2 0
2 1 0
3 2 1
29
20
14
6 4 3
4 3 2
3 2 1
x
3
=3;
-x
2
-2x
3
=-8;
x
2
=2;
x
1
+2*2+3*3=14;
x
1
=14-4-9=1.

Math for CS Tutorial 2-3 12
LU Factorization
b L x U
b L x A L
LU A
b x A
1
1 1

=
=
=
=
(
(
(

(
(
(

(
(
(

(
(
(

(
(
(

(
(
(

3
8
14
1 0 0
2 1 0
3 2 1
1 2 1
0 1 2
0 0 1
13
8
14
3 2 0
2 1 0
3 2 1
1 0 3
0 1 2
0 0 1
29
20
14
6 4 3
4 3 2
3 2 1
1 0 0
0 1 0
0 0 1
L
-1
U=L
-1
A
(
(
(

=
(
(
(

(
(
(

1 0 0
2 1 0
3 2 1
6 4 3
4 3 2
3 2 1
1 2 1
0 1 2
0 0 1
Check that there were no mistakes:
L
-1
A

U

Math for CS Tutorial 2-3 13
LU Factorization
(
(
(

=
(
(
(

(
(
(

1 0 0
2 1 0
3 2 1
6 4 3
4 3 2
3 2 1
1 2 1
0 1 2
0 0 1
Check that there were no mistakes:
L
-1
A

U

(
(
(

=
(
(
(

(
(
(

3
8
14
29
20
14
1 2 1
0 1 2
0 0 1
L
-1
b b

L
-1

x
3
=3;
-x
2
-2x
3
=-8;
x
2
=2;
x
1
+2*2+3*3=14;
x
1
=14-4-9=1.

Math for CS Tutorial 2-3 14
the Gram-Schmidt procedure
for vector orthogonalization

The purpose
we can construct a set of orthonormal
vectors u
i
from a set of n-dimensional
vectors v
i
. 1im

And v
i
can be represented by the linear
combination of u
i

Math for CS Tutorial 2-3 15
G-S procedure (cont.)
1. Select a vector from vi arbitrarily, say v1

2. By normalizing its length, we obtain the first vector, say



3. Select v2 and subtract the projection of v2 onto u1, we
get

w
2
=v
2
(v
2
u
1
)u
1
v
v
u
1
1
1
=
Math for CS Tutorial 2-3 16
G-S procedure (cont.)
4. And normalizing



5. Now we can check, that

0 , , , ,
2 1 2 1 1 1 , 2 2 1 2 1
= = = v u v u u u v v u u u
w
w
u
2
2
2
=
Math for CS Tutorial 2-3 17
G-S procedure (cont.)
6. The procedure continues by selecting v
3
and subtract
its projection on u
1
and u
2
, we have
w
3
=v
3
(v
3
u
1
)u
1
(v
3
u
2
)u
2
7. Then, the orthonormal vector u
3
is

8. By continuing this procedure, we shall construct the
set of orthonormal vectors u
i

w
w
u
3
3
3
=
Math for CS Tutorial 2-3 18
G-S procedure: Example
Consider v
1
=(0,1,1); v
2
=(1,-1,0); v
3
=(1,0,1)
They are not orthogonal: (v
i
,v
j
)0.

Following G-S:
) 1 , 1 , 0 (
2
1
1
= u
)
2
1
,
2
1
, 1 ( ) 1 , 1 , 0 (
2
1
) 0 , 1 , 1 ( ) 1 , 1 , 0 (
2
1
2
1
) 0 , 1 , 1 (
1 2 1 2 2
= + =

= = u v u v w
Math for CS Tutorial 2-3 19
Example (cont.)
)
2
1
,
2
1
, 1 (
3
2
)
2
1
,
2
1
, 1 (
2
2
2
= = =
w
w
u
0 )
2
1
2
1
0 (
3
1
2 1
= + = u u
We can check, that:
2 2 3 1 1 3 3 3
u u v u u v v w =
Math for CS Tutorial 2-3 20
Example (cont.)
)
2
1
,
2
1
, 1 (
3
2
)
2
1
0 1 (
3
2
) 1 , 1 , 0 (
2
1
2
1
) 1 , 0 , 1 (
3

|
|
.
|

\
|
+ + = w
)
2
1
,
2
1
, 1 ( )
2
1
,
2
1
, 0 ( ) 1 , 0 , 1 ( )
2
1
,
2
1
, 1 (
2
3
3
2
) 1 , 1 , 0 (
2
1
) 1 , 0 , 1 (
3
= = w
) 0 , 0 , 0 (
3
= w
Why?