CSE6643/MATH6643:Numerical Linear Algebra: Haesun Park

CSE6643/MATH6643:Numerical Linear Algebra
Haesun Park
hpark@cc.gatech.edu
School of Computational Science and Engineering

College of Computing
Georgia Institute of Technology
Atlanta, GA 30332, USA
Lecture 2
Introduction
1
1.1
Introduction
Topics
1. Linear system solving Ax = b

2. Least square problem Ax b min kAx bk2
x
3. Eigenvalue problem Ax = x, (x 6= 0)
4. Linear programming min f (x) s.t. Ax = b where f (x) is a linear function.
CSE6643/MATH6643:Numerical Linear Algebra p.1/18
Introduction
1.2
Sherman-Morrison formula
Sherman-Morrison formula:
A is a nonsingular, and v T A1 u 6= 1 then
T 1
(A + uv )
=A
A1 uv T A1
1 + v T A1 u
where we call uv T rank 1 update.

Proof:
T
1
T 1
A1 uv T A1
Iuv T A1
uv T A1 uv T A1
(A+uv ) A 1+vT A1 u = I 1+v
+uv
A
=
T A1 u
1+v T A1 u
I
uv T A1 +(v T A1 u)uv T A1
1+v T A1 u
+ uv T A1 = I
Introduction
Sherman-Morrison-Woodbury formula:
A is nonsingular and A Rnn . U Rnk , V Rnk and Ik + V T A1 U is
nonsingular, then
(A + U V T )1 = A1 A1 U (Ik + V T A1 U )1 V A1
0
1
0
1
1 2 3
1
B
C
B
C
Example: A = @ 4 5 6 A, b = @ 1 A
7 8 10
1
Assume
to 3,0i.e. 1
0 A(1, 1) changes
1
3 2 3
2
B
C
B
C
A = @ 4 5 6 A = A + @ 0 A 1 0 0
7 8 10
0
1 uT vA1
1
1
A
b
then x
= A b = A
T
1+v Au
Introduction
1.3
Misc
Outer product u v T : n n
Inner product: uT v, 1 1
Range(A) or Ran(A)
Space spanned by columns of A

rank(A) = dim(Range(A))
orthogonal vectors u,v: v t u = kvk2 kuk2 cos = 0
orthonormal vectors u,v kuk2 = 1, kvk2 = 1
orthogonal matrix A Rnn satisfies AT A = In :

T
6
A A=6
4
aT1
h
.. 7
7
. 5 a1
aTn
an
= (aTi aj ) = In
Introduction
Special matrix:
diagonal matrix
triangle matrix: upper and lower
Hessenberg matrix: upper and lower
Symmetric matrix AT = A
Symmetric positive (semi) definite: if for all v 6= 0, v T Av > 0 (v T Av 0)
Norms & Floating-Point Error

2
2.1
Vector norm
Definition: Vector norm is a function f : Rn R s.t. f (x) = kxk?
1. kxk 0 and kxk = 0 iff x = 0

2. kxk = || kxk for all R
3. kx + yk kxk + kyk
Common vector norms
`Pn
p 1/p
|x
|
p-norms kxkp =
Pn i=1 i
p = 1, kxk1 = i=1 |xi |
qP
2
n
p = 2, kxk2 =
|x
|
i
i=1
p = , kxk = max1in |xi |
kxk = lim kxkp = lim (|xi |p )1/p = max1in |xi |
p

2.2
Equivalence of Vector Norms
All norms on Rn are equivalent: For two norms kk and kk

there is a constant C > 0 s.t. kxk C kxk for all x Rn .
\
1
2
1
1
1
n
1
1
n
1

2.3
Matrix Norm
Definition: matrix Norm is a function f : Rmn R s.t. f (A) = kAk
1.
2.
3.
4.
kAk 0 and kAk = 0 iff A = 0

kAk = || kAk for all R
kA + Bk kAk + kBk
kABk kAk kBk

Common matrix norms
kAx k
P-norm kAkp = max kxk p for all x Rn1

x6=0
Pm p
kAk1 = max1jn P
i=1 |aij |
|aij |
kAk = max1im n
3
2 j=1
1 2
h
i
6
7
e.g.kAk1 = 12, A = 4 3 4 5 9 12 12
5 6
2
3
2
3
1
2
3
6
7
6
7
e.g.kAk = 11, A = 4 3
4 5 4 7 5 11
5
6
11
kAkp is the p-norm of the largest vector obtained by applying A to a unit
p-norm vector.
Frobenius
norm, NOT one of p-norm
qP
Pm
2
n
|a
|
kAkF =
ij
i=1
j=1

2.4
Equivalence of Matrix Norms
For A Rmn :
\
1
2
1
1
m
m
2
n
1
m
n
n
1
F
n
n
nm
we have kAk2 kAkF
X:n n,non-singular
kAkX = X AX
e.g. kA BkF = 105
Exercise: prove
kAk1 n kAk
kAk1 n kAkF
m
1
n
1

2.5
Orthogonal Transformation Invariant
F-norm and 2-norm are invariant w.r.t. orthogonal transformation

i.e. Q Rmm and Z Rnn are orthogonal matrices
kQAZk2 = kAk2
kQAZkF = kAkF
Will show
1. kQAkF = kAkF
2. kAZkF = kAkF

Proof
h
1. kQAkF = Q a1
kQa1 k22
kQan k22
T
an
i2
2
2
= kQa1 k + + kQan k =
F
+ +
note: kXk22 = X X
kQai k22 = (Qai )T Qai = aTi QT Qai = aTi ai = kai k22
2
3
T
1
6 . 7
7
6
2. exercise kAZkF =
4 .. 5 Z
a
m
F

2.6
Floating-Point Arithmetic
Example. Assume 4-digits of accuracy

f l(1 + 104 1) = f l(f l(1 + 104 ) 1) = 0
f l(1 + 104 1) = f l(f l(1 1) + 104 ) = 104
f l (f l(f l(1 + ) + ) + + ) 6= f l(1 + + f l(f l( + ) + )
Floating-point representation:
Set of floating point numbers
F = (b, t, L, u) = {d1 dt be | 0 dk b 1, d16=0 , L e U } , {0}
where [L, U ] is exponent range, b is the base, t is mantissa length.
For any f F , m |F | M , m = bL1 , M = bu (1 bt ).
overflow |f | > M
underflow |f | < M

Theorem: f l(x) = x(1 + ), || < where is unit round-off , or machine
precision, the smallest number that satisfies f l(1 + ) 6= 1.
e.g. = 2 1016 in matlab
f l(x) = x + x
|f l(x) x| = |x| |x| absolute error
|f l(x)x|
relative error
|x|
e.g. C = AB has two sources of errors
1. representation f l(A) = A + EA , f l(B) = B + EB
2. computation f l(AB) = AB+?
Theorem: A = f l(A) = A + E where E = (eij ), |eij aij |
LU Factorization
3
LU Factorization
3.1
Linear Equations
Given A Rnn and b Rn1 . find x such that Ax = b. Solution exists?
1. When A is nonsingular, i.e. det(A) 6= 0, i.e. As columns are linearly

independent , solution unique x = A1 b.
2. When A is singular, b Range(A), infinite solutions.
3. When A is singular, b
/ Range(A), no solution
LU Factorization
For what matrix A is Ax = b an easy problem?
1. A is diagonal
2. A is triangular, focus on upper-triangular here
"
#"
# "
#
a11 a12
x1
b1
e.g. n = 2,
=
0
a12
x2
b2
a22 x2 = b2 x2 = b2 /a22
consider general n, k-th equation is
0 + + 0 + akk xk + ak,k+1 xk+1 + + ak,n xn = bk
n
P
xk = (bk
akj xj )/akk
j=k+1
LU Factorization
Algorithm 1: Back Substitution
P
2
for k = n : 1 : 1 > n
(2(n
k)
+
1)
n
flops
k=1
for j = k + 1 : n -> 2(n k) flops
bk = bk akj xj
end
xk = bk /ak,k
end
LU Factorization
Flops
1 flop one (, /, +, )
(old version) 1 flop one (or/, +or)
What about general A?
6
6
Lets assume A can be factored into A = LU where L = 6
4
and U is upper-triangular.
Then Ax = b can be solved by Ax = b L(U x) = b
1. Solve Ly = b for y
2. Solve U x = y for x
1
x
x
x
1
x
x
1
x
3
1
7
7
7,
5

CSE6643/MATH6643:Numerical Linear Algebra: Haesun Park

Uploaded by

Copyright:

Available Formats

CSE6643/MATH6643:Numerical Linear Algebra: Haesun Park

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CSE6643/MATH6643:Numerical Linear Algebra: Haesun Park

Uploaded by

Copyright:

Available Formats

CSE6643/MATH6643:Numerical Linear Algebra

School of Computational Science and Engineering

1. Linear system solving Ax = b

4. Linear programming min f (x) s.t. Ax = b where f (x) is a linear function.

CSE6643/MATH6643:Numerical Linear Algebra p.1/18

where we call uv T rank 1 update.

CSE6643/MATH6643:Numerical Linear Algebra p.2/18

CSE6643/MATH6643:Numerical Linear Algebra p.3/18

Space spanned by columns of A

orthogonal matrix A Rnn satisfies AT A = In :

CSE6643/MATH6643:Numerical Linear Algebra p.4/18

CSE6643/MATH6643:Numerical Linear Algebra p.5/18

Norms & Floating-Point Error

Norms & Floating-Point Error

Definition: Vector norm is a function f : Rn R s.t. f (x) = kxk?

1. kxk 0 and kxk = 0 iff x = 0

Common vector norms

CSE6643/MATH6643:Numerical Linear Algebra p.6/18

Norms & Floating-Point Error

Equivalence of Vector Norms

All norms on Rn are equivalent: For two norms kk and kk

CSE6643/MATH6643:Numerical Linear Algebra p.7/18

Norms & Floating-Point Error

Definition: matrix Norm is a function f : Rmn R s.t. f (A) = kAk

kAk 0 and kAk = 0 iff A = 0

CSE6643/MATH6643:Numerical Linear Algebra p.8/18

Norms & Floating-Point Error

P-norm kAkp = max kxk p for all x Rn1

Norms & Floating-Point Error

Equivalence of Matrix Norms

CSE6643/MATH6643:Numerical Linear Algebra p.10/18

Norms & Floating-Point Error

Orthogonal Transformation Invariant

F-norm and 2-norm are invariant w.r.t. orthogonal transformation

CSE6643/MATH6643:Numerical Linear Algebra p.11/18

Norms & Floating-Point Error

CSE6643/MATH6643:Numerical Linear Algebra p.12/18

Norms & Floating-Point Error

Example. Assume 4-digits of accuracy

CSE6643/MATH6643:Numerical Linear Algebra p.13/18

Norms & Floating-Point Error

CSE6643/MATH6643:Numerical Linear Algebra p.14/18

Given A Rnn and b Rn1 . find x such that Ax = b. Solution exists?

1. When A is nonsingular, i.e. det(A) 6= 0, i.e. As columns are linearly

CSE6643/MATH6643:Numerical Linear Algebra p.15/18

CSE6643/MATH6643:Numerical Linear Algebra p.16/18

CSE6643/MATH6643:Numerical Linear Algebra p.17/18

CSE6643/MATH6643:Numerical Linear Algebra p.18/18

You might also like