Numerical Methods

Linear systems of equations 1

 Motivating example:
Chemical reaction network

• We know:

Numerical Methods
• initial concentrations of all
components A0 , B0 , C0 , D0 , E0 , F0

• kinetic rate constants of reactions

k12 , …, k65 Constantinides & Mostoufi, Numerical methods for
Chemical Engineers with MATLAB applications, p 139

• We want to compute
• steady-state concentrations of all
components A, B, C, D, E, F

• We model all reaction rates as 1st

order kinetics 2
 Chemical reaction network: equations
• Mass balances:

dA/dt = k12B + k13C - k21A - k31A

dB/dt = k21A + k23C - k12B - k32B

Numerical Methods
dC/dt = k31A + k32B + k34D - k13C- k23C- k43C

dD/dt = k43C + k45E + k46F - k34D - k54D - k64D

dE/dt = k54D + k56F - k45E - k65E

Constantinides & Mostoufi, Numerical methods for
dF/dt = k65E + k64D - k46F - k56F Chemical Engineers with MATLAB applications, p 139

• Conservation of the species:

dA/dt+dB/dt+dC/dt+dD/dt+dE/dt+dF/dt = 0

A + B + C + D + E + F = const

• The data
A0 =1 ; B0 =0 ; C0 =0 ; D0 =0 ; E0 =1 ; F0 =0 ;
3
k21 =0.2 ; k31 =0.1 ; k32 =0.1 ; k34 =0.1 ; k54 =0.05 ;
k64 =0.2 ; k65 =0.1 ; k12 =0.1 ; k13 =0.05 ; k23 =0.05 ;
k43 =0.2 ; k45 =0.1 ; k46 =0.2 ; k56 =0.1 ; ;
 Chemical reaction network: steady state

Numerical Methods
Constantinides & Mostoufi, Numerical methods for
Chemical Engineers with MATLAB applications, p 139

• At the steady state, the studied system of ODE equations reduces to :

A + B + C + D + E + F = 2

0.2A - 0.2B + 0.05C = 0

0.1A + 0.1B - 0.3C + 0.1D = 0

0.2C - 0.35D + 0.1E + 0.2F = 0

0.05D - 0.2E + 0.1F = 0 4

0.2D + 0.1E - 0.3F = 0
 Another motivating example:
Radiosity methods

• Solve the diffusion of light for a room

Numerical Methods
• The radiosity of a pixel i, Bi, can be computed as a
solution of the following linear system of n
equations for a given 3D model with n pixels:

Bi = Ei + ρi ෍ Bj Fi,j
j

where: Bi is radiosity of pixel i, E𝑖 is the emission

of pixel i, ρi is the reflectivity of pixel i, and Fi,j is
the fraction of energy leaving pixel i arriving
directly at j

5
 Linear systems of equations (LSE)

• Linear systems of equations are ubiquitous in other

numerical problems:

Numerical Methods
• Interpolation (e.g., construction of the cubic spline
interpolant)
• Boundary value problems (BVP) of ordinary differential
equations (ODE)
• Partial differential equations (PDE)
• …

https://vimeo.com/160322285
 Linear systems of equations (LSE)

• Find:

Numerical Methods
• vector x ∈ Rn such that it satisfies Ax = b
where:
• System matrix A ∈ Rn,n with coefficients aij, i,j=1..n
• Right hand side vector (RHS) b ∈ Rn with coefficients bj,
j=1..n

• System matrix A can be:

• Full matrix
• Sparse matrix, that can have a sparsity pattern
7
• Diagonal, tridiagonal, band-matrix, block-diagonal
 Linear systems of equations (LSE)

• Solution methods:

Numerical Methods
• Direct methods
• provide the exact solution in a finite number of
operations (Gauss elimination, etc.)
• Iterative methods
• start with an approximate (guess) solution and
iterate to approach to the exact solution
• In all methods, we assume that matrix A is :
• regular/invertible <-> full column rank <-> full row
rank <-> non-zero determinant 8
 Gauss elimination method
• LSE of n equations and n unknowns

Numerical Methods
• Phase 1: transform to a upper triangular LSE

• Phase 2: solve using the back substitution method 9

 Phase 1: forward elimination
pivotal
element
• LSE

Numerical Methods
2x1 - x2 + x3 + 2x4 = 1 × -0.5
× -1.5
× -0.5
x1 + 4x2 + 2x3 - 4x4 = -2 +

3x1 + x2 - x3 - 10x4 = 5 +

x1 + x2 - x3 - 6x4 = 3 +

10
 Phase 1: forward elimination

pivotal
• LSE element

Numerical Methods
2x1 - x2 + x3 + 2x4 = 1

0 + 4.5x2 + 1.5x3 - 5x4 = -2.5 × -1/3

×-5/9

0 + 2.5x2 - 2.5x3 - 13x4 = 3.5 +

0 + 1.5x2 - 1.5x3 - 7x4 = 2.5 +

11
Eliminated
elements
 Phase 1: forward elimination

• LSE

Numerical Methods
2x1 - x2 + x3 + 2x4 = 1
pivotal
element

0 + 4.5x2 + 1.5x3 - 5x4 = -2.5

0 0 - 3⅓ x3 - 102⁄9 x4 = 4 8⁄9 × -3/5

0 0 - 2x3 - 5⅓ x4 = 3⅓ +

Eliminated 12
elements
 Phase 1: forward elimination

• LSE

Numerical Methods
2x1 - x2 + x3 + 2x4 = 1

0 + 4.5x2 + 1.5x3 - 5x4 = -2.5

0 0 - 3⅓ x3 - 102⁄9 x4 = 4 8⁄9

0 0 0 0.8 x4 = 0.4

Eliminated 13
elements
 Forward elimination: algorithm
pivotal
element

Numerical Methods
for i in range(n - 1): # Pivoting rows
for j in range(i + 1, n):
elFact = -a[j, i] / a[i, i]
for k in range(i, n):
a[j, k] += elFact * a[i, k]
b[j] += elFact * b[i]
return a, b

Eliminated in Element
previous i-1 steps Elements to compute
to eliminate
14
 Phase 2: back substitution

• LSE

Numerical Methods
2x1 - x2 + x3 + 2x4 = 1

0 + 4.5x2 + 1.5x3 - 5x4 = -2.5

0 0 - 3⅓ x3 - 102⁄9 x4 = 4 8⁄9

0 0 0 0.8 x4 = 0.4

x4 = 0.5 17
 Phase 2: back substitution

• LSE

Numerical Methods
2x1 - x2 + x3 = 0

+ 4.5x2 + 1.5x3 = 0

- 3⅓ x3 = 10

0.8 x4 = 0.4

x3 = -3 18
 Phase 2: back substitution

• LSE

Numerical Methods
2x1 - x2 = 3

+ 4.5x2 = 4.5

- 3⅓ x3 = 10

0.8 x4 = 0.4

x2 = 1 19
 Phase 2: back substitution

• LSE

Numerical Methods
2x1 = 4

+ 4.5x2 = 4.5

- 3⅓ x3 = 10

0.8 x4 = 0.4

x1 = 2 20
Back substitution: algorithm

Numerical Methods
bn(n)
xn = (n)
ann
1 é (n-1) (n-1) ù
xn-1 = b - an-1n xn û
(n-1) ë n-1
an-1n-1
1 é (i ) n (i ) ù
xi = (i ) êbi - å aik xk ú i = n -1,n - 2,…,1 21
aii ë k=i+1 û
 Gauss elimination algorithm: cost

• No of arithmetic operations required to solve the LSE

Numerical Methods
n(n -1)(2n + 5)
• Additions + subtractions: 6

n(n 2 + 3n -1)
• Multiplications + divisions
3

• The total cost is around 2 3

n
3
• Compare with Cramer’s rule (n!):
• on a gigaFLOPS machine, and n=50 we need to run for 1046 years 22

• in comparison, Gauss elimination requires 8.3*10 -5 secs

 Pivoting

• A simple example:

Numerical Methods
é 0.001 1 ù é x1 ù é 1 ù × -1000
ê úê ú=ê ú
ë 1 1 ûëê x 2 úû ë 2 û +
é 0.001 1 ù é x1 ù é 1 ù
ê úê ú=ê ú
ë 0 -999 û êë 2 úû ë -998 û
x

é x1 ù é 1000(1- 998 / 999) ù human é 1000 ù

ê ú=ê ú ¾¾¾¾®
intervention ê 999 ú
êë x2 úû ê 998 ú ê 998 ú
ë 999 û ê
ë 999 úû

• Assume now that we are using 4-digit arithmetic

é x1 ù é 1000(1- 0.998) ù é 2 ù
ê ú=ê ú=ê ú
x
êë 2 úû êë 0.998 úû ë 0.998 û
23
 Pivoting

• Pivoting (swapping the rows)

Numerical Methods
é 1 1 ù é x1 ù é 2 ù ×-0.001
ê úê ú=ê ú
ë 0.001 1 x
û êë 2 úû ë 1 û +
é 1 1 ù é x1 ù é 2 ù
ê úê ú=ê ú
ë 0 0.999 ê x
û ë 2 úû ë 0.998 û

é x1 ù é 2 - 998 / 999 ù human é 1000 ù

ê ú = ê 998 ú ¾intervention ê 999 ú
¾¾¾® ê ú
êë 2 úû êë
x ú
û êë
998
999 999 úû

• Assume now that we are using 4-digit arithmetic

é x1 ù é 2 - 0.998 ù é 1.002 ù
ê ú=ê ú=ê ú
êë x2 úû ë 0.998 û ë 0.998 û
24
 Partial (row) pivoting

Numerical Methods
Swap
these rows

Largest in
magnitude 25

• Swapping rows only changes the order of equations

 Full pivoting

Numerical Methods
Swap
these rows

Largest in
magnitude

Swap these columns 26

• When swapping columns, remember to swap the solution vector, i.e., xp becomes xi and
vice versa!
 Pivoting recapitulation
• Errors due to finite-precision arithmetic are introduced in each

Numerical Methods
arithmetic operation

• Introduced errors propagate

• When the pivotal element, a(i,i), is very small, the multiplying

factor in the process of elimination, elFact=a(j,i)/a(i,i), will be
very large

• In a limiting case, when a(i,i) is zero, one has to perform

division by zero!

• Solution: swap rows/columns or both rows and columns to use 27

a pivotal element with the largest magnitude

 LU decomposition

Numerical Methods
A L U

Ax = b
thus, we are Ly = b - forward substitution
LUx = b solving instead: Ux = y - back substitution
y

28
 LU decomposition: example
é a11 a12 a13 ù é l11 0 0 ù é 1 u12 u13 ù
ê ú ê úê ú
ê a21 a22 a23 ú Þ ê l21 l22 0 úê 0 1 u23 ú

Numerical Methods
ê a a32 a33 ú ê l31 l32 l33 ú êë 0 0 1 úû
ë 31
û ë û

l11 = a11 l21 = a21 l31 = a31

a12 l31u13 + l32u23 + l33 = a33
l11u12 = a12 Þ u12 =
l11
ß
a13
l11u13 = a13 Þ u13 = l33 = a33 - l31u13 - l32 u23
l11

l21u12 + l22 = a22 Þ l22 = a22 - l21u12

l31u12 + l32 = a32 Þ l32 = a32 - l31u12 29
a23 - l21u13
l21u13 + l22u23 = a23 Þ u23 =
l22
 LU decomposition: algorithm

Numerical Methods
1 Compute 1st column of L as: li1 = ai1
a1 j
2 Compute 1st row of U as: u1 j =
l 11

3 Compute sequentially columns of L and rows of U as:

j −1
lij = aij −  lik u kj j  i, i = 1,2,  , n
k =1

aij − k =1 lik ukj

i −1
30
uij = i  j, j = 2,3,  , n
lii
 LU decomposition: remarks
• Solving a LSE using LU decomposition requires

Numerical Methods
• factorization of A as LU
2 3
• forward substitution n + n2 + n2 total operations
3
• backward substitution
• For any number of RHS vectors (vectors b) we need to perform
LU decomposition only once!

31
 LU decomposition: remarks
• Solving a LSE using LU decomposition requires

Numerical Methods
• factorization of A as LU
2 3
• forward substitution n + n2 + n2 total operations
3
• backward substitution
• For any number of RHS vectors (vectors b) we need to perform
LU decomposition only once!
• Each element of A matrix is used only once to compute the
corresponding element of L or U matrix –> so L and U can be
stored in A
32
• Partial pivoting is sufficient, and widely implemented
 Cholesky decomposition

• For symmetric matrices, we can use the following decomposition:

Numerical Methods
 a11 a12 a13  l11 0 0  l11 l21 l31 
    
a12 a22 a23   l21 l22 0  0 l22 l32 
a a33  l31 l33   0 l33 
 13 a23 l32 0

• Since A=LLT, i.e. U=LT, we need to compute only L

• Half as many operations as LU

• A must be symmetric, positive definite, i.e., xTAx > 0, for all x≠0

• Recommended method for solving symmetric positive definite 33

systems
 Cholesky decomposition: example

 a11 a12 a13  l11 0 0  l11 l21 l31 

    
a23   l21

Numerical Methods
a12 a22 l22 0  0 l22 l32 
a a33  l31 l33   0 l33 
 13 a23 l32 0

l112 = a11 Þ l11 = a11

a12
l11l21 = a12 Þ l21 =
l11
a
l11l31 = a13 Þ l31 = 13
l11
l212 + l22 2 = a22 Þ l22 = a22 - l212
a23 - l21l31
l21l31 + l22l32 = a23 Þ l32 = 34
l22
l312 + l32 2 + l332 = a33 Þ l33 = a33 - l312 - l32 2
 Cholesky decomposition: algorithm

 a11 a12 a13  l11 0 0  l11 l21 l31 

    
a23   l21

Numerical Methods
a12 a22 l22 0  0 l22 l32 
a a33  l31 l33   0 l33 
 13 a23 l32 0

l11 = a11 i-1

l22 = a22 - l212 Þ lii = aii - å lik2 i = 1…n
k=1
l33 = a33 - l312 - l32 2
i-1

l21 = a12 l11 aij - å lik l jk

l31 = a13 l11 Þ l ji = k=1
j = i +1…n
l32 = (a23 - l21l31 ) l22 lii

• The algorithm would fail if it would be required to take the square

root of a negative number 35

• Therefore, another condition on A -> positive definiteness

 Recapitulation on direct methods
• Idea: transform the original LSE into a simpler,
equivalent LSE that is ‘easier’ to solve

Numerical Methods
• Easy to solve LSE: diagonal, upper or lower triangular
• Finitely many elementary operations – number
depending on n
• Gauss elimination methods, LU decomposition,
Cholesky decomposition
• Some well-known methods such as the Cramer’s rule or
the explicit computation of the inverse of A are
36
computationally prohibitive.
 Sparse systems
• For sparse systems
 a11 0 a13 a14 0
0 a 0 a24 0 
 22

a31 0 a33 0 a35 

Numerical Methods
 
a41 a42 0 a44 0
 0 0 a53 0 a55 
• During the process of forward elimination there is a fill-in effect
a11 0 a13 a14 0 non-zero, fill-in, terms
0 a 
a23 a24 0 
 22

0 0 
a33 
a34  
a35
 
0 0 0 
a44  
a45
 0 0 0 0  
a55

• Fill-in terms can considerably increase storage requirements 37

• For large and sparse systems - use iterative methods instead, or
direct methods adapted for sparse systems
 Iterative methods - Main idea

• The original system a11 x1 + a12 x2 + a13 x3 = b1

Numerical Methods
a21 x1 + a22 x2 + a23 x3 = b2
a31 x1 + a32 x2 + a33 x3 = b3
• Convert to
b1 a12 a13
x1 = - x2 - x3
a11 a11 a11
b2 a21 a
x2 = - x1 - 23 x3 Cx + d
a22 a22 a22
b3 a31 a32
x3 = - x1 - x2
a33 a33 a33
• Solve iteratively 38

x ( k ) = Cx ( k −1) + d
 Iterative methods for solving LSE

• If LSE are of a very large size, the computational efforts required for

Numerical Methods
applying direct methods are prohibitively expensive

• Iterative techniques for LSE (Relaxation methods)

• Jacobi method

• Gauss-Seidel method

• Successive Over Relaxation (SOR) method

• One iteration step typically costs ~n arithmetic operations in case of

a sparse matrix

• The cost depends on how many iteration steps are required to

39
obtain a certain accuracy.
 Jacobi method

x1(k ) = −
a12 (k −1) a13 (k −1) b1
x2 − x3 +
a11 a11 a11

Numerical Methods
x 2 (k ) = −
a21 (k −1) a23 (k −1) b2
x1 − x3 +
a22 a22 a22

x3 (k ) = −
a31 (k −1) a32 (k −1) b3
x1 − x2 +
a33 a33 a33

• Needs an initial solution vector

• Compute new values of solution, x(k), using the values from the
previous iteration, x(k-1)

40
 Jacobi method: example
• Apply the Jacobi method to solve the system:

5x1 - x2 = -7

Numerical Methods
x1 + 4x2 = 7
• we got
a12 (k-1) b1 1 (k-1) 7
x 1
(k )
=- x2 + = x2 -
a11 a11 5 5
a21 (k-1) b2 1 (k-1) 7
x2 (k )
=- x1 + = - x1 +
a22 a22 4 4
• if we take the starting point (x1, x2)=(0, 0) and perform iterations

1 7 7 1 7 7 21
x1(1) = ×0 - = - x1(2) = × - =- x1(6) = -1.0001
5 5 5 5 4 5 20
... 41
1 7 7
x2 (1) = - × 0 + =
1 7 7 42
x2 (2) = - × - + = x2 (6)
= 2.0002
4 4 4 4 5 4 20
 Gauss-Seidel method

• As soon as an element (k) a12 ( k-1) a13 ( k-1) b1

x1 = - x2 - x3 +
a11 a11 a11

Numerical Methods
from x is updated,
(k) a21 ( k ) a23 ( k-1) b2
it is used subsequently
x2 = - x1 - x3 +
a22 a22 a22
(k) a31 ( k ) a32 ( k ) b3
x3 =- x1 - x2 +
a33 a33 a33

• Typically converges more rapidly than the Jacobi method

• More difficult to parallelize

42
 Gauss-Seidel method: example
• System: 5x1 - x2 = -7
x1 + 4x2 = 7

Numerical Methods
• Gauss-Seidel iteration: a12 (k-1) b1 1 (k-1) 7
x1(k ) = - x2 + = x2 -
a11 a11 5 5
a21 (k ) b2 1 7
x2(k ) = - x1 + = - x1(k ) +
a22 a22 4 4
• if we take the starting point (x1, x2)=(0, 0) and perform iterations:

1 7 7 1 21 7 49 1 399 7
x1(1) = × 0 - = - x1(2) = × - = - x1(3) = × - = -1.001
5 5 5 5 10 5 50 5 200 5
1 7 7 21 1 49 7 399 1 7
x2(1) = - × - + = x2(2) = - × - + = x2(3) = - × -1.001+ = 2.0001
4 5 4 10 4 50 4 200 4 4

1 8001 7
x1(4) = × - = 0.99995 x1(6) = -1.0001 43
5 4000 5 Compared to Jacobi:
1 7 x2 (6) = 2.0002
x2(4) = - × -0.99995 + = 1.99999
4 4
 Successive Over Relaxation (SOR)
w
x1( k ) = (1- w )x1( k-1) + (b1 - a12 x2 ( k-1) - a13 x3( k-1) )
a11

Numerical Methods
w
x2 ( k ) = (1- w )x2( k-1) + (b2 - a21 x1( k ) - a23 x3( k-1) )
a22
w
x3( k ) = (1- w )x3( k-1) + (b3 - a31 x1( k ) - a32 x2( k ) )
a33

• Can be derived by multiplying the decomposed system obtained

from the Gauss-Seidel method by the relaxation parameter ω

• The iterative parameter ω should always be chosen such that

0 < ω < 2 (1< ω < 2 – over-relaxation, 0 < ω < 1 – dampening)

• Number of iterations to reach desired accuracy depends on ω 44

 On convergence of iterative methods

• A necessary and sufficient condition for convergence of these methods:

Numerical Methods
the magnitude of the largest eigenvalue of C should be smaller than 1
• If A is diagonally dominant matrix (i.e., the size of the diagonal element
is larger than the sum of the moduli of the other elements in the row -
|aii|>|ai1|+|ai2|+…+|aii-1|+|aii+1|+…+|ain|) then Jacobi and Gauss-
Seidel converge (sufficient condition)

• If A is symmetric and positive definite, then Gauss-Seidel converges

(Jacobi not necessarily) (sufficient condition)
• A necessary condition for convergence of SOR is 0 <ω < 2. If, in
addition, A is symmetric and positive definite, then this condition is
45
also sufficient
 Recapitulation on iterative solution methods

• Starting from an initial solution x0, the solution vector x is

Numerical Methods
iteratively computed: xk+1 = xk + w(b - Axk), w≠0
• Number of iterations (and therefore elementary operations) is
apriori unknown
• May not converge
• Jacobi method, Gauss-Seidel method, Succesive Over Relaxation
(SOR)
• There exist other methods that solve linear systems by
minimizing the residual of the equation: rk = (b - Axk) – see
46
Krylov subspace methods
 Sensitivity of linear systems

• For a given system Ax=b, the exact solution is given by x*=A-1b

Numerical Methods
• In many real world applications A and b are known only
approximately -> which might prevent us to find x*
෠
• Assume that we know the exact A but not b (we know 𝑏)
• Q: how sensitive is our solution with respect to uncertainty in b?

47
 Motivational example: deblurring images

• Images blurred: the lens out of focus, defects in lens or

Numerical Methods
optical system, turbulence…
• Frequent problem in astronomy (e.g., Hubble)
• Linear system Ax=b:
• A – blurring matrix
• x – sharp image
• b – blurred image

48

taken from L. Vandenberghe

 Deblurring images

Numerical Methods
49
Blurred image (b) Blurred image + noise
(jitter in b)
taken from L. Vandenberghe
 Deblurring images

• Solve the system for the two blurred images

Numerical Methods
What happened here???

50
𝐴−1 𝑏 𝐴−1 𝑏෠
taken from L. Vandenberghe
 Sensitivity of linear systems

• For a given system Ax=b, the exact solution is given by x*=A-1b

Numerical Methods
• In many real world applications A and b are known only
approximately -> which might prevent us to find x*
෠
• Assume that we know the exact A but not b (we know 𝑏)
• Q: how sensitive is our solution with respect to uncertainty in b?
෠ we can define
• If we denote 𝑥ො = 𝐴−1 𝑏,
• Deviation: e = 𝑥 ∗ − 𝑥ො
• Residual: r = 𝑏 − 𝐴𝑥ො = 𝐴𝑒

• We need residual to be small in some sense -> we need a

51
distance measure in the vector space
 Vector norms

• It is a mapping Rn -> R satisfying

Numerical Methods
• 𝑥 > 0, 𝑥 = 0 iff 𝑥 = 0 – positivity
• 𝑎𝑥 = 𝑎 𝑥 - homogeneity
• 𝑥+𝑦 ≤ 𝑥 + 𝑦 - triangle inequality

• P-norm
𝑝 𝑝 𝑝 1/𝑝
• 𝑥 𝑝 = 𝑥1 + 𝑥2 + ⋯ + 𝑥𝑛
• Manhattan norm (1 norm) : 𝑝 = 1
• Euclidian norm (2 norm): 𝑝 = 2
52
• Maximum norm (∞ norm): 𝑥 ∞ = 𝑚𝑎𝑥1≤i≤n 𝑥𝑖
 Matrix norms

Numerical Methods
• It is a mapping Rmxn -> R satisfying the three properties presented
for the vector norms. A matrix p-norm can additionally be
• 𝐴𝑥 ≤ 𝐴 𝑥 - consistent

• 𝐴𝐵 ≤ 𝐴 𝐵 - submultiplicative

• 2-norm
• 𝐴 2 = 𝜆𝑚𝑎𝑥 (𝐴𝑇 𝐴)

53
 Condition number
• Relates relative errors in the inputs (b) to relative errors in
solutions:

Numerical Methods
𝑥 ∗ −𝑥ො 𝑏∗ −𝑏෠
𝛿𝑥 = 𝑥∗
∝ 𝛿𝑏 = 𝑏∗

𝑥 ∗ − 𝑥ො ෠
𝐴−1 (𝑏 ∗ −𝑏) ෠
𝐴−1 (𝑏 ∗ −𝑏) 𝑏∗
𝛿𝑥 = ∗
= ∗
≤
𝑥 𝑥 𝑥∗ 𝑏∗

𝐴−1 𝐴−1
𝛿𝑥 ≤ 𝑏 ∗ 𝛿𝑏 = 𝐴𝑥 ∗ 𝛿𝑏 ≤ 𝐴−1 𝐴 𝛿𝑏
𝑥∗ 𝑥∗

• Condition number: 𝐴−1 𝐴

54
• For 2- norm ( 𝐴 2 = 𝜆𝑚𝑎𝑥 (𝐴𝑇 𝐴)), the condition number is equal
to the ratio of the largest and smallest eigenvalue
 Condition number

• Some properties
• Scale-invariant cond(aA)=cond(A), for all a

Numerical Methods
• It is norm dependent
• In general, cond(A) >> 1 system is highly sensitive:
• a small jitter in RHS (𝛿𝑏) results in big errors in solutions
(𝛿𝑥)
• ill-conditioned systems (otherwise well-conditioned
systems)

• When there is uncertainty in A

𝛿𝑥 ≤ 𝑐𝑜𝑛𝑑(𝐴)(𝛿𝐴 + 𝛿𝑏)
55
 Deblurring images

• A is nonsingular with condition number ~109

𝛿𝑥 ≤ 𝐴−1 𝐴 𝛿𝑏

Numerical Methods
56

𝐴−1 𝑏 𝐴−1 𝑏෠
taken from L. Vandenberghe