Lecture Notes for Math-CSE 451:

Introduction to Numerical Computation

Wen Shen

2011
2

NB! These notes are used by myself in the course. They are provided to students as a
supplement to the textbook. They can not substitute the textbook.
Textbook:
Numerical Mathematics and Computing,
by Ward Cheney and David Kincaid, published by Brooks/Cole publishing Company,
2000. ISBN 0-534-35184-0
Contents

1 Computer arithmetic 9
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Representation of numbers in different bases . . . . . . . . . . . . . . . . 10
1.3 Floating point representation . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Loss of significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.5 Review of Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2 Polynomial interpolation 19
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Lagrange interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Newton’s divided differences . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Errors in Polynomial Interpolation . . . . . . . . . . . . . . . . . . . . . 25
2.5 Numerical differentiations . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3 Splines 31
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 First degree and second degree splines . . . . . . . . . . . . . . . . . . . 32
3.3 Natural cubic splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4 Numerical integration 39
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 Trapezoid rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3 Simpson’s rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4 Recursive trapezoid rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5 Romberg Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.6 Adaptive Simpson’s quadrature scheme . . . . . . . . . . . . . . . . . . 47

3
4 CONTENTS

4.7 Gaussian quadrature formulas . . . . . . . . . . . . . . . . . . . . . . . . 49

5 Numerical solution of nonlinear equations. 51

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.2 Bisection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.3 Fixed point iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.4 Newton’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.5 Secant method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.6 System of non-linear equations . . . . . . . . . . . . . . . . . . . . . . . 59

6 Direct methods for linear systems 61

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6.2 Gaussian elimination, simplest version . . . . . . . . . . . . . . . . . . . 62
6.3 Gaussian Elimination with scaled partial pivoting . . . . . . . . . . . . . 63
6.4 LU-Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.5 Review of linear algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.6 Tridiagonal and banded systems . . . . . . . . . . . . . . . . . . . . . . 69

7 Iterative solvers for linear systems 73

7.1 General introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.2 Jacobi iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
7.3 Gauss-Seidal iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
7.4 SOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
7.5 Writing all methods in matrix-vector form . . . . . . . . . . . . . . . . . 77
7.6 Analysis for errors and convergence . . . . . . . . . . . . . . . . . . . . . 79

8 Least Squares 81
8.1 Problem description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
8.2 Linear regression and basic derivation . . . . . . . . . . . . . . . . . . . 81
8.3 LSM with parabola . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8.4 LSM with non-polynomial . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8.5 General linear LSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
8.6 Non-linear LSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

9 ODEs 89
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
CONTENTS 5

9.2 Taylor series methods for ODE . . . . . . . . . . . . . . . . . . . . . . . 90

9.3 Runge Kutta methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
9.4 An adaptive Runge-Kutta-Fehlberg method . . . . . . . . . . . . . . . . 94
9.5 Multi-step methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
9.6 Methods for first order systems of ODE . . . . . . . . . . . . . . . . . . 98
9.7 Higher order equations and systems . . . . . . . . . . . . . . . . . . . . 99
9.8 Stiff systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6 CONTENTS
List of Figures

1.1 The big picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2 32-bit computer with single precision . . . . . . . . . . . . . . . . . . . . 12
1.3 Mean Value Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.1 Finite differences to approximate derivatives . . . . . . . . . . . . . . . . 29

3.1 Linear splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1 Trapezoid rule: straight line approximation in each sub-interval. . . . . 40

4.2 Simpson’s rule: quadratic polynomial approximation (thick line) in each
sub-interval. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.3 Simpson’s rule: adding the constants in each node. . . . . . . . . . . . . 43
4.4 Recursive division of intervals, first few levels . . . . . . . . . . . . . . . 45
4.5 Romberg triangle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.1 Newton’s method: linearize f (x) at xk . . . . . . . . . . . . . . . . . . . . 56

7.1 Splitting of A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

8.1 Linear regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

7
8 LIST OF FIGURES
Chapter 1

Computer arithmetic

1.1 Introduction
What are numeric methods? They are algorithms that compute approximations to
solutions of equations or similar things.
Such algorithms should be implemented (programmed) on a computer.

physical model
Y
verification
? physical explanation of the results
mathematical model

presentation of results
* visualization
R
solve the model with
numerical methods
 I

mathematical theorems computer

numerical analysis programming

Figure 1.1: The big picture

Numerical methods are not about numbers. It is about mathematical insights.

We will study some basic classical types of problems:

• development of algorithms;

• implementation;

9
10 CHAPTER 1. COMPUTER ARITHMETIC

• a little bit of analysis, including error-estimates, convergence, stability etc.

We will use Matlab throughout the course for programming purpose.

1.2 Representation of numbers in different bases

Some bases for numbers:

10: decimal, daily use;

2: binary, computer use;

8: octal;

18: hexadecimal, ancient China;

20: ancient France;

• etc...

In principle, one can use any number β as the base.

integer part fractional part

 
z }| { z }| {
an an−1 · · · a1 a0 . b1 b2 b3 · · ·
β
n n−1
= an β + an−1 β + · · · + a1 β + a0 (integer part)
−1 −2 −3
+b1 β + b2 β + b3 β + ··· (fractonal part)

Converting between different bases:

Example 1. octal → decimal

(45.12)8 = 4 × 82 + 5 × 8 + 1 × 8−1 + 2 × 8−2 = (37.15625)10

Example 2. octal → binary

1.2. REPRESENTATION OF NUMBERS IN DIFFERENT BASES 11

Observe
(1)8 = (1)2
(2)8 = (10)2
(3)8 = (11)2
(4)8 = (100)2
(5)8 = (101)2
(6)8 = (110)2
(7)8 = (111)2
(8)8 = (1000)2

Then,
(5034)8 = (|{z}
101 |{z}
000 |{z}
011 |{z}
100 )2
5 0 3 4
and
(110 010 111 001)2 = (|{z}
6 2
|{z} 7
|{z} 1 )8
|{z}
110 010 111 001

Example 3. decimal → binary: write (12.45)10 in binary base.

Answer. integer part
2 12 0
2 6 0
⇒ (12)10 = (1100)2
2 3 1
2 1 1
fractional part
0.45 × 2
0.9 × 2
1.8 × 2
1.6 × 2
1.2 × 2 ⇒ (0.45)10 = (0.01110011001100 · · · )2 .
0.4 × 2
0.8 × 2
1.6 × 2
···
Put together:
(10.45)10 = (1100.01110011001100 · · · )2
12 CHAPTER 1. COMPUTER ARITHMETIC

1.3 Floating point representation

normalized scientific notation
decimal: x = ±r × 10n , 10−1 ≤ r < 1 . (ex: r = 0.d1 d2 d3 · · · , d1 6= 0)
binary: x = ±r × 10n , 2−1 ≤r<1
octal: x = ±r × 8n , 8−1 ≤r<1
r: normalized mantissa
n: exponent
Computers represent numbers with finite length. These are called machine numbers.
In a 32-bit computer, with single-precision:

1 byte 8 bytes radix point 23 bytes

? ? ? ?
s c f
6 6 6
sign of
mantissa biased exponent mantissa

Figure 1.2: 32-bit computer with single precision

The exponent: 28 = 256. It can represent numbers from −127 to 128.

The value of the number:
(−1)s × 2c−127 × (1.f )2
This is called: single-precision IEEE standard floating-point.
smallest representable number: xmin = 2−127 ≈ 5.9 × 10−39 .
largest representable number: xmax = 2128 ≈ 2.4 × 1038 .
We say that x underflows if x < xmin , and consider x = 0.
We say that x overflows if x > xmax , and consider x = ∞.

Computer errors in representing numbers:

• round off relative error: ≤ 0.5 × 2−23 ≈ 0.6 × 10−7

• chopping relative error: ≤ 1−23 ≈ 1.2 × 10−7

Floating point representation of a number x: call it fl(x)

fl(x) = x · (1 + δ)

fl(x) − x
relative error: = =δ
x
1.4. LOSS OF SIGNIFICANCE 13

absolute error: = fl(x) − x = δ · x

|δ| ≤ ε, where ε is called machine epsilon, which represents the smallest positive number
detectable by the computer, such that fl(1 + ε) > 1.
In a 32-bit computer: ε = 2−23 .

Error propagation (through arithmetic operation)

Example 1. Addition, z = x + y.
Let
fl(x) = x(1 + δx ), fl(y) = y(1 + δy )
Then

fl(z) = fl (fl(x) + fl(y))

= (x(1 + δx ) + y(1 + δy )) (1 + δz )
= (x + y) + x · (δx + δz ) + y · (δy + δz ) + (xδx δz + yδy δz )
≈ (x + y) + x · (δx + δz ) + y · (δy + δz )

absolute error = fl(z) − (x + y) = x · (δx + δz ) + y · (δy + δz )

= x · δx + y · δy + (x + y) · δz
| {z } | {z } | {z }
abs. err. abs. err. round off err
for x for y
| {z }
propagated error

fl(z) − (x + y) xδx + yδy

relative error = = + δz
x+y x+y |{z}
| {z }
propagated err round off err

1.4 Loss of significance

This typically happens when one gets too few significant digits in subtraction.
For example, in a 8-digit number:

x = 0.d1 d2 d3 · · · d8 × 10−a

d1 is the most significant digit, and d8 is the least significant digit.

Let y = 0.b1 b2 b3 · · · b8 × 10−a . We want to compute x − y.
14 CHAPTER 1. COMPUTER ARITHMETIC

If b1 = d1 , b2 = d2 , b3 = d3 , then

x − y = 0.000c4 c5 c6 c7 c8 × 10−a

We lose 3 significant digits.

Example 1. Find the roots of x2 − 40x + 2 = 0. Use 4 significant digits in the

computation.
Answer. The roots for the equation ax2 + bx + c = 0 are

1  p 
r1,2 = −b ± b2 − 4ac
2a
In our case, we have √
x1,2 = 20 ± 398 ≈ 20 ± 19.95
so
x1 ≈ 20 + 19.95 = 39.95, (OK)

x2 ≈ 20 − 19.95 = 0.05, not OK, lost 3 sig. digits

To avoid this: change the algorithm. Observe that x1 x2 = c/a. Then

c 2
x2 = = ≈ 0.05006
ax1 1 · 39.95

We get back 4 significant digits in the result.

1.5 Review of Taylor Series

Given f (x), smooth function. Expand it at point x = c:

1 ′′ 1
f (x) = f (c) + f ′ (c)(x − c) + f (c)(x − c)2 + f ′′′ (c)(x − c)3 + · · ·
2! 3!
or using the summation sign
∞
X 1 (k)
f (x) = f (c)(x − c)k .
k!
k=0

This is called Taylor series of f at the point c.

Special case, when c = 0, is called Maclaurin series:
∞
1 ′′ 1 X 1
f (x) = f (0) + f ′ (0)x + f (0)x2 + f ′′′ (0)x3 + · · · = f (k) (0)xk .
2! 3! k!
k=0
1.5. REVIEW OF TAYLOR SERIES 15

Some familiar examples

∞
X xk x2 x3
ex = = 1+x+ + + ··· , |x| < ∞
k! 2! 3!
k=0
∞
X x2k+1 x3 x5 x7
sin x = (−1)k = x− + − + ··· , |x| < ∞
(2k + 1)! 3! 5! 7!
k=0
∞
X x2k x2 x4 x6
cos x = (−1)k = 1− + − + ··· , |x| < ∞
(2k)! 2! 4! 6!
k=0
∞
1 X
= xk = 1 + x + x2 + x3 + x4 + · · · , |x| < 1
1−x
k=0
etc.

This is actually how computers calculate many functinos!

For example:
N
X xk
ex ≈
k!
k=0
for some large integer N such that the error is sufficiently small.

Example 1. Compute e to 6 digit accuracy.

Answer. We have
1 1 1 1
e = e1 = 1 + 1 + + + + + ···
2! 3! 4! 5!
And
1
= 0.5
2!
1
= 0.166667
3!
1
= 0.041667
4!
···
1
= 0.0000027 (can stop here)
9!
so
1 1 1 1 1
e ≈1+1+ + + + + · · · = 2.71828
2! 3! 4! 5! 9!
Error and convergence: Assume f (k) (x) (0 ≤ k ≤ n) are continuous functions. Call
n
X 1 (k)
fn (x) = f (c)(x − c)k
k!
k=0
16 CHAPTER 1. COMPUTER ARITHMETIC

the first n + 1 terms in Taylor series.

Then, the error is
∞
X 1 (k) 1
En+1 = f (x) − fn (x) = f (c)(x − c)k = f (n+1) (ξ)(x − c)n+1
k! (n + 1)!
k=n+1

where ξ is some point between x and c.

Observation: A Taylor series convergence rapidly if x is near c, and slowly (or not at
all) if x is far away from c.
Special case: n = 0, we have the “Mean-Value Theorem”:
If f is smooth on the interval (a, b), then

f (a) − f (b) = (b − a)f ′ (ξ), for some ξ in (a, b).

See Figure 1.3.

f (x)
6 ξ

- x
a b

Figure 1.3: Mean Value Theorem

This implies
f (b) − f (a)
f ′ (ξ) =
b−a
So, if a, b are close to each other, this can be used as an approximation for f ′ .
Given h > 0 very small, we have

f (x + h) − f (x)
f ′ (x) ≈
h
f (x) − f (x − h)
f ′ (x) ≈
h
f (x + h) − f (x − h)
f ′ (x) ≈
2h
1.5. REVIEW OF TAYLOR SERIES 17

Another way of writing Taylor Series:

∞ n
X 1 (k) X 1 (k)
f (x + h) = f (x)hk = f (x)hk + En+1
k! k!
k=0 k=0

where
∞
X 1 (k) 1
En+1 = f (x)hk = f (n+1) (ξ)hn+1
k! (n + 1)!
k=n+1

for some ξ that lies between x and x + h.

18 CHAPTER 1. COMPUTER ARITHMETIC
Chapter 2

Polynomial interpolation

2.1 Introduction
Problem description:
Given (n + 1) points, (xi , yi ), i = 0, 1, 2, · · · , n, with distinct xi such that

x0 < x1 < x2 < · · · < xn ,

find a polynomial of degree n

Pn (x) = a0 + a1 x + a2 x2 + · · · + an xn

such that it interpolates these points:

Pn (xi ) = yi , i = 0, 1, 2, · · · , n

Why should we do this?

• Find the values between the points;

• To approximate a (probably complicated) function by a polynomial

Example 1. Given table

xi 0 1 2/3
yi 1 0 0.5
Note that
yi = cos(π/2)xi
Interpolate with a polynomial with degree 2.
Answer. Let
P2 (x) = a0 + a1 x + a2 x2

19
20 CHAPTER 2. POLYNOMIAL INTERPOLATION

Then

x = 0, y = 1 : P2 (0) = a0 = 1
x = 1, y = 0 : P2 (1) = a0 + a1 + a2 = 0
x = 2/3, y = 0.5 : P2 (2/3) = a0 + (2/3)a1 + (4/9)a2 = 0.5

In matrix-vector form     
1 0 0 a0 1
1 1 1  a1  =  0 
1 23 4
9 a2 0.5
Easy to solve in Matlab (homework 1)

a0 = 1, a1 = −1/4, a2 = −3/4.

Then
1 3
P2 (x) = 1 − x − x2 .
4 4

Back to general case with (n + 1) points:

Pn (xi ) = yi , i = 0, 1, 2, · · · , n

We will have (n + 1) equations:

Pn (x0 ) = y0 : a0 + x0 a1 + x20 a2 + · · · + xn0 an = y0

Pn (x1 ) = y1 : a0 + x1 a1 + x21 a2 + · · · + xn1 an = y1
···
Pn (xn ) = yn : a0 + xn a1 + x2n a2 + · · · + xnn an = yn

In matrix-vector form
    
1 x0 x20 · · · xn0 a0 y0
 1 x1 x2 · · · n
x1   a1   y1 
   
 1 
 .. .. .. . . ..   ..  =  .. 
. . . . .  .   . 
2
1 xn xn · · · xnn an yn
or
X~a = ~y

where
x : (n + 1) × (n + 1) matrix, given, (van der Monde matrix)
~a : unknown vector, (n + 1)
~y : given vector, (n + 1)
2.2. LAGRANGE INTERPOLATION 21

Known: if xi ’s are distinct, then X is invertible, therefore ~a has a unique solution.

In Matlab, the command vander([x1 , x2 , · · · , xn ]) gives this matrix.
But: X has very large condition number, not effective to solve.
Other more efficient and elegant methods

• Lagrange polynmial

• Newton’s divided differences

2.2 Lagrange interpolation

Given points: x0 , x1 , · · · , xn
Define the cardinal functions: l0 , l1 , · · · , ln :∈ P n (polynomials of degree n)

1 , i=j
li (xj ) = δij = i = 0, 1, · · · , n
0 , i 6= j

The Lagrange form of the interpolation polynomial is

n
X
Pn (x) = li (x) · yi .
i=0

We check the interpolating property:

n
X
Pn (xj ) = li (xj ) · yi = yj , ∀j.
i=0

li (x) can be written as

n  
Y x − xj
li (x) =
xi − xj
j=0,j6=i
x − x0 x − x1 x − xi−1 x − xi+1 x − xn
= · ··· · ···
xi − x0 xi − x1 xi − xi−1 xi − xi+1 xi − xn

One can easily check that li (xi ) = 1 and li (xj ) = 0 for i 6= j, i.e., li (xj ) = δij .

Example 2. Consider again (same as in Example 1)

xi 0 1 2/3
yi 1 0 0.5

Write the Lagrange polynomial.

22 CHAPTER 2. POLYNOMIAL INTERPOLATION

Answer. We have
x − 2/3 x − 1 3 2
l0 (x) = · = (x − )(x − 1)
0 − 2/3 0 − 1 2 3
x−0 x−1 9
l1 (x) = · = − x(x − 1)
2/3 − 0 2/3 − 1 2
x − 0 x − 2/3 2
l2 (x) = · = 3x(x − )
1 − 0 1 − 2/3 3
so

P2 (x) = l0 (x)y0 + l1 (x)y1 + l2 (x)y2

3 2 9
= (x − )(x − 1) − x(x − 1)(0.5) + 0
2 3 2
3 2 1
= − x − x+1
4 4
This is the same as in Example 1.

Pros and cons of Lagrange polynomial:

• Elegant formula, (+)

• slow to compute, each li (x) is different, (-)

• Not flexible: if one changes a points xj , or add on an additional point xn+1 , one
must re-compute all li ’s. (-)

2.3 Newton’s divided differences

Given a data set
xi x0 x1 · · · xn
yi y0 y1 · · · yn

n=0 : P0 (x) = y0

n=1 : P1 (x) = P0 (x) + a1 (x − x0 )

Determine a1 : set in x = x1 , then P1 (x1 ) = P0 (x1 ) + a1 (x1 − x0 )
y1 − y0
so y1 = y0 + a1 (x1 − x0 ), we get a1 =
x1 − x0
n=2 : P2 (x) = P1 (x) + a2 (x − x0 )(x − x1 )
set in x = x2 : then y2 = P1 (x2 ) + a2 (x2 − x0 )(x2 − x1 )
y2 − P1 (x2 )
so a2 =
(x2 − x0 )(x2 − x1 )
General expression for an :
2.3. NEWTON’S DIVIDED DIFFERENCES 23

Assume that Pn−1 (x) interpolates (xi , yi ) for i = 0, 1, · · · , n − 1. We will find Pn (x) that
interpolates (xi , yi ) for i = 0, 1, · · · , n, in the form

Pn (x) = Pn−1 (x) + an (x − x0 )(x − x1 ) · · · (x − xn−1 )

where
yn − Pn−1 (xn )
an =
(xn − x0 )(xn − x1 ) · · · (xn − xn−1 )
Check by yourself that such polynomial does the interpolating job!

Newtons’ form:

pn (x) = a0 + a1 (x − x0 ) + a2 (x − x0 )(x − x1 ) + · · ·
+an (x − x0 )(x − x1 ) · · · (x − xn−1 )

The constants ai ’s are called divided difference, written as

a0 = f [x0 ], a1 = f [x0 , x1 ] · · · ai = f [x0 , x1 , · · · , xi ]

And we have (see textbook for proof)

f [x1 , x1 , · · · , xk ] − f [x0 , x1 , · · · , xk−1 ]
f [x0 , x1 , · · · , xk ] =
xk − x0

Compute f ’s through the table:

x0 f [x0 ] = y0
f [x1 ]−f [x0 ]
x1 f [x1 ] = y1 f [x0 , x1 ] = x1 −x0

f [x2 ]−f [x1 ]

x2 f [x2 ] = y2 f [x1 , x2 ] = x2 −x1 f [x0 , x1 , x2 ] = · · ·
.. .. .. .. ..
. . . . .
f [xn ]−f [xn−1 ]
xn f [xn ] = yn f [xn−1 , xn ] = xn −xn−1 f [xn−2 , xn−1 , xn ] = · · · ... f [x0 , x1 , · · · , xn ]

Example : Use Newton’s divided difference to write the polynomial that interpolates
the data
xi 0 1 2/3 1/3
yi 1 0 1/2 0.866

Answer. Set up the trianglar table for computation

0 1
1 0 -1
2/3 0.5 -1.5 -0.75
1/3 0.8660 -1.0981 -0.6029 0.4413
24 CHAPTER 2. POLYNOMIAL INTERPOLATION

So
P3 (x) = 1 + -1 x + -0.75 x(x − 1) + 0.4413 x(x − 1)(x − 2/3).

Flexibility of Newton’s form: easy to add additional points to interpolate.

Nested form:

Pn (x) = a0 + a1 (x − x0 ) + +a2 (x − x0 )(x − x1 ) + · · ·

+an (x − x0 )(x − x1 ) · · · (x − xn−1 )
= a0 + (x − x0 ) (a1 + (x − x1 )(a2 + (x − x2 )(a3 + · · · + an (x − xn−1 ))))

Effective to compute in a program:

• p = an

• for k = n − 1, n − 1, · · · , 0

– p = p(x − xk ) + ak

• end

Some theoretical parts:

Existence and Uniqueness theorem for polynomial interpolation:
Given (xi , yi )ni=0 , with xi ’s distinct. There exists one and only polynomial Pn (x) of
degree ≤ n such that
Pn (xi ) = yi , i = 0, 1, · · · , n

Proof. : Existence: OK from construction

Uniqueness: Assume we have two polynomials, call them p(x) and q(x), of degree ≤ n,
both interpolate the data, i.e.,

p(xi ) = yi , q(xi ) = yi , i = 0, 1, · · · , n

Now, let g(x) = p(x) − q(x), which will be a polynomial of degree ≤ n. Furthermore,
we have
g(xi ) = p(xi ) − q(xi ) = yi − yi = 0, i = 0, 1, · · · , n
So g(x) has n + 1 zeros. We must have g(x) ≡ 0, therefore p(x) ≡ q(x).
2.4. ERRORS IN POLYNOMIAL INTERPOLATION 25

2.4 Errors in Polynomial Interpolation

Given a function f (x), and a ≤ x ≤ b, a set of distinct points xi , i = 0, 1, · · · , n, and
xi ∈ [a, b]. Let Pn (x) be a polynomial of degree ≤ n that interpolates f (x) at xi , i.e.,

Pn (xi ) = f (xi ), i = 0, 1, · · · , n

Define the error

e(x) = f (x) − Pn (x)

Theorem There exists a point ξ ∈ [a, b], such that

n
1 Y
e(x) = f (n+1) (ξ) (x − xi ), for all x ∈ [a, b].
(n + 1)!
i=0

Proof. . If f ∈ Pn , then f (x) = Pn (x), trivial.

Now assume f ∈
/ Pn . For x = xi , we have e(xi ) = f (xi ) − Pn (xi ) = 0, OK.
Now fix an a such that a 6= xi for any i. We define
n
Y
W (x) = (x − xi ) ∈ Pn+1
i=0

and a constant
f (a) − Pn (a)
c= ,
W (a)
and another function
ϕ(x) = f (x) − Pn (x) − cW (x).
Now we find all the zeros for this function ϕ:

ϕ(xi ) = f (xi ) − Pn (xi ) − cW (xi ) = 0, i = 0, 1, · · · , n

and
ϕ(a) = f (a) − Pn (a) − cW (a) = 0
So, ϕ has at least (n + 2) zeros.
Here goes our deduction:
ϕ(x) has at least n+2 zeros.
ϕ′ (x) has at least n+1 zeros.
ϕ′′ (x) has at least n zeros.
..
.
ϕ(n+1) (x) has at least 1 zero. Call it ξ.
26 CHAPTER 2. POLYNOMIAL INTERPOLATION

So we have
ϕ(n+1) (ξ) = f (n+1) (ξ) − 0 − cW (n+1) (ξ) = 0.
Use
W (n+1) = (n + 1)!
we get
f (a) − Pn (a)
f (n+1) (ξ) = cW (n+1) (ξ) = (n + 1)!.
W (a)
Change a into x, we get
n
1 1 Y
e(x) = f (x) − Pn (x) = f (n+1) (ξ)W (x) = f (n+1) (ξ) (x − xi ).
(n + 1)! (n + 1)!
i=0

Example n = 1, x0 = a, x1 = b, b > a.
We have an upper bound for the error, for x ∈ [a, b],
1  ′′  1 (b − a)2 1
|e(x)| = f (ξ) · |(x − a)(x − b)| ≤ f ′′ ∞ = f ′′ ∞ (b − a)2 .
2 2 4 8

Observation: Different distribution of nodes xi would give different errors.

Uniform nodes: equally distribute the space. Consider an interval [a, b], and we
distribute n + 1 nodes uniformly as
b−a
xi = a + ih, h= , i = 0, 1, · · · , n.
n
One can show that
n
Y 1 n+1
|x − xi | ≤ h · n!
4
i=0
(Try to prove it!)
This gives the error estimate
1   Mn+1 n+1
 (n+1)  n+1
|e(x)| ≤ f (x) h ≤ h
4(n + 1) 4(n + 1)
where  
Mn+1 = max f (n+1) (x) .
 
x∈[a,b]

Example Consider interpolating f (x) = sin(πx) with polynomial on the interval [−1, 1]
with uniform nodes. Give an upper bound for error, and show how it is related with
total number of nodes with some numerical simulations.
2.4. ERRORS IN POLYNOMIAL INTERPOLATION 27

Answer. We have  
 (n+1) 
f (x) ≤ π n+1
so the upper bound for error is
 n+1
π n+1 2
|e(x)| = |f (x) − Pn (x)| ≤ .
4(n + 1) n

Below is a table of errors from simulations with various n.

n error bound measured error

4 4.8 × 10−1 1.8 × 10−1
8 3.2 × 10−3 1.2 × 10−3
16 1.8 × 10−9 6.6 × 10−10

Problem with uniform nodes: peak of errors near the boundaries. See plots.

Chebychev nodes: equally distributing the error.

Type I: including the end points.
For interval [−1, 1] : x̄i = cos( ni π), i = 0, 1, · · · , n

For interval [a, b] : x̄i = 12 (a + b) + 12 (b − a) cos( ni π), i = 0, 1, · · · , n

With this choice of nodes, one can show that
n
Y n
Y
|x − x̄k | = 2−n ≤ |x − xk |
k=0 k=0

where xk is any other choice of nodes.

This gives the error bound:
1  
 (n+1)  −n
|e(x)| ≤ f (x) 2 .
(n + 1)!

Example Consider the same example with uniform nodes, f (x) = sin πx. With Cheby-
shev nodes, we have
1
|e(x)| ≤ π n+1 2−n .
(n + 1)!
The corresponding table for errors:

n error bound measured error

4 1.6 × 10−1 1.15 × 10−1
8 3.2 × 10−4 2.6 × 10−4
16 1.2 × 10−11 1.1 × 10−11
28 CHAPTER 2. POLYNOMIAL INTERPOLATION

The errors are much smaller!

Type II: Chebyshev nodes can be chosen strictly inside the interval [a, b]:

1 1 2i + 1
x̄i = (a + b) + (b − a) cos( π), i = 0, 1, · · · , n
2 2 2n + 2

See slides for examples.

Theorem If Pn (x) interpolates f (x) at xi ∈ [a, b], i = 0, 1, · · · , n, then

n
Y
f (x) − Pn (x) = f [x0 , x1 , · · · , xn , x] · (x − xi ), ∀x 6= xi .
i−0

Proof. Let a 6= xi , let q(x) be a polynomial that interpolates f (x) at x0 , x1 , · · · , xn , a.

Newton’s form gives
n
Y
q(x) = Pn (x) + f [x0 , x1 , · · · , xn , a] (x − xi ).
i=0

Since q(a) = f (a), we get

n
Y
f (a) = q(a) = Pn (a) + f [x0 , x1 , · · · , xn , a] (a − xi ).
i=0

Switching a to x, we prove the Theorem.

As a consequence, we have:
1 (n)
f [x0 , x1 , · · · , xn ] = f (ξ), ξ ∈ [a, b].
n!

Proof. Let Pn−1 (x) interpolate f (x) at x0 , · · · , xn−1 . The error formula gives
n
1 (n) Y
f (xn ) − Pn−1 (xn ) = f (ξ) (xn − xi ), ξ ∈ (a, b).
n!
i=0

From above we know

n
Y
f (xn ) − Pn−1 (xn ) = f [x0 , · · · , xn ] (xn − xi )
i=0
2.5. NUMERICAL DIFFERENTIATIONS 29

Comparing the rhs of these two equation, we get the result.

Observation: Newton’s divided differences are related to derivatives.

n=1 : f [x0 , x1 ] = f ′ (ξ), ξ ∈ (x0 , x1 )

n=2 : f [x0 , x1 , x2 ] = f ′′ (ξ). Let x0 = x − h, x1 = x, x2 = x + h, then

1 1
f [x0 , x1 , x2 ] = 2
[f (x + h) − 2f (x) + f (x + h)] = f ′′ (ξ), ξ ∈ [x − h, x + h].
2h 2

2.5 Numerical differentiations

Finite difference:

1
(1) f ′ (x) ≈ (f (x + h) − f (x))
h
1
(2) f ′ (x) ≈ (f (x) − f (x − h))
h
1
(3) f ′ (x) ≈ (f (x + h) − f (x − h)) (central difference)
2h
1
f ′′ (x) ≈ (f (x + h) − 2f (x) + f (x − h))
h2

f (x)
6 (1) (3)

f ′ (x)

(2)

- x
x−h x x+h

Figure 2.1: Finite differences to approximate derivatives

Truncation erros in Taylor expansion

30 CHAPTER 2. POLYNOMIAL INTERPOLATION

1 1 3 ′′′
f (x + h) = f (x) + hf ′ (x) + h2 f ′′ (x) + h f (x) + O(h4 )
2 6
1 2 ′′ 1 3 ′′′
′
f (x − h) = f (x) − hf (x) + h f (x) − h f (x) + O(h4 )
2 6
Then,

f (x + h) − f (x) 1
= f ′ (x) + hf ′′ (x) + O(h2 ) = f ′ (x) + O(h), (1st order)
h 2
similarly

f (x) − f (x − h) 1
= f ′ (x) − hf ′′ (x) + O(h2 ) = f ′ (x) + O(h), (1st order)
h 2
and
f (x + h) − f (x − h) 1
= f ′ (x) − h2 f ′′′ (x) + O(h2 ) = f ′ (x) + O(h2 ), (2nd order)
2h 6

finally

f (x + h) − 2f (x) + f (x − h) 1
2
= f ′′ (x)+ h2 f (4) (x)+O(h4 ) = f ′′ (x)+O(h2 ), (2nd order)
h 12

Richardson Extrapolation : will be discussed later, in numerical integration, with

Romberg’s algorithm
Chapter 3

Piece-wise polynomial
interpolation. Splines

3.1 Introduction
Usage:

• visualization of discrete data

• graphic design –VW car design

Requirement:

• interpolation

• certain degree of smoothness

Disadvantages of polynomial interpolation Pn (x)

• n-time differentiable. We do not need such high smoothness;

• big error in certain intervals (esp. near the ends);

• no convergence result;

• Heavy to compute for large n

Suggestion: use piecewise polynomial interpolation.

Problem setting : Given a set of data

x t0 t1 · · · tn
y y0 y1 · · · yn

31
32 CHAPTER 3. SPLINES

Find a function S(x) which interpolates the points (ti , yi )ni=0 .

The set t0 , t1 , · · · , tn are called knots.
S(x) consists of piecewise polynomials


 S0 (x), t0 ≤ x ≤ t1

 S1 (x), t1 ≤ x ≤ t2
S(x)=˙ ..


 .

Sn−1 (x), tn−1 ≤ x ≤ tn

S(x) is called a spline of degree n, if

• Si (x) is a polynomial of degree n;

• S(x) is (n − 1) times continuous differentiable, i.e., for i = 1, 2, · · · , n − 1 we have
Si−1 (ti ) = Si (ti ),
′
Si−1 (ti ) = Si′ (ti ),
..
.
(n−1) (n−1)
Si−1 (ti ) = Si (ti ),

Commonly used ones:

• n = 1: linear splines (simplest)

• n = 1: quadratic splines
• n = 3: cubic splines (most used)

3.2 First degree and second degree splines

Linear splines: n = 1. Piecewise linear interpolation, i.e., straight line between 2
neighboring points. See Figure 3.1.
So
Si (x) = ai + bi x, i = 0, 1, ·, n − 1

Requirements:
S0 (t0 ) = y0
Si−1 (ti ) = Si (ti ) = yi , i = 1, 2, · · · , n − 1
Sn−1 (tn ) = yn .

Easy to find: write the equation for a line through two points: (ti , yi ) and (ti+1 , yi+1 ),
yi+1 − yi
Si (x) = yi + (x − ti ), i = 0, 1, · · · , n − 1.
ti+1 − ti
3.3. NATURAL CUBIC SPLINES 33

S(x) 6 y1
×

y2
× y3
y0 ×
×

-
x
t0 t1 t2 t3

Figure 3.1: Linear splines

Accuracy Theorem for linear splines: Assume t0 < t1 < t2 < · · · < tn , and let

h = max(ti+1 − ti )
i

Let f (x) be a given function, and let S(x) be a linear spline that interpolates f (x) s.t.

S(ti ) = f (ti ), i = 0, 1, · · · , n

We have the following, for x ∈ [t0 , tn ],

(1) If f ′ exists and is continuous, then

1  
|f (x) − S(x)| ≤ h max f ′ (x) .
2 x

(2) If f ′′ exits and is continuous, then

1  
|f (x) − S(x)| ≤ h2 max f ′′ (x) .
8 x

Quadratics splines. read the textbook if you want.

3.3 Natural cubic splines

Given t0 < t1 < · · · < tn , we define the cubic spline S(x) = Si (x) for ti ≤ x ≤ ti+1 . We
′′ (t ) = 0,
require that S, S ′ , S ′′ are all continuous. If in addition we require S0′′ (t0 ) = Sn−1 n
then it is called natural cubic spline.
34 CHAPTER 3. SPLINES

Write
Si (x) = ai x3 + bi x2 + ci x + di , i = 0, 1, · · · , n − 1
Total number of unknowns= 4 · n.
Equations we have

equation number 



(1) Si (ti ) = yi , i = 0, 1, · · · , n − 1 n 





(2) Si (ti+1 ) = yi+1 , i = 0, 1, · · · , n − 1 n 



(3) Si′ (ti+1 ) = ′ (t ),
Si+1 i i = 0, 1, · · · , n − 2 n−1 total = 4n.


(4) Si′′ (ti+1 ) = Si+1
′′ (t ), 
i i = 0, 1, · · · , n − 2 n−1  



(5) S0′′ (t0 ) = 0, 1 





′′ (t ) = 0,
(6) Sn−1 1.
n

How to compute Si (x)? We know:

Si : polynomial of degree 3
Si′ : polynomial of degree 2
Si′′ : polynomial of degree 1
procedure:

• Start with Si′′ (x), they are all linear, one can use Lagrange form,

• Integrate Si′′ (x) twice to get Si (x), you will get 2 integration constant

• Determine these constants by (2) and (1). Various tricks on the way...

Details: Define zi as

zi = S ′′ (ti ), i = 1, 2, · · · , n − 1, z0 = zn = 0

NB! These zi ’s are our unknowns.

Introduce the notation hi = ti+1 − ti .
Lagrange form
zi+1 zi
Si′′ (x) = (x − ti ) − (x − ti+1 ).
hi hi
Then
zi+1 zi
Si′ (x) = (x − ti )2 − (x − ti+1 )2 + Ci − Di
2hi 2hi
zi+1 zi
Si (x) = (x − ti )3 − (x − ti+1 )3 + Ci (x − ti ) − Di (x − ti+1 ).
6hi 6hi

(You can check by yourself that these Si , Si′ are correct.)

3.3. NATURAL CUBIC SPLINES 35

Interpolating properties:
(1). Si (ti ) = yi gives
zi 1 yi hi
yi = − (−hi )3 − Di (−hi ) = zi h2i + Di hi ⇒ Di = − zi
6hi 6 hi 6

(2). Si (ti+1 ) = yi+1 gives

zi+1 3 yi+1 hi
yi+1 = h + Ci hi , ⇒ Ci = − zi+1 .
6hi i hi 6

We see that, once zi ’s are known, then (Ci , Di )’s are known, and so Si , Si′ are known.
 
zi+1 3 zi 3 yi+1 hi
Si (x) = (x − ti ) − (x − ti+1 ) + − zi+1 (x − ti )
6hi 6hi hi 6
 
yi hi
− − zi (x − ti+1 ).
hi 6
z i+1 zi yi+1 − yi zi+1 − zi
Si′ (x) = (x − ti )2 − (x − ti+1 )2 − hi .
2hi 2hi hi 6

How to compute zi ’s? Last condition that’s not used yet: continuity of S ′ (x), i.e.,
′
Si−1 (ti ) = Si′ (ti ), i = 1, 2, · · · , n − 1
We have
zi yi+1 − yi zi+1 − zi
Si′ (ti ) = − (−hi )2 + − hi
2hi hi 6
| {z }
bi
1 1
= − hi zi+1 − hi zi + bi
6 3
′ 1 1
Si−1 (ti ) = zi−1 hi−1 + zi hi−1 + bi−1
6 3
Set them equal to each other, we get

hi−1 zi−1 + 2(hi−1 + hi )zi + hi zi+1 = 6(bi − bi−1 ), i = 1, 2, · · · , n − 1
z0 = zn = 0.

In matrix-vector form:
H · ~z = ~b
where
 
2(h0 + h1 ) h

 h1 2(h1 + h2 ) h2 


 h2 2(h2 + h3 ) h3 

H= .. .. .. 

 . . . 

 hn−3 2(hn−3 + hn−2 ) hn−2 
hn−2 2(hn−2 + hn−1 )
36 CHAPTER 3. SPLINES

and    
z1 6(b1 − b0 )

 z2


 6(b2 − b1 )


 z3  6(b3 − b2 )


~z = 

,
.. ~b = 
 
...

 .





 .
zn−2  6(bn−2 − bn−3 )
zn−1 6(bn−1 − bn−2 )

Here, H is a tri-diagonal matrix, symmetric, and diagonal dominant

2 |hi−1 + hi | > |hi | + |hi−1 |

which implies unique solution for ~z.

See slides for Matlab codes and solution graphs.

Theorem on smoothness of cubic splines. If S is the natural cubic spline function

that interpolates a twice-continuously differentiable function f at knots

a = t0 < t1 < · · · < tn = b

then Z Z
b 2 b 2
′′
S (x) dx ≤ f ′′ (x) dx.
a a

R
Note that (f ′′ )2 is related to the curvature of f .
Cubic spline gives the least curvature, ⇒ most smooth, so best choice.

Proof. Let
g(x) = f (x) − cS(x)
Then
g(ti ) = 0, i = 0, 1, · · · , n
and f ′′ = S ′′ + g ′′ , so
(f ′′ )2 = (S ′′ )2 + (g′′ )2 + 2S ′′ g′′
Z b Z b Z b Z b
′′ 2 ′′ 2 ′′ 2
⇒ (f ) dx = (S ) dx + (g ) dx + 2S ′′ g′′ dx
a a a a

Claim that Z b
S ′′ g′′ dx = 0
a
then this would imply
Z b Z b
(f ′′ )2 dx ≥ (S ′′ )2 dx
a a
3.3. NATURAL CUBIC SPLINES 37

and we are done.

Proof of the claim: Using integration-by-parts,
Z b b Z b
S ′′ g′′ dx = S ′′ g′  − S ′′′ g′ dx

a a a

Since g(a) = g(b) = 0, so the first term is 0. For the second term, since S ′′′ is piecewise
constant. Call
ci = S ′′′ (x), for x ∈ [ti , ti+1 ].
Then
Z b n−1
X Z ti+1 n−1
X
′′′ ′ ′
S g dx = ci g (x) dx = ci [g(ti+1 ) − g(ti )] = 0,
a i=0 ti i=0

(b/c g(ti ) = 0).

38 CHAPTER 3. SPLINES
Chapter 4

Numerical integration

4.1 Introduction

Problem: Given a function f (x) on interval [a, b], find an approximation to the inte-
gral
Z b
I(f ) = f (x) dx
a

Main idea:

• Cut up [a, b] into smaller sub-intervals

• In each sub-interval, find a polynomial pi (x) ≈ f (x)

• Integrate pi (x) on each sub-interval, and sum up

4.2 Trapezoid rule

The grid: cut up [a, b] into n sub-intervals:

x0 = a, xi < xi+1 , xn = b

On interval [xi , xi+1 ], approximate f (x) by a linear polynomial

We use
Z xi+1 Z xi+1
1
f (x) dx ≈ pi (x) dx = (f (xi+1 ) + f (xi )) (xi+1 − xi )
xi xi 2

39
40 CHAPTER 4. NUMERICAL INTEGRATION

pi (x)
f (x)

x0 = a xi xi+1 xn = b

Figure 4.1: Trapezoid rule: straight line approximation in each sub-interval.

Summing up all the sub-intervals

Z b n−1
X Z xi+1
f (x) dx = f (x) dx
a i=0 xi
n−1
X Z xi+1
≈ pi (x) dx
i=0 xi
n−1
X 1
= (f (xi+1 ) + f (xi )) (xi+1 − xi )
2
i=0

b−a
Uniform grid: h = n , xi+1 − xi = h,

Z b n−1
X h
f (x) dx = (f (xi+1 ) + f (xi ))
a 2
i=0
"n−1
#
1 X 1
= h f (x0 ) + f (xi ) + f (xn )
2 2
i=1
| {z }
T (f ; h)

so we can write Z b
f (x) ≈ T (f ; h)
a

Error estimates.
n−1
X Z xi+1  
ET (f ; h) = I(f ) − T (f ; h) = f (x) − pi (x) dx
i=0 xi
4.2. TRAPEZOID RULE 41

Known from interpolation:

1
f (x) − pi (x) = f ′′ (ξi )(x − xi )(x − xi+1 )
2
Basic error: error on each sub-interval:
Z xi+1
1 ′′ 1
ET,i (f ; h) = f (ξi ) (x − xi )(x − xi+1 ) dx = − h3 f ′′ (ξi ).
2 xi 12

Total error:
n−1 n−1
"n−1 #
X X 1 3 ′′ 1 3 X ′′ 1 b−a
ET (f ; h) = ET,i (f ; h) = − h f (ξi ) = − h f (ξi ) · ·
12 12 n | {z
h }
i=0 i=0
| i=0 {z }
= f ′′ (ξ) =n

Total error is
b − a 2 ′′
ET (f ; h) = h f (ξ), ξ ∈ (a, b).
12
Error bound
b−a 2  
ET (f ; h) ≤ h max f ′′ (x) .
12 x∈(a,b)

Example Consider function f (x) = ex , and the integral

Z 2
I(f ) = ex dx
0

Require error ≤ 0.5 × 10−4 . How many points should be used in the Trapezoid rule?
Answer. We have

f ′ (x) = ex , f ′′ (x) = ex , a = 0, b=2

so  
max f ′′ (x) = e2 .
x∈(a,b)

By error bound, it is sufficient to require

1
|ET (f ; h)| ≤ h2 e2 ≤ 0.5 × 10−4
6
⇒ h2 ≤ 0.5 × 10−4 × 6 × e−2 ≈ 4.06 × 10−5
2 p
⇒ = h ≤ 4.06 × 10−5 = 0.0064
n
2
⇒ n≥ ≈ 313.8
0.0064

We need at least 314 points.

42 CHAPTER 4. NUMERICAL INTEGRATION

4.3 Simpson’s rule

Cut up [a, b] into 2n equal sub-intervals

b−a
x0 = a, x2n = b, h= , xi+1 − xi = h
2n

Consider the interval [x2i , x2i+2 ]. Find a 2nd order polynomial that interpolates f (x) at
the points

x2i , x2i+1 , x2i+2

pi (x)
f (x)

x2i x2i+1 x2i+2

Figure 4.2: Simpson’s rule: quadratic polynomial approximation (thick line) in each
sub-interval.

Lagrange form gives

(x − x2i+1 )(x − x2i+2 ) (x − x2i )(x − x2i+2 )

pi (x) = f (x2i ) + f (x2i+1 )
(x2i − x2i+1 )(x2i − x2i+2 ) (x2i+1 − x2i )(x2i+1 − x2i+2 )
(x − x2i )(x − x2i+1 )
+f (x2i+2 )
(x2i+2 − x2i )(x2i+2 − x2i+1 )
1 1
= f (x2i )(x − x2i+1 )(x − x2i+2 ) − 2 f (x2i+1 )(x − x2i )(x − x2i+2 )
2h2 h
1
+ 2 f (x2i+2 )(x − x2i )(x − x2i+1 )
2h
4.3. SIMPSON’S RULE 43

Then
Z x2i+2 Z x2i+2
f (x) dx ≈ pi (x) dx
x2i x2i
Z x2i+2
1
= f (x2i )f (x2i ) (x − x2i+1 )(x − x2i+2 ) dx
2h2 x2i
| {z }
2 3
h
Z 3
x2i+2
1
− f (x2i+1 ) (x − x2i )(x − x2i+2 ) dx
h2 x2i
| {z }
4
− h3
Z 3
x2i+2
1
+ 2 f (x2i+2 ) (x − x2i )(x − x2i+1 ) dx
2h x2i
| {z }
2 3
h
3
h
= [f (x2i ) + 4f (x2i+1 ) + f (x2i+2 )]
3
Putting together
Z b
f (x) dx ≈ S(f ; h)
a
n−1
X Z x2i+2
= pi (x) dx
i=0 x2i
n−1
h X
= [f (x2i ) + 4f (x2i+1 ) + f (x2i+2 )]
3
i=0

1 4 1
=2
1 4 1

x2i−2 x2i−1 x2i x2i+1 x2i+2

Figure 4.3: Simpson’s rule: adding the constants in each node.

See Figure 4.3 for the counting of coefficients on each node. Wee see that for x0 , x2n we
get 1, and for odd indices we have 4, and for all remaining even indices we get 2.
The algorithm looks like:
" n n−1
#
h X X
S(f ; h) = f (x0 ) + 4 f (x2i−1 ) + 2 f (x2i ) + f (x2n )
3
i=1 i=1
44 CHAPTER 4. NUMERICAL INTEGRATION

Error estimate. basic error is

1 5 (4)
− h f (ξi ), ξi ∈ (x2i , x2i+2 )
90
so
n−1
1 X 1 b−a b − a 4 (4)
ES (f ; h) = I(f ) − S(f ; h) = − h5 f (4) (ξi ) · =− h f (ξ), ξ(a, b)
90 n 2h 180
i=0

Error bound  
b−a 4
h max f (4) (x) .
 
|ES (f ; h)| ≤
180 x∈(a,b)

Example With f (x) = ex in [0, 2], now use Simpson’s rule, to achieve an error ≤
0.5 × 10−4 , how many points must one take?
Answer. We have
2 4 2
|ES (f ; h)| ≤h e ≤ 0.5 × 10−4
180
⇒ h4 ≤ 0.5−4 × 180/e2 = 1.218 × 10−3
⇒ h ≤ 0.18682
b−a
⇒ n= = 5.3 ≈ 6
2h
We need at least 2n + 1 = 13 points.
Note: This is much fewer points than using Trapezoid Rule.

4.4 Recursive trapezoid rule

These are called composite schemes.

Divide [a, b] into 2n equal sub-intervals.

b−a 1
hn = , hn+1 = h
2n 2

So
" n −1
2X
#
1 1
T (f ; hn ) = hn · f (a) + f (b) + f (a + ihn )
2 2
i=1
 
2n+1 −1
1 1 X
T (f ; hn+1 ) = hn+1 ·  f (a) + f (b) + f (a + ihn+1 )
2 2
i=1
4.5. ROMBERG ALGORITHM 45
n=1
n=2
n=3
n=4
n=5
n=6
n=7

Figure 4.4: Recursive division of intervals, first few levels

We can re-arrange the terms in T (f ; hn+1 ):

 
2Xn −1 n −1
2X
hn  1 1
T (f ; hn+1 ) = f (a) + f (b) + f (a + ihn ) + f (a + (2j + 1)hn+1 )
2 2 2
i=1 j=1
2n −1
1 X
= T (f ; hn ) + hn+1 f (a + (2j + 1)hn+1 )
2
j=1

Advantage: One case keep the computation for a level n. If this turns out to be not
accurate enough, then add one more level to get better approximation. ⇒ flexibility.

4.5 Romberg Algorithm

If f (n) exists and is bounded, then we have the Euler MacLaurin’s formula for error

E(f ; h) = I(f ) − T (f ; h) = a2 h2 + a4 h4 + a6 h6 + · · · + an hn
h h h h h h
E(f ; ) = I(f ) − T (f ; ) = a2 ( )2 + a4 ( )4 + a6 ( )6 + · · · + an ( )n
2 2 2 2 2 2
Here an depends on the derivatives f (n) .
We have

(1) I(f ) = T (f ; h) + a2 h2 + a4 h4 + a6 h6 + · · ·
h h h h h
(2) I(f ) = T (f ; ) + a2 ( )2 + a4 ( )4 + a6 ( )6 + · · · + an ( )n
2 2 2 2 2
The goal is to use the 2 approximations T (f ; h) and T (f ; h2 ) to get one that’s more
accurate, i.e., we wish to cancel the leading error term, the one with h2 .
Multiply (2) by 4 and subtract (1), gives

3 · I(f ) = 4 · T (f ; h/2) − T (f ; h) + a′4 h4 + a′6 h6 + · · ·

4 1
⇒ I(f ) = T (f ; h/2) − T (f ; h) +ã4 h4 + ã6 h6 + · · ·
3
| {z 3 }
U (h)
46 CHAPTER 4. NUMERICAL INTEGRATION

R(h) is of 4-th order accuracy! Better than T (f ; h). We now write:

T (f ; h/2) − T (f ; h)
U (h) = T (f ; h/2) +
22 − 1
This idea is called the Richardson extrapolation.
Take one more step:

(3) I(f ) = U (h) + ã4 h4 + ã6 h6 + · · ·

(4) I(f ) = U (h/2) + ã4 (h/2)4 + ã6 (h/2)6 + · · ·

To cancel the term with h4 : (4) × 24 − (3)

(24 − 1)I(f ) = 24 U (h/2) − U (h) + ã′6 h6 + · · ·

Let
24 U (h/2) − U (h) U (h/2) − U (h)
V (h) = 4
= U (h/2) + .
2 −1 24 − 1
Then
I(f ) = V (h) + ã′6 h6 + · · ·
So V (h) is even better than U (h).
One can keep doing this several layers, until desired accuracy is reached.
This gives the Romberg Algorithm: Set H = b − a, define:
H
R(0, 0) = T (f ; H) = (f (a) + f (b))
2
R(1, 0) = T (f ; H/2)
R(2, 0) = T (f ; H/(22 ))
..
.
R(n, 0) = T (f ; H/(2n ))

Here R(n, 0)’s are computed by the recursive trapezoid formula.

Romberg triangle: See Figure 4.5.
The entry R(n, m) is computed as
R(n, m − 1) − R(n − 1, m − 1)
R(n, m) = R(n, m − 1) +
22m−1
Accuracy:
H
I(f ) = R(n, m) + O(h2(m+1) ), h= .
2n
Algorithm can be done either column-by-column or row-by-row.
Here we give some pseudo-code, using column-by-column.
4.6. ADAPTIVE SIMPSON’S QUADRATURE SCHEME 47
R(0, 0)
s
-
R(1, 0) R(1, 1)
s
- s
-
R(2, 0) R(2, 1) R(2, 2)
s
- s
- s
-
R(3, 0) R(3, 1) R(3, 2)

R(n, 0) R(n, 1) R(n, 3) R(n, n)

Figure 4.5: Romberg triangle

R =romberg(f, a, b, n)

R = n × n matrix
h = b − a; R(1, 1) = [f (a) + f (b)] ∗ h/2;
for i = 1 to n − 1 do %1st column recursive trapezoid
R(i + 1, 1) = R(i, 1)/2;
h = h/2;
for k = 1 to 2i−1 do
R(i + 1, 1) = R(i + 1, 1) + h ∗ f (a + (2k − 1)h)
end
end
for j = 2 to n do %2 to $n$ column
for i = j to n do
1
R(i, j) = R(i, j − 1) + 4j −1
[R(i, j − 1) − R(i − 1, j − 1)]
end
end

4.6 Adaptive Simpson’s quadrature scheme

Same idea can be adapted to Simpson’s rule instead of trapezoid rule.
b−a
For interval [a, b], h = 2 ,
 
b−a a+b
S1 [a, b] = f (a) + 4f ( ) + f (b)
6 2

Error form:
1 5 (4)
E1 [a, b] = − h f (ξ), ξ(a, b)
90
48 CHAPTER 4. NUMERICAL INTEGRATION

Then
I(f ) = S1 [a, b] + E1 [a, b]
a+b
Divide [a, b] up in the middle, let c = 2 .

I(f )[a, b] = I(f )[a, c] + I(f )[c, b]

= S1 [a, c] + E1 [a, c] + S1 [c, b] + E1 [c, b]
= S2 [a, b] + E2 [a, b]

where

S2 [a, b] = S1 [a, c] + S1 [c, b]

1 h i
E2 [a, b] = E1 [a, c] + E1 [c, b] = − (h/2)5 f (4) (ξ1 ) + f (4) (ξ2 )
90

Assume f (4) does NOT change much, then E1 [a, c] ≈ E1 [c, b], and

1 1
E2 [a, b] ≈ 2E1 [a, c] = 2 E1 [a, b] = 4 E1 [a, b]
25 2
This gives

S2 [a, b] − S1 [a, b] = (I − E2 [a, b]) − (I − E1 [a, b]) = E1 − E2 = 24 E2 − E2 = 15E2

This means, we can compute the error E2 :

1
E2 = (S2 − S1 )
15
If we wish to have |E2 | ≤ ε, we only need to require

S2 − S1
≤ε
24 − 1
This gives the idea of an adaptive recursive formula:

(A) I = S1 + E1
1
(B) I = S2 + E2 = S2 + E1
24
(B) ∗ 24 − (A) gives
(24 − 1)I = 24 S2 − S1
24 S2 − S1 S2 − S1
⇒ I= = S2 +
24 − 1 15

Note that this gives the best approximation when f (4) ≈ const.
Pseudocode: f : function, [a, b] interval, ε: tolerance for error
4.7. GAUSSIAN QUADRATURE FORMULAS 49

answer=simpson(f, a, b, ε)
compute S1 and S2
If |S2 − S1 | < 15ε
answer= S2 + (S2 − S1 )/15;
else
c = (a + b)/2;
Lans=simpson(f, a, c, ε/2);
Rans=simpson(f, c, b, ε/2);
answer=Lans+Rans;
end

In Matlab, one can use quad to compute numerical integration. Try help quad, it will
give you info on it. One can call the program by using:
a=quad(’fun’,a,b,tol)
It uses adaptive Simpson’s formula.
See also quad8, higher order method.

4.7 Gaussian quadrature formulas

We seek numerical integration formulas of the form
Z b
f (x) dx ≈ A0 f (x0 ) + A1 f (x1 ) + · · · + An f (xn ),
a
with the weights Ai , (i = 0, 1, · · · , n) and the nodes
xi ∈ (a, b), i = 0, 1, · · · , n

How to find nodes and weights?

Nodes xi : are roots of Legendre polynomials qn+1 (x). These polynomials satisfies
Z b
xk qn+1 (x) dx = 0, (0 ≤ k ≤ n)
a
Examples for n ≤ 3, for interval [−1, 1]
q0 (x) = 1
q1 (x) = x
3 2 1
q2 (x) = x −
2 2
5 3 3
q3 (x) = x − x
2 2
50 CHAPTER 4. NUMERICAL INTEGRATION

The roots are

q1 : 0
√
q2 : ±1/ 3
p
q3 : 0, ± 3/5

For general interval [a, b], use the transformation:

2x − (a + b) 1 1
t= , x = (b − a)t + (a + b)
b−a 2 2
so for −1 ≤ t ≤ 1 we have a ≤ x ≤ b.

Weights Ai : Recall li (x), the Cardinal form in Lagrange polynomial:

n
Y x − xj
li (x) =
xi − xj
j=0,j6=i

Then Z b
Ai = li (x) dx.
−a

There are tables of such nodes and weights (Table 5.1 in textbook).
We skip the proof. See textbook if interested.

Advantage: Since all nodes are in the interior of the interval, these formulas can han-
dle integrals of function that tends to infinite value at one end of the interval (provided
that the integral is defined). Examples:
Z 1 Z 1 p
x −1
dx, (x2 − 1)1/3 sin(ex − 1) dx.
0 0
Chapter 5

Numerical solution of nonlinear

equations.

5.1 Introduction
Problem: f (x) given function, real-valued, possibly non-linear. Find a root r of f (x)
such that f (r) = 0.

Example 1. Quadratic polynomials: f (x) = x2 + 5x + 6.

f (x) = (x + 2)(x + 3) = 0, ⇒ r1 = −2, r2 = −3.

Roots are not unique.

Example 2. f (x) = x2 + 4x + 10 = (x + 2)2 + 6. There are no real r that would satisfy

f (r) = 0.
√
Example 3. f (x) = x2 + cos x + ex + x + 1. Roots can be difficult/impossible to find
analytically.

Our task now: use a numerical method to find an approximation to a root.

Overview of the chapter:

• Bisection (briefly)

• Fixed point iteration (main focus): general iteration, and Newton’s method

• Secant method

Systems (*) optional...

51
52 CHAPTER 5. NUMERICAL SOLUTION OF NONLINEAR EQUATIONS.

5.2 Bisection method

Given f (x), continuous function.

• Initialization: Find a, b such that f (a) · f (b) < 0.

This means there is a root r ∈ (a, b) s.t. f (r) = 0.
a+b
• Let c = 2 , mid-point.

• If f (c) = 0, done (lucky!)

• else: check if f (c) · f (a) < 0 or f (c) · f (b) < 0.

• Pick that interval [a, c] or [c, b], and repeat the procedure until stop criteria satis-
fied.

Stop Criteria:

1) interval small enough

2) |f (cn )| almost 0

3) max number of iteration reached

4) any combination of the previous ones.

Convergence analysis: Consider [a0 , b0 ], c0 = a0 +b+02 , let r ∈ (a0 , b0 ) be a root. The

error:
b0 − a0
e0 = |r − c0 | ≤
2
Denote the further intervals as [an , bn ] for iteration no. n. Then
bn − a n b0 − a0 e0
en = |r − cn | ≤ ≤ n+1 = n .
2 2 2
If the error tolerance is ε, we require en ≤ ε, then
b0 − a 0 log(b − a) − log(2ε)
≤ε ⇒ n≥ , (# of steps)
2n+1 log 2

Remark: very slow convergence.

5.3 Fixed point iterations

Rewrite the equation f (x) = 0 into the form x = g(x).
Remark: This can always be achieved, for example: x = f (x) + x. The catch is that,
the choice of g makes a difference in convergence.
Iteration algorithm:
5.3. FIXED POINT ITERATIONS 53

• Choose a start point x0 ,

• Do the iteration xk+1 = xk , k = 0, 1, 2, · · · until meeting stop crietria.

Stop Criteria: Let ε be the tolerance

• |xk − xk−1 | ≤ ε,

• |xk − g(xk )| ≤ ε,
• max # of iteration reached,

• any combination.

Example 1. f (x) = x − cosx.

Choose g(x) = cos x, we have x = cos x.
Choose x0 = 1, and do the iteration xk+1 = cos(xk ):

x1 = cos x0 = 0.5403
x2 = cos x1 = 0.8576
x3 = cos x2 = 0.6543
..
.
x23 = cos x22 = 0.7390
x24 = cos x23 = 0.7391
x25 = cos x24 = 0.7391 stop here

Our approximation to the root is 0.7391.

Example 2. Consider f (x) = e−2x (x − 1) = 0. We see that r = 1 is a root.

Rewrite as
x = g(x) = e−2x (x − 1) + x
Choose an initial guess x0 = 0.99, very close to the real root. Iterations:

x1 = cos x0 = 0.9886
x2 = cos x1 = 0.9870
x3 = cos x2 = 0.9852
..
.
x27 = cos x26 = 0.1655
x28 = cos x27 = −0.4338
x29 = cos x28 = −3.8477 Diverges. It does not work.

Convergence depends on x0 and g(x)!

54 CHAPTER 5. NUMERICAL SOLUTION OF NONLINEAR EQUATIONS.

Convergence analysis. Let r be the exact root, s.t., r = g(r).

Our iteration is xk+1 = g(xk ).
Define the error: ek = xk − r. Then,

ek+1 = xk+1 − r = g(xk ) − r = g(xk ) − g(r)

= g′ (ξ)(xk − r) (ξ ∈ (xk , r), since g is continuous)
′
= g (ξ)ek
 
⇒ |ek+1 | ≤ g ′ (ξ) |ek |

Observation:

• If |g′ (ξ)| < 1, error decreases, the iteration convergence. (linear convergence)

• If |g′ (ξ)| ≥ 1, error increases, the iteration diverges.

Convergence condition: There exists an interval around r, say [r − a, r + a] for some

a > 0, such that |g ′ (x)| < 1 for almost all x ∈ [r − a, r + a], and the initial guess x0 lies
in this interval.
In Example 1, g(x) = cos x, g ′ (x) = sin x, r = 0.7391,
 ′ 
g (r) = |sin(0.7391)| < 1. OK, convergence.

In Example 2, we have

g(x) = e−2x (x − 1) + x,
g′ (x) = −2e−2x (x − 1) + x−2x + 1

With r = 1, we have
 ′ 
g (r  = e−2 + 1 > 1

Divergence.

Pseudo code:

r=fixedpoint(’g’, x,tol,nmax}
r=g(r);
nit=1;
while (abs(r-g(r))>tol and nit < nmax) do
r=g(r);
nit=nit+1;
end
5.4. NEWTON’S METHOD 55

How to compute the error?

Assume |g ′ (x)| ≤ m < 1 in [r − a, r + a].
We have |ek+1 | ≤ m |ek |.
This gives

|e1 | ≤ m |e0 | , |e2 | ≤ m |e1 | ≤ m2 |e0 | , ··· |ek | ≤ mk |e0 |

We also have

|e0 | = |r − x0 | = |r − x1 + x1 − x0 | ≤ |e1 | + |x1 − x0 | ≤ m |e0 | + |x1 − x0 |

then
1
|e0 | ≤ |x1 − x0 | , (can be computed)
1−m
Put together
mk
|ek | ≤ |x1 − x0 | .
1−m
If the error tolerance is ε, then

mk ε(1 − m) ln(ε(1 − m)) − ln |x1 − x0 |

|x1 − x0 | ≤ ε, ⇒ mk ≤ ⇒ k≥
1−m |x1 − x0 | ln m

which give the maximum number of iterations needed to achieve an error ≤ ε.

Example cos x − x = 0, so

x = g(x) = cos x, g ′ (x) = − sin x

Choose x0 = 1. We know r ≈ 0.74. We see that the iteration happens between x = 0

and x = 1. For x ∈ [0, 1], we have
 ′ 
g (x) ≤ sin 1 = 0.8415 = m

And x1 = cos x0 = cos 1 = 0.5403. Then, to achieve an error ≤ ε = 10−5 , the maximum
# iterations needed is

ln(ε(1 − m)) − ln |x1 − x0 |

k≥ ≈ 73.
ln m

Of course that is the worst situation. Give it a try and you will find that k = 25 is
enough.
56 CHAPTER 5. NUMERICAL SOLUTION OF NONLINEAR EQUATIONS.

f (x) 6
f ′ (xk )

-
x
xk+1 xk

Figure 5.1: Newton’s method: linearize f (x) at xk .

5.4 Newton’s method

Goal: Given f (x), find a root r s.t. f (r) = 0.
Choose an initial guess x0 .
Once you have xk , the next approximation xk+1 is determined by treating f (x) as a
linear function at xk . See Figure 5.1.
We have
f (xk )
= f ′ (xk )
xk − xk+1
which gives
f (xk ) f (xk )
xk+1 = xk − = xk − ∆xk , ∆xk = .
f ′ (xk ) f ′ (xk )

Connection with fixed point iterations:

f (x) = 0, ⇒ b(x)f (x) = 0, ⇒ x = x − b(x)f (x) = g(x)
Here the function b(x) is chosen in such a way, to get fastest possible convergence. We
have
g′ (x) = 1 − b′ (x)f (x) − b(x)f ′ (x)
Let r be the root, such that f (r) = 0, and r = g(r). We have
 
g′ (r) = 1 − b(r)f ′ (r), smallest possible: g′ (r) = 0.
Choose now
1
1 − b(x)f ′ (x) = 0, ⇒ b(x) =
f ′ (x)
we get a fixed point iteration for
f (x)
x = g(x) = x − .
f ′ (x)
5.4. NEWTON’S METHOD 57

Convergence analysis. Let r be the root so f (r) = 0 and r = g(r). Define error:
ek+1 = |xk+1 − r| = |g(xk ) − g(r)|
Taylor expansion for g(xk ) at r:
1
g(xk ) = g(r) + (xk − r)g′ (r) + (xk − r)2 g′′ (ξ), ξ ∈ (xk , r)
2
′
Since g (r) = 0, we have
1
g(xk ) = g(r) + (xk − r)2 g′′ (ξ)
2
Back to the error, we now have
1   1  
ek+1 = (xk − r)2 g ′′ (ξ) = e2k g′′ (ξ)
2 2
Write again m = maxx |g′′ (ξ)|, we have
ek+1 ≤ m e2k
This is called Quadratic convergence. Guaranteed convergence if e0 is small enough! (m
can be big, it would effect the convergence!)
Proof for the convergence: (can drop this) We have
e1 ≤ m e0 e0
If e0 is small enough, such that m e0 < 1, then e1 < e0 .
Then, this means m e1 < me0 < 1, and so
e2 ≤ m e1 e1 < e1 , ⇒ m e2 < me1 < 1

Continue like this, we conclude that ek+1 < ek for all k, i.e., error is strictly decreasing
after each iteration. ⇒ convergence.
√
Example Find an numerical method to compute a using only +, −, ∗, / arithmetic
operations. Test it for a = 3.
√
Answer. It’s easy to see that a is a root for f (x) = x2 − a.
Newton’s method gives
f (xk ) x2k − a xk a
xk+1 = xk − = x k − = +
f ′ (xk ) 2xk 2 2xk
Test it on a = 3: Choose x0 = 1.7.
error
x0 = 1.7 7.2 × 10−2
x1 = 1.7324 3.0 × 10−4
x2 = 1.7321 2.6 × 10−8
x3 = 1.7321 4.4 × 10−16

Note the extremely fast convergence. Usually, if the initial guess is good (i.e., close to
r), usually a couple of iterations are enough to get an very accurate approximation.
58 CHAPTER 5. NUMERICAL SOLUTION OF NONLINEAR EQUATIONS.

Stop criteria: can be any combination of the following:

• |xk − xk−1 | ≤ ε

• |f (xk )| ≤ ε

• max number of iterations reached.

Sample Code:

r=newton(’f’,’df’,x,nmax,tol)
n=0; dx=f(x)/df(x);
while (dx > tol) and (f(x) > tol) and (n<nmax) do
n=n+1;
x=x-dx;
dx=f(x)/df(x);
end
r=x-dx;

5.5 Secant method

If f (x) is complicated, f ′ (x) might not be available.
Solution for this situation: using approximation for f ′ , i.e.,
f (xk ) − f (xk−1 )
f ′ (xk ) ≈
xk − xk−1
This is secant method:
xk − xk−1
xk+1 = xk − f (xk )
f (xk ) − f (xk−1 )

Advantages include

• No computation of f ′ ;

• One f (x) computation each step;

• Also rapid convergence.

A bit on convergence: One can show that

1 √
ek+1 ≤ Ceαk , α = (1 + 5) ≈ 1.62
2
This is called super linear convergence. (1 < α < 2)
5.6. SYSTEM OF NON-LINEAR EQUATIONS 59

It converges for all function f if x0 and x1 are close to the root r.

√
Example Use secant method for computing a.
Answer. The iteration now becomes

(x2k − a)(xk − xk−1 ) x2k − a

xk+1 = xk − = x k −
(x2k − a) − (x2k−1 − a) xk + xk+1

Test with a = 3, with initial data x0 = 1.65, x1 = 1.7.

error
x1 = 1.7 7.2 × 10−2
x2 = 1.7328 7.9 × 10−4
x3 = 1.7320 7.3 × 10−6
x4 = 1.7321 1.7 × 10−9
x5 = 1.7321 3.6 × 10−15

It is a little but slower than Newton’s method, but not much.

5.6 System of non-linear equations

Consider the system

F(~x) = 0, F = (f1 , f2 , · · · , fn )t , ~x = (x1 , x2 , · · · , xn )t

write it in detail: 

 f1 (x1 , x2 , · · · , xn ) = 0

 f2 (x1 , x2 , · · · , xn ) = 0
..


 .

fn (x1 , x2 , · · · , xn ) = 0

We use fixed point iteration. Same idea: choose ~x0 . Rewrite as

~x = G(~x)

The iteration is simply:

~xk+1 = G(~xk ).

Newton’s mathod:
~xk+1 = ~xk − Df (~xk )−1 · F(~xk )
60 CHAPTER 5. NUMERICAL SOLUTION OF NONLINEAR EQUATIONS.

where Df (~xk ) is a Jacobian matrix of f

 ∂f ∂f1 ∂f1 
1
···
 ∂x1 ∂x2 ∂xn 
 
 ∂f2 ∂f2 ∂f2 

 ∂x ··· 
 1 ∂x2 ∂xn 

Df =  
 .. .. .. .. 
 . . . . 
 
 
 ∂f ∂fn ∂fn 
n
···
∂x1 ∂x2 ∂xn

and Df (~xk )−1 is the inverse matrix of Df (~xk ).

Chapter 6

Direct methods for linear systems

6.1 Introduction
The problem:


 a11 x1 + a12 x2 + · · · + a1n xn = b1 (1)

 a21 x1 + a22 x2 + · · · + a2n xn = b2 (2)
(A) : ..


 .

an1 x1 + an2 x2 + · · · + ann xn = bn (n)

We have n equations, n unknowns, can be solved for xi if aij are “good”.

In compact form, for equation i:
n
X
aij xj = bi , i = 1, · · · , n.
j=1

Or in matrix-vector form:
A~x = ~b,
where A ∈ IRn×n , ~x ∈ IRn , ~b ∈ IRn
     
a11 a12 · · · a1n x1 b1
 a21 a22 · · · a2n   x2   b2 

A = {aij } =  . ..

..  ,
 
~x =  .  , ~b =  
 ..  .
 .. ..  .. 
. . .  .
an1 an2 · · · ann xn bn

Our goal: Solve for ~x. — We will spend 3 weeks on it.

Methods and topics:

• Different types of matrix A:

61
62 CHAPTER 6. DIRECT METHODS FOR LINEAR SYSTEMS

1. Full matrix
2. Large sparse system
3. Tri-diagonal or banded systems
4. regularity and condition number

• Methods

A. Direct solvers (exact solutions): slow, for small systems

∗ Gaussian elimination, with or without pivoting
∗ LU factorization
B. Iterative solvers (approximate solutions, for large sparse systems)
∗ more interesting for this course
∗ details later...

6.2 Gaussian elimination, simplest version

Consider system (A). The basic Gaussian elimination takes two steps:
Step 1: Make an upper triangular system – forward elimination.

for k = 1, 2, 3, · · · , n − 1
ajk
(j) ← (j) − (k) × akk , j = k + 1, k + 2, · · · , n

You will make lots of zeros, and the system becomes:



 a11 x1 + a12 x2 + · · · + a1n xn = b1 (1)

 a22 x2 + · · · + a2n xn = b2 (2)
(B) : ..


 .

ann xn = bn (n)

Note, here the aij and bi are different from those in (A).
Step 2: Backward substitution – you get the solutions.

bn
xn =
ann
 
n
1  X
xi = bi − aij xj  , i = n − 1, n − 2, · · · , 1.
aii
j=i+1

Potential problem: In step 1, if some akk is very close to or equal to 0, then you are in
trouble.
6.3. GAUSSIAN ELIMINATION WITH SCALED PARTIAL PIVOTING 63

Example 1. Solve

x + x2 + x3 = 1 (1)
 1


2x1 + 4x2 + 4x3 = 2 (2)



3x1 + 11x2 + 14x3 = 6 (3)

Forward elimination:
(1) ∗ (−2) + (2) : 2x2 + 2x3 = 0 (2′ )
(1) ∗ (−3) + (3) : 8x2 + 11x3 = 3 (3′ )
(2′ ) ∗ (−4) + (3′ ) : 3x3 = 3 (3′′ )

Your system becomes 

x + x2 + x3 = 1
 1


2x2 + 2x3 = 0



3x3 = 3

Backward substitution:

x3 = 1
1
x2 = (0 − 2x3 ) = −1
2
x1 = 1 − x2 − x3 = 1

It works fine here, but not always.

6.3 Gaussian Elimination with scaled partial pivoting

First, an example where things go wrong with Gaussian elimination.

Example 2. Solve the system with 3 significant digits.


0.001x1 −x2 = −1 (1)
x1 2x2 = 3 (2)

Answer. Write it with 3 significant digits


0.00100x1 −1.00x2 = −1.00 (1)
1.00x1 2.00x2 = 3.00 (2)

Now, (1) ∗ (−1000) + (2) gives

(1000 + 2)x2 = 1000 + 3

⇒ 1.00 · 103 x2 = 1.00 · 103
⇒ x2 = 1.00
64 CHAPTER 6. DIRECT METHODS FOR LINEAR SYSTEMS

Put this back into (1) and solve for x1 :

1 1
x1 = (−1.00 + 1.00x2 ) = ·0 0
0.001 0.001
.
Note that x1 is wrong!!
What is the problem? We see that 0.001 is a very small number!
One way around this difficulty: Change the order of two equations.

1.00x1 2.00x2 = 3.00 (1)
0.00100x1 −1.00x2 = −1.00 (2)

Now run the whole procedure again: (1) ∗ (−0.001) + (2) will give us x2 = 1.00.
Set it back in (1):
x1 = 3.00 − 2.00x2 = 1.00
Solution now is correct for 3 digits.
Conclusion: The order of equations can be important!

Consider 
a11 x1 + a12 x2 = b1
a21 x1 + a22 x2 = b2
Assume that we have computed x̃2 = x2 + ε2 where ε2 is the error (machine error, round
off error etc).
We then compute x1 with this x̃2 :
1
x̃1 = (b1 − a12 x̃2 )
a11
1
= (b1 − a12 x2 − a12 ε2 )
a11
1 a12
= (b1 − a12 x2 ) − ε2
a a
| 11 {z } | 11{z }
= x1 − ε1
a12 a12
Note that ε1 = a11 ε2 . Error in x2 propagates with a factor of a11 .
For best results, we wish to have |a11 | as big as possible.

Scaled Partial Pivoting. Idea: use maxk≤i≤n |aik | for aii .

Procedure:

1. Compute a scaling vector

~s = [s1 , s2 , · · · , sn ], where si = max |aij |
1≤j≤n

Keep this ~s for the rest of the computation.

6.3. GAUSSIAN ELIMINATION WITH SCALED PARTIAL PIVOTING 65

2. Find the index k s.t.    

 ak1   ai1 
 sk  ≥  si  , i = 1, · · · , n
   

Exchange eq (k) and (1), and do 1 step of elimination. You get


 ak1 x1 + ak2 x2 + · · · + akn xn = bk


(1)


 a 22 x 2 + · · · + a 2n x n = b 2 (2)


 ..
.

 a x
12 2 + · · · + a x
1n n = b1 (k)



 .
..



an2 x2 + · · · + ann xn = bn (n)

3. Repeat (2) for the remaining (n − 1) × (n − 1) system, and so on.

Example Solve the system using scaled partial pivoting.



 x1 + 2x2 + x3 = 3 (1)

3x1 + 4x2 + 0x3 = 3 (2)



2x1 + 10x2 + 4x3 = 10 (3)

Answer. We follow the steps.

1. Get ~s.
~s = [2, 4, 10]

2. We have
a11 1 a21 3 a31 2
= , = , = , ⇒ k=2
s1 2 s2 4 s3 10
Exchange eq (1) and (2), and do one step of elimination


 3x1 + 4x2 + 0x3 = 3 (2)

2
3 x2 + x3 = 2 (1′ ) = (1) + (2) ∗ ( 31 )

 22
(3′ ) = (3) + (2) ∗ (− 23 )

3 x2 + 4x3 = 8

3. For the 2 × 2 system,

ā12 2/3 1 ā32 22/3 22
= = , = = ⇒ k = 3.
s1 2 3 s3 10 30
Exchange (3’) and (1’) and do one step of elimination


 3x1 + 4x2 + 0x3 = 3 (2)

22
3 x2 + 4x3 = 8 (3′ )

 7 14 1
(1′′ ) = (1′ ) + (3′ ) ∗ (− 11

11 x3 = 11 )
66 CHAPTER 6. DIRECT METHODS FOR LINEAR SYSTEMS

4. Backward substitution gives

x3 = 2, x2 = 0, x1 = 1.

In MAtlab, to solve Ax = b, one can use:

> x= A\b;

6.4 LU-Factorization

Without pivoting: One can write A = LU where

L: lower triangular matrix with unit diagnal

U : upper triangular matrix

   
1 0 0 ··· 0 u11 u12 ··· u1,(n−1) u1n
 l21 1 0 ··· 0  0 u22 ··· u2,(n−1) u2n 
   
L =  l31 l32 1 ··· 0 U =  ... .. .. ..
  
, . . . 
 .. .. .. ..   
 . . . .  0 0 ··· u(n−1),(n−1) u(n−1),n 
ln1 ln2 ln3 · · · 1 0 0 ··· 0 unn

Theorem If Ax = b can be solved by Gaussian elimination without pivoting, then we

can write A = LU uniquely.
Use this to solve Ax = b: Let y = U x, then we have

Ux = y
Ly = b

Two triangular system. We first solve y (by forward substitution), then solve x (by
backward substitution).

With pivoting:
LU = P A

where P is the pivoting matrix.

Used in Matlab:
6.5. REVIEW OF LINEAR ALGEBRA 67

> [L,U]=lu(A);
> y = L \ b;
> x = U \ y;

+ transparences.
Work amount for direct solvers for A ∈ IRn×n : operation count
flop: one float number operation (+, -, *, / )
1 3
Elimination: 3 (n − n) flops
1 2
Backward substitution: 2 (n − n) flops
1 3
Total work amount is about 3n for large n.
This is very slow for large n. We will need something more efficient.

6.5 Review of linear algebra

Consider a square matrix A = {aij }.

Diagonal dominant system. If

n
X
|aii | > |aij | , i = 1, 2, · · · , n
j=1,j6=i

then A is called strictly diagonal dominant. And A has the following properties:

• A is regular, invertible, A−1 exists, and Ax = b has a unique solution.

• Ax = b can be solved by Gaussian Elimination without pivoting.

One such example: the system from natural cubic spline.

Vector and matrix norms:

A norm: measures the “size” of the vector and matrix.

General norm properties: x ∈ IRn or x ∈ IRn×n . Then, kxk satisfies

1. kxk ≥ 0, equal if and only if x = 0;

2. kaxk = |a| · kxk, a: is a constant;

3. kx + yk ≤ kxk + kyk, triangle inequality.

68 CHAPTER 6. DIRECT METHODS FOR LINEAR SYSTEMS

Examples of vector norms: x ∈ IRn

n
X
1. kxk1 = |xi |, l1 -norm
i=1

n
!1/2
X
2. kxk2 = x2i , l2 -norm
i=1

3. kxk∞ = max |xi |, l∞ -norm

1≤i≤n

Matrix norm related to the corresponding vector norm, A ∈ IRn×n

kAxk
kAk = max
x 6= 0
~ kxk

Obviously we have
kAxk
kAk ≥ ⇒ kAxk ≤ kAk · kxk
kxk
In addition we have
kIk = 1, kABk ≤ kAk · kBk .

Examples of matrix norms:

n
X
l1 − norm : kAk1 = max |aij |
1≤j≤n
i=1
l2 − norm : kAk2 = max |λi | , λi : eigenvalues of A
i
n
X
l∞ − norm : kAk∞ = max |aij |
1≤i≤n
j=1

Eigenvalues λi for A:

Av = λv, λ : eigenvalue, v : eigenvector

(A − λI)v = 0, ⇒ det(A − λI) = 0 : polynomial of degree n

In general, one gets n eigenvalues counted multiplicity.
Property:
1
λi (A−1 ) =
λi (A)
This implies
−1  
A = max λi (A−1 ) = max 1 1
2 −1
=
i i |λi (A )| mini |λi (A−1 )|
6.6. TRIDIAGONAL AND BANDED SYSTEMS 69

Condition number of a matrix A: Want to solve Ax = b. Put some perturbation:

Ax̄ = b + p
Relative error in perturbation:
kpk
eb =
kbk
Relative change in solution is
kx̄ − xk
ex =
kxk
We wish to find a relation between ex and eb . We have
A(x̄ − x) = p, ⇒ x̄ − x = A−1 p
so −1 −1
kx̄ − xk A p A · kpk
ex = = ≤ .
kxk kxk kxk
By using the following
1 kAk
Ax = b ⇒ kAXk = kbk ⇒ kAk kxk ≥ kbk ⇒ ≤
kxk kbk
we get
kAk
ex ≤ A−1 · kpk = kAk A−1 eb = κ(A)eb ,
kbk
Here
maxi |λi |
κ(A) = kAk A−1 =
mini |λi |
is called the condition number of A. Error in b propagates with a factor of κ(A) into the
solution.
If κ(A) is very large, Ax = b is very sensitive to perturbation, therefore difficult to solve.
We call this ill-conditioned system.
In Matlab: cond(A) gives the condition number of A.

6.6 Tridiagonal and banded systems

Tridiagonal system is when A is a tridiagonal matrix:
 
d1 c1 0 · · · 0 0 0
a1 d2 c2 · · · 0 0 0 
 
 0 a2 d3 . . .
 
 0 0 0 
A= . .. .. .. .. 
 .. . . . . 
 
 0 0 0 · · · dn−2 cn−2 0 
 
 0 0 0 · · · an−2 dn−1 cn−1 
0 0 0 ··· 0 an−1 dn
70 CHAPTER 6. DIRECT METHODS FOR LINEAR SYSTEMS

or we can write
A = tridiag(ai , di , ci ).
This can be solved very efficiently:

• Elimination (without pivoting)

for i = 2, 3, · · · , n
di ← di − adi−1 i−1
ci−1
ai−1
bi ← bi − di−1 bi−1
end
Now the A matrix looks like
 
d1 c1 0 ··· 0 0 0
0 d2 c2 ··· 0 0 0 
 
 .. 
0
 0 d3 . 0 0 0 
A= . .. .. .. .. 
 .. . . . . 
 
0 0 0 ··· dn−2 cn−2 0 
 
0 0 0 ··· 0 dn−1 cn−1 
0 0 0 ··· 0 0 dn

• Backward substitution
xn ← bn /dn
for i = n − 1, n − 2, · · · , 1
xi ← d1i (bi − ci xi+1 )
end

Amount of work: O(Cn) where C is a fixed small constant.

Penta-diagonal system. We can write

A = pentadiag(ei , ai , di , ci , fi )
which means
 
d1 c1 f1 0 0 ··· 0 0 0
 a1 d2 c2 f2 0 ··· 0 0 0 
 
 e1 a2 d3 c3 f3 · · · 0 0 0 
 
 .. .. .. .. .. .. .. 

 . . . . . . . 


A= .. .. .. .. .. 
 . . . . . 

 .. .. .. .. .. 
 . . . . . 
 
0 0 0 ··· dn−2 cn−2 fn−2 
 
0 0 0 0 ··· an−2 dn−1 cn−1 
0 0 0 0 0 ··· en−2 an−1 dn
6.6. TRIDIAGONAL AND BANDED SYSTEMS 71

Band matrix: This is a more general description:

 
.. .. ..
d . . . 0
 1 
 .. .. .. .. 
 . d2 . . . 
 
. . . . . . ..
.
..
.
..
. 
 d3 
. . .. .. .. .. 
A =   .. .. . d4 . . .

 .. .. .. .. .. .. 

 . . . . . .

 .. .. .. .. .. 

 . . . . .

.. .. ..
0 . . . dn
|←k→|

Here k ≥ 0 is the band width, meaning aij = 0 for all |i − j| > k.

We have

• diagonal matrix: k = 0,

• tridiagonal matrix: k = 1,

• pentadiagonal matrix: k = 2.

Gaussian elimination is efficient if k << n.

Some Matlab commands:

[L,U] = lu(A); % LU-factorization

norm(x); % vector norm
eig(A); % eigenvalue/eigen vector of a matrix
cond(A); % condition number of A
72 CHAPTER 6. DIRECT METHODS FOR LINEAR SYSTEMS
Chapter 7

Iterative solvers for linear

systems

7.1 General introduction

Want to solve Ax = b, where A ∈ IRn×n , n is very large, and A is sparse.

Two classes of solvers:

• Direct solvers (Gaussian Eliminatio, LU factorizatio etc): very slow.

• Iterative methods: finding only approximation to solutions. Useful for large sparse
systems, usually coming from discretization of differential equations.

Properties of such systems

• Large, n is very big, for example n = O(106 ).

• A is sparse, with a large percent of 0 entries.

• A is structured. (meaning: the product Ax can be computed efficiently)

Two classes of iterative methods

1. Fixed-point iterations

• Jacobi
• Gauss-Seidal
• SOR

2. Krylove technics (not covered in this course, though widely used)

73
74 CHAPTER 7. ITERATIVE SOLVERS FOR LINEAR SYSTEMS

7.2 Jacobi iterations

Want to solve Ax = b. Write it out


 a11 x1 + a12 x2 + · · · + a1n xn = b1

 a21 x1 + a22 x2 + · · · + a2n xn = b2
..


 .

an1 x1 + an2 x2 + · · · + ann xn = bn

Rewrite it in another way:


1

 x1 = a11 (b1 − a11 x1 + a12 x2 + · · · a1n xn )


 1
 x2 =

a22 (b2 − a21 x1 + a22 x2 + · · · a2n xn )
 ..


 .

 1
 xn = ann (b2 − an1 x1 + an2 x2 + · · · ann xn )

or in a compact for:
 
n
1  X
xi = bi − aij xj  , i = 1, 2, · · · , n
aii
j=1,j6=i

This gives the Jacobi iterations:

• Choose a start point, x0 = (x01 , x02 , · · · , x0n )t . For example, one may choose x0i = 1
for all i, or xi = bi /aii .

• for k = 0, 1, 2, · · · until stop criteria

for i = 1, 2, · · · , n
 
n
1  X
xk+1
i = bi − aij xkj 
aii
j=1,j6=i

end

end

Stop Criteria could be any combinations of the following

• xk close enough to xk−1 , for example xk − xk−1 ≤ ε for certain vector norms.

• Residual r k = Axk − b is small: for example r k ≤ ε.

• or others...
7.3. GAUSS-SEIDAL ITERATIONS 75

About the algorithm:

• Must make 2 vectors for the computation, xk and xk+1 .

• Great for parallel computing.

Example 1. Solve the following system with Jacobi iterations.


 2x1 − x2 = 0
−x1 + 2x2 − x3 = 1

−x2 + 2x3 = 2
given the exact solution x = (1, 2, 2)t .
Answer. Choose x0 by x0i = bi /aii :
x0 = (1, 1/2, 1)t
The iteration is  k+1
1 k

 x1 = 2 x2


xk+1
2 = 1
2 (1 + xk1 + xk3 )



 k+1 1
x3 = 2 (2 + xk2 )
We run a couple of iterations, and get
x1 = (0.25, 1, 1.25)t
x2 = (0.5, 1.25, 1.5)t
x3 = (0.625, 1.5, 1.625)t

Observations:

• Looks like it is converging. Need to run more steps to be sure.

• Rather slow convergence rate.

7.3 Gauss-Seidal iterations

An improved version: observe that in Jacobi iteration, we write
 
n
1 X
xk+1
i = bi − aij xkj 
aii
j=1,j6=i
 
i−1 n
1  X X
= bi − aij xkj − aij xkj 
aii
j=1 j=i+1

In the first summation term, all xkj are already computed for step k + 1.
We will replace all these xkj with xk+1
j . This gives:
76 CHAPTER 7. ITERATIVE SOLVERS FOR LINEAR SYSTEMS

Gauss-Seidal iterations: use the latest computed values of xi .

for k = 0, 1, 2, · · · , until stop criteria

for i = 1, 2, · · · , n
 
i−1 n
1 X X
xk+1
i =  bi − aij xk+1
j − aij xkj 
aii
j=1 j=i+1

end

end

About the algorithm:

• Need only one vector for both xk and xk+1 , saves memory space.

• Not good for parallel computing.

Example 2. Try it on the same Example 1, with x0 = (0, 0.5, 1)t . The iteration now is:
 k+1

 x1 = 12 xk2


xk+1
2 = 12 (1 + xk+1
1 + xk3 )



 k+1
x3 = 12 (2 + xk+1
2 )

We run a couple of iterations:

x1 = (0.25, 1.125, 1.5625)t

x2 = (0.5625, 1.5625, 1.7813)t

Observation: Converges a bit faster than Jacobi iterations.

7.4 SOR
SOR (Successive Over Relaxation) is a more general iterative method. It is based on
Gauss-Seidal.
 
i−1 n
1  X X
xk+1
j = (1 − w)xki + w · bi − aij xk+1
j − aij xkj 
aii
j=1 j=i+1

Note the second term is the Gauss-Seidal iteration multiplied with w.

w: relaxation parameter.
Usual value: 0 < w < 2 (for convergence reason)
7.5. WRITING ALL METHODS IN MATRIX-VECTOR FORM 77

• w = 1: Gauss-Seidal

• 0 < w < 1: under relaxation

• 1 < w < 2: over relaxation

Example Try this on the same example with w = 1.2. General iteration is now:
 k+1

 x1 = −0.2xk1 + 0.6xk2


xk+1
2 = −0.2xk2 + 0.6 ∗ (1 + xk+1
1 + xk3 )


 k+1
= −0.2xk3 + 0.6 ∗ (2 + xk+1

x3 2 )

With x0 = (0, 0.5, 1)t , we get

x1 = (0.3, 1.28, 1.708)t

x2 = (0.708, 1.8290, 1.9442)t

Recall the exact solution x = (1, 2, 2)t .

Observation: faster convergence than both Jacobi and G-S.

7.5 Writing all methods in matrix-vector form

Want to solve Ax = b.
Splitting of the matrix A:
A=L+D+U
where

• L is the lower triangular part of A:


aij , i>j
L = {lij }, lij =
0 i≤j

• D is the diagonal part of A:


aij = aii , i=j
D = {dij }, dij =
0 i 6= j

• U is the upper triangular part of A:


aij , i<j
U = {uij }, uij =
0 i≥j
78 CHAPTER 7. ITERATIVE SOLVERS FOR LINEAR SYSTEMS

A = D

Figure 7.1: Splitting of A.

See the graph in Figure 7.1 for an illustration.

Now we have
Ax = (L + D + U )x = Lx + Dx + U x = b

Jacobi iterations:
Dxk+1 = b − Lxk − U xk
so
xk+1 = D −1 b − D −1 (L + U )xk = yJ + MJ xk
where
yJ = D −1 b, MJ = −D −1 (L + U ).

Gauss-Seidal:
Dxk+1 + Lxk+1 = b − U xk

so
xk+1 = (D + L)−1 b − (D + L)−1 U xk = yGS + MGS xk
where
yGS = (D + L)−1 b, MGS = −(D + L)−1 U.

SOR:
xk+1 = (1 − w)xk + wD −1 (b − Lxk+1 − U xk )

⇒ Dxk+1 = (1 − w)Dxk + wb − wLxk+1 − wU xk

⇒ (D + wL)xk+1 = wb + [(1 − w)D − wU ]xk

so

xk+1 = (D + wL)−1 b + (D + wL)−1 [(1 − w)D − wU ]xk = ySOR + MSOR xk

where
ySOR = (D + wL)−1 b, MSOR = (D + wL)−1 [(1 − w)D − wU ].
7.6. ANALYSIS FOR ERRORS AND CONVERGENCE 79

7.6 Analysis for errors and convergence

Consider an iteration of the form

xk+1 = y + M xk

Assume s is the solution s.t. As = b.

This means s is a fixed point of the iteration: s = y + M s.
Define the error:
ek = xk − s
We have

ek+1 = xk+1 − s = y + M xk − (y + M s) = M (xk − s) = M ek .

This gives the propagation of error:

ek+1 = M ek .

Take the norm on both sides:

k+1
e = M e ≤ kM k · ek
k

This implies:
k
e ≤ M k e0 , e0 = x0 − s.

Theorem If kM k < 1 for some norm k·k, then the iterations converge.

NB! Convergence only depends on the iteration matrix M .

Check our methods: A = D + L + U .

• Jacobi: M = −D −1 (L + U ), given by A;

• G-S: M = −(D + L)−1 U , given by A;

• SOR: M = (D + wL)−1 [(1 − w)D − wU ], can adjust w to get a smallest posible

kM k. More flexible.

Example Let’s check the same example we have been using. We have
 
2 −1 0
A = −1 2 −1 ,
0 −1 2
80 CHAPTER 7. ITERATIVE SOLVERS FOR LINEAR SYSTEMS

so      
0 0 0 2 0 0 0 −1 0
L = −1 0 0 , D = 0 2 0 , U = 0 0 −1 .
0 −1 0 0 0 2 0 0 0
The iteration matrix for each method:
     
0 0.5 0 0 0.5 0 −0.2 0.6 0
MJ = 0.5 0 0.5 , MGS = 0 0.25 0.5  , MSOR =  −0.12 0.16 0.6 
0 0.5 0 0 0.125 0.25 −0.072 0.096 0.16

We list their various norms:

M l1 norm l2 norm l∞ norm

Jacobi 1 0.707 1
G-S 0.875 0.5 0.75
SOR 0.856 0.2 0.88

The l2 norm is the most significant one. We see now why SOR converges fastest.

Convergence Theorem. If A is diagonal dominant, i.e.,

n
X
|aii | > |aij | , ∀i = 1, 2, · · · , n.
j=1,j6=i

Then, all three iteration methods converge for all initial choice of x0 .
NB! If A is not diagonal dominant, it might still converge, but there is no guarantee.
Chapter 8

The Method of Least Squares

8.1 Problem description

Given data set

x x0 x1 x2 ··· xm
y y0 y1 y2 ··· yM
Data come from observation (measured) or experiments.
These yi ’s can have error (called “noise”) in measuring or experimenting.
y has a relation with x from physical model: y = y(x).
Then, our data is
yk = y(xk ) + ek
where ek is error.

8.2 Linear regression and basic derivation

Example 1. If y = ax + b, this is called linear regression. Your lab data would not lie
exact on a straight line (or your lab instructor will be very suspicious!). Our job now is
to find a straight line that “best” fit our data. See Figure 8.1 for an illustration.
A more specific way of saying the same thing: Find a, b, such that when we use y = ax+b,
the “error” becomes smallest possible.
How to measure error?

1. max |y(xk ) − yk | — l∞ norm

k
m
X
2. |y(xk ) − yk | — l1 norm
k=0

81
82 CHAPTER 8. LEAST SQUARES

y 6

-
x

Figure 8.1: Linear regression.

m
X
3. [y(xk ) − yk ]2 — l2 norm, used in Least Square Method. (LSM)
k=0

Our problem can now be stated as a minimization problem:

Find a and b such that the error function ψ(a, b) defined as

m
X
ψ(a, b) = (axk + b − yk )2
k=0

is minimized.

How to find a and b?

At the minimum of a function, we have

∂ψ ∂ψ
= =0
∂a ∂b

In detail:
m
∂ψ X
=0 : 2(axk + b − yk )xk = 0, (I)
∂a
k=0
m
∂ψ X
=0 : 2(axk + b − yk ) = 0, (II)
∂b
k=0

Solve (I), (II) for (a, b). Rewrite it as a system

 ! !
 Xm Xm Xm
 2


 xk · a + xk · b = xk · y k

k=0 k=0 k=0
m
! m

 X X

 x · a + (m + 1) · b = yk

 k
k=0 k=0
8.2. LINEAR REGRESSION AND BASIC DERIVATION 83

These are called the normal equations.

Example 2. Consider the data set (from textbook, p.428)

Tk 0 10 20 30 40 80 90 95
Sk 68.0 67.1 66.4 65.6 64.6 61.8 61.0 60.0

where

S: surface tension in a liquid

T : temperature

From physics we know that they have a linear relation

S = aT + b

Use MLS to find the best fitting a, b.

Answer. We have m = 7, and

7
X
Tk2 = 0 + 102 + 202 + · · · + 902 + 952 = 26525
k=0
X7
Tk = · · · = 365
k=0
7
X
Tk Sk = · · · = 22685
k=0
7
X
Sk = · · · = 514.5
k=0

The normal equations are


26525 a + 365 b = 22685
365 a + 8 b = 514.5

Solve it
a = −0.079930, b = 67.9593.

So
S(T ) = −0.079930 T + 67.9593.
84 CHAPTER 8. LEAST SQUARES

8.3 LSM with parabola

Example 3. Given m + 1 data (xk , yk )m 2

k=0 . Find y(x) = ax + bx + c that best fit our
data.
Answer. Define the error (in least squares sense)
m
X
2
ψ(a, b, c) = ax2k + bxk + c − yk
k=0

At minimum, we have
∂ψ ∂ψ ∂ψ
= = =0
∂a ∂b ∂c
In detail:
m
∂ψ X

=0 : 2 ax2k + bxk + c − yk · x2k = 0
∂a
k=0
m
∂ψ X

=0 : 2 ax2k + bxk + c − yk · xk = 0
∂b
k=0
m
∂ψ X

=0 : 2 ax2k + bxk + c − yk = 0
∂c
k=0

The normal equations:

 m
! m
! m
! m
 X X X X
4


 xk · a + x3k ·b+ x2k ·c = x2k yk


 k=0
 k=0 k=0 k=0

 m
! m
! m
! m
 X X X X
x3k · a + x2k ·b+ xk ·c = xk y k



 k=0 k=0 k=0 k=0

 m
! m
! m

 X X X
2


 xk · a + xk · b + (m + 1) · c = yk
k=0 k=0 k=0

We have 3 linear equations for 3 unknown. The system is symmetric.

Need to compute only:
X X X X X X X
x4k , x3k , x2k , xk , x2k yk , xk y k , yk

8.4 LSM with non-polynomial

Example 4. Non-polynomial example. Given data set (xk , yk ) for k = 0, 1, · · · , m.

Find a
y(x) = a · f (x) + b · g(x) + c · h(x)
8.5. GENERAL LINEAR LSM 85

which best fit the data. This means we need to find (a, b, c).
Here f (x), g(x), h(x) are given functions, for example
f (x) = ex , g(x) = ln(x), h(x) = cos x,
but not restricted to these.
Define the error function:
m
X m
X
ψ(a, b, c) = (y(xk ) − yk )2 = (a · f (xk ) + b · g(xk ) + c · h(xk ) − yk )2
k=0 k=0
At minimum, we have
m h i
∂ψ X
=0 : 2 a · f (xk ) + b · g(xk ) + c · h(xk ) − yk · f (xk ) = 0
∂a
k=0
m h i
∂ψ X
=0 : 2 a · f (xk ) + b · g(xk ) + c · h(xk ) − yk · g(xk ) = 0
∂b
k=0
m h i
∂ψ X
=0 : 2 a · f (xk ) + b · g(xk ) + c · h(xk ) − yk · h(xk ) = 0
∂c
k=0

The normal equations are

 m
! m
! m
! m
 X X X X
 2


 f (xk ) a + f (xk )g(xk ) b + f (xk )h(xk ) c = f (xk )yk

 k=0 k=0 k=0 k=0



 ! ! !

 X m X m Xm Xm
2
f (xk )g(xk ) a + g(xk ) b + h(xk )g(xk ) c = g(xk )yk



 k=0 k=0 k=0 k=0

 ! ! !

 Xm Xm Xm Xm

 2


 f (x k )h(x k ) a + f (x k )h(x k ) b + h(x k ) c = h(xk )yk
k=0 k=0 k=0 k=0

We note that the system of normal equations is always symmetric. We only need to
compute half of the entries.

8.5 General linear LSM

Let g0 , g1 , g2 , · · · gn be n + 1 given functions (they don’t need to be linear).
Given data set (xk , yk ), k = 0, 1, · · · , m. (m and n are in general different).
We search for a function in the form
n
X
y(x) = ci gi (x)
i=0
that best fit the data.
Here gi ’s are called basis functions.
86 CHAPTER 8. LEAST SQUARES

How to choose the basis functions? They are chosen such that the system of the
normal equations is regular (invertible) and well-conditioned.

Requirements. gi ’s must be a set of linearly independent functions. Meaning: one

can not be written as linear combination of the others, or

n
X
ci gi (x) = 0, if and only if c0 = c1 = c2 = · · · = cn = 0.
i=0

Define error function

m h m
" n #2
X i2 X X
ψ(c0 , c1 , · · · , cn ) = y(xk ) − yk = ci gi (x) − yk
k=0 k=0 i=0

At minimum, we have
∂ψ
= 0, j = 0, 1, ·, n.
∂cj

This gives:
m
" n
#
X X
2 ci gi (xk ) − yk gj (xk ) = 0
k=0 i=0

Re-arranging the ordering of summation signs:

n m
! m
X X X
gi (xk )gj (xk ) ci = gj (xk )yk , j = 0, 1, ·, n.
i=0 k=0 k=0

This gives the system of normal equations:

A~c = ~b

where ~c = (c0 , c1 , · · · , cn )t and

m
X
A = {aij }, aij = gi (xk )gj (xk )
k=0
m
X
~b = {bj }, bj = gj (xk )yk .
k=0

We note that this A is symmetric.

8.6. NON-LINEAR LSM 87

8.6 Non-linear LSM

Next is an example of quasi-linear LSM.

Example 5. Consider fitting the data with the function

y(x) = a · bx

This means, we need to find (a, b) such that this y(x) best fit the data.
Do a variable change:
ln y = ln a + x · ln b .
Let
S = ln y, ā = ln a, b̄ = ln b.
Given data set (xk , yk ). Compute Sk = ln yk for all k.
We can now find (ā, b̄) such that Sk best fits (xk , Sk ).
Then, transform back to the original variable

a = exp{ā}, b = exp{b̄}.

Example 6. Non-linear LSM. For example,

y(x) = ax · sin(bx).

We can not find a variable change that can change this problem into a linear one. So
we will now deal with it as a non-linear problem.
Define error
m h
X i2 m h
X i2
ψ(a, b) = y(xk ) − yk = axk · sin(bxk ) − yk .
k=0 k=0

At minimum:
m h i
∂ψ X
=0: 2 axk · sin(bxk ) − yk · [xk · sin(bxk )] = 0
∂a
k=0
m h i
∂ψ X
=0: 2 axk · sin(bxk ) − yk · [axk · cos(bxk )xk ] = 0
∂a
k=0

We now have a 2 × 2 system of non-linear equations to solve for (a, b)!

May use Newton’s method to find a root.
May have several solutions, including all the maximum, minimum and saddle points.
(see slides)
88 CHAPTER 8. LEAST SQUARES
Chapter 9

Numerical solution of ordinary

differential equations (ODE)

9.1 Introduction
Definition of ODE: an equation which contains one or more ordinary derivatives of an
unknown function.

Example 1. Let x = x(t) be the unknown function of t, ODE examples can be

x′ = x2 , x′′ + x · x′ + 4 = 0, etc.

We consider the initial-value problem for first-order ODE

 ′
x = f (t, x), − − differential equation
(∗)
x(t0 ) = x0 − − −initial condition, given

Some examples:
x′ = x + 1, x(0) = 0. solution: x(t) = et − 1
x′ = 2, x(0) = 0. solution: x(t) = 2t.

In many situations, exact solutions can be very difficult/impossible to obtain.

Numerical solutions: Given (*), find xn = x(tn ), n = 1, 2, · · · , N , and t0 < t1 <

· · · < tN . Here tN is final computing time.
Take uniform time step: Let h be the time step length
tn+1 − tn = h, tk = t0 + kh

Overview:

89
90 CHAPTER 9. ODES

• Taylor series method, and error estimates

• Runge-Kutta methods
• Multi-step methods
• System of ODE
• High order equations and systems
• Stiff systems
• Matlab solvers

9.2 Taylor series methods for ODE

Given
x′ (t) = f (t, x(t)), x(t0 ) = x0 .
Let’s find x1 = x(t1 ) = x(t0 + h). Taylor expansion gives
∞
′ 1 2 ′′ X 1 m (m)
x(t0 + h) = x(t0 ) + hx (t0 ) + h x (t0 ) + · · · = h x (t0 )
2 m!
m=0

Taylor series method of order m: take the first (m + 1) terms in Taylor expansion.
1 1 m (m)
x(t0 + h) = x(t0 ) + hx′ (t0 ) + h2 x′′ (t0 ) + · · · + h x (t0 ).
2 m!
Error in each step:
∞
X 1 k (k) 1
x(t0 + h) − x1 = h x (t0 ) = hm+1 x(m+1) (ξ)
k! (m + 1)!
k=m+1

for some ξ ∈ (t0 , t1 ).

For m = 1, we have Euler’s method:

x1 = x0 + hx′ (t0 ) = x0 + h · f (t0 , x0 )

General formula for step number k:

xk+1 = xk + h · f (tk , xk ), k = 0, 1, 2, · · ·

For m = 2, we have
1
x1 = x0 + hx′ (t0 ) + h2 x′′ (t0 )
2
9.2. TAYLOR SERIES METHODS FOR ODE 91

Using

d
x′′ (t0 ) = f (t0 , x(t0 )) = ft (t0 , x0 ) + fx (t0 , x0 ) · x′ (t0 ) = ft (t0 , x0 ) + fx (t0 , x0 ) · f (t0 , x0 )
dt
we get
1
x1 = x0 + hf (t0 , x0 ) + h2 [ft (t0 , x0 ) + fx (t0 , x0 ) · f (t0 , x0 )]
2
For general step k, we have
1
xk+1 = xk + hf (tk , xk ) + h2 [ft (tk , xk ) + fx (tk , xk ) · f (tk , xk )]
2

Example 2. Set up Taylor series methods with m = 1, 2 for

x′ = −x + e−t , x(0) = 0.

(Exact solution is x(t) = te−t .)

Answer. The initial data gives x0 = 0.
For m = 1, we have

xk+1 = xk + h(−xk + e−tk ) = (1 − h)xk + he−tk

For m = 2, we have

x′′ = (−x + e−t )′ = −x′ − e−t = x − e−t − e−t = x − 2e−t

so
1
xk+1 = xk + hx′k + h2 x′′k
2
1
= xk + h (−xk + exp{−tk }) + h2 [xk − 2 exp{−tk }]
2
1 2 2
= (1 − h + h )xk + (h − h ) exp{−tk }
2

Example 3. Set up Taylor series methods with m = 1, 2, 3, 4 for

x′ = x, x(0) = 1.

(Exact solution x(t) = et )

Answer. We set x0 = 1. Note that

x′′ = x′ = x, x′′′ = x′′ = x, ··· x(m) = x

92 CHAPTER 9. ODES

We have

m=1: xk+1 = xk + hxk = (1 + h)xk

1 1
m=2: xk+1 = xk + hxk + h2 xk = (1 + h + h2 )xk
2 2
1 2 1 3 1 1
m=3: xk+1 = xk + hxk + h xk + h xk = (1 + h + h2 + h3 )xk
2 6 2 6
1 2 1 3 1 4
m=4: xk+1 = (1 + h + h + h + h )xk
2 6 24
See slides.

Error analysis. Given ODE

x′ = f (t, x), x(t0 ) = x0 .

Local truncation error (error in each time step) for Taylor series method of order m is
 m+1   m+1 m 
(k)  h (m+1)
  h d f (ξ) 
eL = |xk+1 − x(tk + h)| =  x (ξ) = 
  , ξ ∈ (tk , tk+1 ).
(m + 1)! (m + 1)! dtm 
Here we use the fact that
dm f
x(m+1) =
dtm
Assume now that  m 
d f 
 dtm  ≤ M.
 

We have
(k) M
eL ≤ hm+1 = O(hm+1 ).
(m + 1)!
Total error: sum over all local errors.
Detail: We want to compute x(T ) for some time t = T . Choose an h. Then total
number of steps is
N = f racT h, i.e., T = N h.
Then
N   N
X  (k)  X M
E = eL  ≤ hm+1
(m + 1)!
k=1 k=1
M M MT
= N hm+1 = (N h) hm = hm = O(hm )
(m + 1)! (m + 1)! (m + 1)!

Therefore, the method is of order m.

In general: If the local truncation error is of order O(ha+1 ), then the total error is of
O(ha ), i.e., one order less.
9.3. RUNGE KUTTA METHODS 93

9.3 Runge Kutta methods

Difficulty in high order Taylor series methods:

x′′ , x′′′ , · · · , might be very difficult to get.

A better method: should only use f (t, x), not its derivatives.
1st order method: The same as Euler’s method.
2nd order method: Let h = tk+1 − tk . Given xk , the next value xk+1 is computed as

1
xk+1 = xk + (K1 + K2 )
2
where (
K1 = h · f (tk , xk )

K2 = h · f (tk+1 , xk + K1 )
This is called Heun’s method.

Proof that this is a second order method: Taylor expansion in two variables

f (t + h, x + K1 ) = f (t, x) + hft (t, x) + K1 fx (t, x) + O(h2 , K12 ).

We have K1 = hf (t, x), abd

h i
K2 = h f (t, x) + hft (t, x) + hf (t, x)fx (t, x) + O(h2 )

Then, our method is:

1h i
xk+1 = xk + hf + hf + h2 ft + h2 f fx + O(h3 )
2
1
= xk + hf + h2 [ft + f fx ] + O(h3 )
2
Compare this with Taylor expansion for x(tk+1 ) = x(tk + h)

1
x(tk + h) = x(tk ) + hx′ (tk ) + h2 x′′ (tk ) + O(h3 )
2
1
= x(tk ) + hf (tk , xk ) + h2 [ft + fx x′ ] + O(h3 )
2
1 2
= x(tk ) + hf + h [ft + fx f ] + O(h3 ).
2
We see the first 3 terms are identical, this gives the local truncation error:

eL = O(h3 )
94 CHAPTER 9. ODES

meaning that this is a 2nd order method.

In general, Rung-Kutta methods of order m takes the form

xk+1 = xk + w1 K1 + w2 K2 + · · · + wm Km

where 

 K1 = h · f (tk , xk )





 K2 = h · f (tk + a2 h, x + b2 K1 )


K3 = h · f (tk + a3 h, x + b3 K1 + c3 K2 )



 ..


 .

 Pm−1
 K = h · f (tk + am h, x + φi Ki )
m i=1

The parameters wi , ai , bi , φi are carefully chosen to guarantee the order m.

NB! The choice is NOT unique!

The classical RK4 : a 4th order method takes the form

1h i
xk+1 = xk + K1 + 2K2 + 2K3 + K4
6
where

K1 = h · f (tk , xk )
1 1
K2 = h · f (tk + h, xk + K1 )
2 2
1 1
K3 = h · f (tk + h, xk + K2 )
2 2
K4 = h · f (tk + h, xk + K3 )

See slides for codes.

9.4 An adaptive Runge-Kutta-Fehlberg method

In general, we have

1. smaller time step h gives smaller error;

2. The error varies at each step, depending on f .

Optimal situation: h varies each step to get uniform error at each step.
9.4. AN ADAPTIVE RUNGE-KUTTA-FEHLBERG METHOD 95

This leads to adaptive methods.

Key point: How to get an error estimate at each time step?
One possibility:

• Compute x(t + h) from x(t) with step h;

• Compute x(t + 21 h) from x(t) with step 21 h, then compute x(t + h) from x(t + 12 h)
with step 12 h; Call this x̄(t + h);

• Then, |x(t + h) − x̄(t + h)| gives a measure to error;

• if error >> tol, half the step size;

• if error << tol, double the step size;

• if error ≈ tol, keep the step size;

But this is rather wasteful of computing time. Although the idea is good.

A better method, by Fehlberg, building upon R-K methods. He has a 4th order method:
25 1408 2197 1
x(t + h) = x(t) + K1 + K3 + K4 − K5 ,
216 2565 4104 5
where

K1 = h · f (t, x)
1 1
K2 = h · f (t + h, x + K1 )
4 4
3 3 9
K3 = h · f (t + h, x + K1 + K2 )
8 32 32
12 1932 7200 7296
K4 = h · f (t + h, x + K1 − K2 + K3 )
13 2197 2197 2197
439 3680 845
K5 = h · f (t + h, x + K1 − 8K2 + K3 − K4 )
216 513 4104

Adding an additional term:

1 8 3544 1859 11
K6 = h · f (t + h, x − K1 + 2K2 − K3 + K4 − K5 )
2 27 2565 4104 40
We obtain a 5th order method:
16 6656 28561 9 2
x̄(t + h) = x(t) + K1 + K3 + K4 − K5 + K6 .
135 12825 56430 50 55

Main interests here: The difference |x(t + h) − x̄(t + h)| gives an estimate for the error.
Pseudo code for adaptive RK45, with time step controller
96 CHAPTER 9. ODES

Given t0 , x0 , h0 , nmax , emin , emax , hmin , hmax

set h = h0 , x = x0 , k = 0,

while k < nmax do

if h < hmin then h = hmin

else if h > hmax then h = hmax
end
Compute RK4, RK5, and e = |RK4 − RK5|
if e > emax , then h = h/2;
else
k=k+1; t=t+h; x=RK5;
If e < emin , the h = 2 ∗ h; end
end

end (while)

9.5 Multi-step methods

Given
x′ = f (t, x), x(t0 ) = x0 ,
Let tn = t0 + nh. If x(tn ) is given, the exact value for x(tn=1 ) would be
Z tn+1
x(tn+1 ) = x(tn ) + x′ (s) ds
tn

Assume we know xn , xn−1 , xn−2 , · · · , xn−k one can approximate the integrand x′ (s) by
using interpolating polynomial.

Example Consider k = 1. Given xn , xn−1 , we can compute fn , fn−1 as

fn = f (tn , xn ), fn−1 = f (tn−1 , xn−1 )

Use now linear approximation, i.e., straight line interpolation,

fn − fn−1
x′ (s) ≈ P1 (s) = fn−1 + (s − tn−1 ).
h
Then Z tn+1
h
xn+1 = xn + P1 (s) ds = xn + (3fn − fn−1 ).
tn 2
9.5. MULTI-STEP METHODS 97

Adams-Bashforth method: The explicit version. Given

x, xn−1 , · · · , xn−k
and
fn , fn−1 , · · · , , fn−k
find interpolating polynomial Pk (t) that interpolates (ti , fi )ni=n−k . Compute
Z tn+1
xn+1 = xn + Pk (s) ds
tn

which will always be in the form

xn+1 = xn + h · (b0 fn + b1 fn−1 + b2 fn−2 + · · · + bk fn−k ).

Good sides: Simple, minimum number of f (·) evaluations. Fast.

Disadvantage: Here we use interpolating polynomial to approximate a function out-

side the interval of interpolating points. This gives bigger error.

Improved version. implicit method, Adams-Bashforth-Moulton (ABM) method.

Find interpolating polynomial Pk+1 (t) that interpolates
(fn+1 , tn+1 ), (fn , tn ), ··· , (fn−k , tn−k ),
and use Z tn+1
xn+1 = xn + Pk+1 (s) ds
tn
which will always be in the form
xn+1 = xn + h · (b−1 fn+1 + b0 fn + b1 fn−1 + b2 fn−2 + · · · + bk fn−k )
where fn+1 = f (tn+1 , xn+1 ) which is unknown. Therefore, we get a non-linear equation.
Can be solved by fixed-point iteration (Newton’s method). Use AB solution as the initial
guess, it will converge in 2-3 iterations.

Example For a couple of k values, we have

k = −1 : xn+1 = xn + h · fn+1 , (implicit backward Euler’s method)
h
k=0: xn+1 = xn + (fn + fn+1 ), (trapezoid rule)
2  
5 3 1
k=1: xn+1 = xn + h · fn+1 + fn − fn−1
12 12 12

With fixed point iteration:

98 CHAPTER 9. ODES

Given xn , xn−1 , fn , fn−1 , compute AB solution with k = 1:

  
 ∗ 3 1
xn+1 = xn + h fn − fn−1
(P ) 2
2
fn=1 = f tn+1 , x∗n+1 .
 ∗

Do one iteration of Newton’s method, to correct the error:


 h ∗

xn+1 = xn + fn+1 + fn
(C) 2
 f = f (t
n+1 n+1 , xn+1 )

Here step (P ) is called the predictor, and step (C) is the corrector.
This is called predictor-corrector’s method.

9.6 Methods for first order systems of ODE

We consider
~x ′ = F (t, ~x), ~x(t0 ) = ~x0
Here ~x = (x1 , x2 , · · · , xn )t is a vector, and F = (f1 , f2 , · · · , fn )t is a vector-values
function.
Write it out  ′
x = f1 (t, x1 , x2 , · · · , xn )
 ′1


x2 = f2 (t, x1 , x2 , · · · , xn )

 ···
 ′
xn = fn (t, x1 , x2 , · · · , xn )

Remark: All methods for scalar equation can be used for systems!

Taylor series methods:

1 1 m (m)
~x(t + h) = ~x + h~x ′ + h2 ~x ′′ + · · · + h ~x
2 m!

Example Consider 
x′1 = x1 − x2 + 2t − t2 − t3
x′2 = x1 + x2 − 4t2 + t3
We will need the high order derivatives:
 ′′
x1 = x′1 − x′2 + 2 − 2t − 3t2
x′′2 = x1 + x2 − 8t + 3t2
9.7. HIGHER ORDER EQUATIONS AND SYSTEMS 99

and 
x′′′
1 = x′′1 − x′′2 − 2 − 6t
x2 = x′′1 + x′′2 − 8 + 6t
′′′

and so on...

Runge-Kutta methods take the same form. For example, RK4:

1h ~ ~ ~ ~
i
~xk+1 = ~xk + K 1 + 2K2 + 2K3 + K4
6
where
~ 1 = h · F (tk , ~xk )
K

~ 2 = h · F (tk + 1 h, ~xk + 1 K
K ~ 1)
2 2
~ 3 = h · F (tk + 1 h, ~xk + 1 K
K ~ 2)
2 2
~ 4 = h · F (tk + h, ~xk + K
K ~ 3)

Here everything is a vector instead of a scalar value.

9.7 Higher order equations and systems

Treatment: Rewrite it into a system of first order equations.

Example Higher order ODE

x(n) = f (t, x, x′ , x′′ , · · · , x(n−1) )
with
x(t0 ), x′ (t0 ), x′′ (t0 ), · · · , x(n−1) (t0 )
given.
Introduce a systematic change of variables
x1 = x, x2 = x′ , x3 = x′′ , ··· xn = x(n−1)

We then have 

 x′1 = x′ = x2



 x′2 = x′′ = x3
x′3 = x′′′ = x4


..


 .
= x(n−1) = xn


 x′
 n−1′

xn = x(n) = f (t, x1 , x2 , · · · , xn )
This is a system of 1st order ODEs.
Systems of high-order equations are treated in the same way.
100 CHAPTER 9. ODES

9.8 Stiff systems

A system is called stiff if the solution consists of components that vary with very
different speed/frequency.

Example Consider
x′ = −ax, x(0) = 1
the exact solution is
x(t) = e−t
We see that
(P 1) x → 0 as t → +∞.

Solve it by Euler’s method:

xn+1 = xn − ahxn = (1 − ah)xn , ⇒ xn = (1 − ah)n x0 = (1 − ah)h

In order to keep the property (P 1), in the approximate solution, i.e.,

xn → 0, as n → +∞,

We must have
2
|1 − ah| < 1, ⇒ h<
a
which gives a restriction to the time step size: it has to be sufficiently small.

Example Now consider a system

 ′ 
x = −20x − 19y x(0) = 2
y ′ = −19x − 20y y(0) = 0

The exact solution is (

x(t) = e−39t + e−t
y(t) = e−39t − e−t

Observations:

• The solution tends to 0, i.e., x → 0, y → 0 as t → +∞.

• Two components in the solution, e−39t and e−t ;

• They decay at a very different rate. The term e−39t tends to 0 much faster than
the term e−t ;

• For large values of t, the term e−t dominate.

• Therefore, e−39t is called the transient term.

9.8. STIFF SYSTEMS 101

Solve it with Euler’s method:

( 
xn+1 = xn + h · (−20xn − 19yn ) x0 = 2
yn+1 = yn + h · (−19xn − 20yn ) y0 = 0

One can show by induction that

(
xn = (1 − 39h)n + (1 − h)n
yn = (1 − 39h)n − (1 − h)n

We must require that

xn → 0, yn → 0 as n → +∞.

This gives the conditions

|1 − 39h| < 1 and |1 − h| < 1

which implies
2
(1) : h < and (2) : h < 2
39
We see that condition (1) is much stronger than condition (2), therefore it must be
satisfied.
Condition (1) corresponds to the term e−39t , which is the transient term and it tends
to 0 very quickly as t grows. Unfortunately, time step size is restricted by this transient
term.

A more stable method: Backward Euler, implicit method

(
xn+1 = xn + h · (−20xn+1 − 19yn+1 )
yn+1 = yn + h · (−19xn+1 − 20yn+1 )

Let      
−20 −19 x xn
A= , ~x = , ~xn = .
−19 −20 y yn
We can write

~xn+1 = ~xn + A · ~xn+1 ,

⇒ (I − hA)~xn+1 = ~xn
⇒ ~xn+1 = (I − hA)−1 ~xn

Take some vector norm on both sides

k~xn+1 k = (I − hA)−1 ~xn ≤ (I − hA)−1 k~xn k .
102 CHAPTER 9. ODES

We see that if (I − hA)−1 < 1, then ~xn → 0 as n → +∞.
Use the l2 norm:
 
(I − hA)−1 = max λi (I − hA)−1  = max 1
i i |(1 − h · λi (A))|

We have
λ1 (A) = −1, λ2 (A) = −39
They are both negative, therefore 1 − hλi > 1, implying

(I − hA)−1 < 1

independent of the value of h.

Therefore, this method is called unconditionally stable.

Advantage: Can choose large h, always stable. Suitable for stiff systems.

Disadvantage: Must solve a system of linear equations at each time step

(I − hA)~xn+1 = ~xn

Longer computing time for each step. Not recommended if the system is not stiff.