WBMT2049-T2/WI2032TH – Numerical Analysis for ODE’s

Lecture 2

mathematical preliminaries

Lecture 2 1 / 23
Landau’s big-O order symbol

Lecture 2 2 / 23
Uses of big-O notation

In Lecture 1 we had already used the O-symbol:

fl(x1 + x2 ) = fl(fl(x1 ) + fl(x2 )) = (x1 (1 + ϵ1 ) + x2 (1 + ϵ2 ))(1 + ϵc2 )

= x1 + x2 + x1 (ϵ1 + ϵc2 ) + x2 (ϵ2 + ϵc2 ) + x1 ϵ1 ϵc2 + x2 ϵ2 ϵc2
= x1 + x2 + x1 (ϵ1 + ϵc2 ) + x2 (ϵ2 + ϵc2 ) + O(ϵ2 )

Lecture 2 3 / 23
Definition: O-symbol
Let f and g be given functions. Then,

f (x) = O(g (x)), for x → 0,

if there exists r > 0 and M < ∞ such that

|f (x)| ≤ M|g (x)|, for all x ∈ [−r , r ]

Lecture 2 4 / 23
Big-O notation: example

Let f (x) = x0 + x1 ϵ + x2 ϵ2 + x3 ϵ3 . Then,

f (x) = x0 + x1 ϵ + O(ϵ2 ), for ϵ → 0.

Proof. We claim that x2 ϵ2 + x3 ϵ3 = O(ϵ2 ) for ϵ → 0, i.e., that there exist

r > 0 and M < ∞, such that

|x2 ϵ2 + x3 ϵ3 | ≤ M|ϵ2 |, for all ϵ ∈ [−r , r ].

Take any 0 < r < ∞. For all ϵ ∈ [−r , r ], it holds that |ϵ| ≤ r and

|x2 ϵ2 + x3 ϵ3 | = |x2 + x3 ϵ| |ϵ2 | ≤ (|x2 | + |x3 | |ϵ|) |ϵ2 |

≤ (|x2 | + |x3 |r ) |ϵ2 | = M|ϵ2 |,

where M = |x2 | + |x3 |r < ∞.

Lecture 2 5 / 23
Computations with O symbol

Let f (x) = O(x p ) and g (x) = O(x q ) for x → 0 with p, q ≥ 0. Then,

f (x) = O(x p ) = O(x s ) for all 0 ≤ s ≤ p

αf (x) + βg (x) = αO(x p ) + βO(x q ) = O(x min(p,q) ) for all α, β ∈ R
f (x)g (x) = O(x p )O(x q ) = O(x p+q )
f (x) O(x p )
= = O(x p−s ) if 0 ≤ s ≤ p.
|x|s |x|s

Lecture 2 6 / 23
Computations with O symbol: example

Let f (x) = O(x 3 ) and g (x) = O(x 2 ). Then,

f (x) = O(x 3 ), but also f (x) = O(x 2 ) and f (x) = O(x)

3O(x 3 ) − 2O(x 2 ) = O(x 2 )
O(x 3 )O(x 2 ) = O(x 5 )
O(x 3 )
= O(x)
|x|2

Lecture 2 7 / 23
Concepts and theorems from analysis

Lecture 2 8 / 23
 Jacques Hadamard (1865–1963)

Definition: well-posed and ill-posed problems

A mathematical problem is well-posed if it satisfies the following
conditions:
1 A solution exists
2 The solution is unique
3 The solution changes continuously with the data
If a problem fails to satisfy any of these conditions, then it is ill-posed.

Lecture 2 9 / 23
Not all well-posed problems are equally good.

Definition: well-conditioned and ill-conditioned problems/models

A problem/model is called well-conditioned if small errors in the data lead
to small errors in the results. Otherwise the problem/model is
ill-conditioned.

One problem can be better conditioned than the other.

Related concept: condition number of a matrix.

Lecture 2 10 / 23
Definition: modulus of a complex number
Let z ∈ C, i.e., z = a + ib, a, b ∈ R. Then,
p
|z| = |a + ib| = a2 + b 2 .

Let z = a − ib be the complex conjugate of z. Then,

zz = |z|2 = a2 + b 2 .

Lecture 2 11 / 23
 Leopold Kronecker (1823–1891)

Definition: Kronecker’s delta

(
1, i = j,
δi,j =
̸ j.
0, i =

Examples:
Z π
π
sin(mϕ) sin(nϕ) dϕ = δm,n (1 − δn,0 )
0 2
n
X Xn
ai,j δj,k = ai,k , in particular, ai,j δj,i = ai,i
j=1 j=1

Lecture 2 12 / 23
Definition: eigenvalues and eigenvectors
Let A ∈ Cm×m be any matrix and I ∈ Rm×m - the identity matrix. Then,
the eigenvalues λ1 , λ2 , . . . , λm of A satisfy:

det(A − λI ) = 0,

and

Avi = λi vi ,

where vi ∈ Cm is the eigenvector corresponding to the eigenvlaue λi .

Lecture 2 13 / 23
Eigen-decomposition
With A ∈ Rn×n , let V = [v1 , . . . , vn ] ∈ Rn×n be the matrix with the
eigenvectors vi as columns, and Λ = diag(λi , . . . , λn ) ∈ Rn×n - the
diagonal matrix with the eigenvalues λi on its diagonal. Then,

Avi = λi vi ⇔ AV = V Λ.

If V −1 exists, then there exists the eigen-decomposition of A:

A = V ΛV −1 ,

and the matrix A can be diagonalized by V :

V −1 AV = Λ

In general, even if all entries [A]i,j of A are real, i.e., A ∈ Rn×n , the
eigenvalues λi of A can be complex.

Lecture 2 14 / 23
Eigen-decomposition of symmetric matrices

Symmetric matrix is such that AT = A, i.e., [A]i,j = [A]j,i . Among many

other, it has the following properties:
Every symmetric matrix can be diagonalized by an orthogonal matrix
V:

V T AV = Λ, with V T V = I .

The eigenvectors are mutually orthogonal, i.e., viT vj = δi,j

The eigenvalues are real, λi ∈ R, i = 1, . . . , n.

Lecture 2 15 / 23
Triangle inequalities
For any x, y ∈ R:

|x + y | ≤ |x| + |y |,
Z b Z b
f (x) dx ≤ |f (x)| dx
a a

Useful for computing upper bounds. Example of application:

|f (x) − g (x)| = |f (x) − v (x) + v (x) − g (x)| ≤ |f (x) − v (x)| + |v (x) − g (x)|

Lecture 2 16 / 23
Definition: sets of continuous functions
The set of all continuous functions on the interval [a, b] is denoted as
C [a, b]. The set of all functions whose p-th derivative is continuous on
[a, b] is denoted as C p [a, b].

Function f (x) is differentiable at x, if its derivative f ′ (x) exists at x.

All differentiable functions are continuous, C 1 [a, b] ⊂ C [a, b].
Not all continuous functions are differentiable.
Not all differentiable functions have a continuous derivative.

Lecture 2 17 / 23
Theorem: intermediate value
Let f ∈ C [a, b] and f (a) ̸= f (b). For any number F between f (a) and
f (b) there exists a number c ∈ (a, b) such that f (c) = F .

Application: consider f (x1 ) ̸= f (x2 ) and the average:

f (x1 ) + f (x2 )
2
It is clear that the number F = (f (x1 ) + f (x2 ))/2 is between f (x1 ) and f (x2 ).
Hence, from the Intermediate Value Theorem, there is a point ξ such that

f (x1 ) + f (x2 )
= f (ξ)
2

Lecture 2 18 / 23
 Michel Rolle (1652–1719)

Theorem (Rolle):
Let f ∈ C [a, b] and f (x) is differentiable on (a, b). If f (a) = f (b), then
there exists a number c ∈ (a, b) such that f ′ (c) = 0.

Application: Rolle’s Theorem helps finding extrema of a function.

Lecture 2 19 / 23
Theorem: mean value
Let f ∈ C [a, b] and f (x) is differentiable on (a, b). Then, there exists a
number c ∈ (a, b) such that

f (b) − f (a)
f ′ (c) =
b−a

Theorem: mean value for integrals

Let g ∈ C [a, b] and f (x) is integrable and does not change sign on [a, b].
Then, there exists a number c ∈ (a, b) such that
Z b Z b
g (x)f (x) dx = g (c) f (x) dx
a a

Lecture 2 20 / 23
 Karl Weierstrass (1815–1897)

Theorem (Weierstrass): uniform polynomial approximation

Let f ∈ C [a, b]. For every ϵ > 0 there exists a polynomial function P(x)
such that:

|f (x) − P(x)| < ϵ, for all x ∈ [a, b]

Lecture 2 21 / 23
 Brook Taylor (1685–1731)

Theorem: Taylor’s polynomial

Let f ∈ C n+1 [a, b]. Then, for all c, x ∈ (a, b) there exists a number ξ between x
and c such that:

f (x) = Pn (x) + Rn (x),

where
f ′′ (c) f (n) (c)
Pn (x) = f (c) + f ′ (c)(x − c) + (x − c) + · · · + (x − c)n
2! n!
is the Taylor polynomial and

f (n+1) (ξ)
Rn (x) = (x − c)n+1
(n + 1)!

is the remainder.
Lecture 2 22 / 23
Taylor series

We shall often deal with the values of f (x) in neighboring points f (x − 2h),
f (x − h), f (x + h), f (x + 2h), etc. These can be related to f and its derivatives
at x via the Taylor series. We use the Taylor polynomial with c = x and with
x + h, x − h, etc., instead of x, so that x − c becomes x + h − x = h or
x − h − x = −h, etc.

f ′′ (x) 2 f ′′′ (x) 3 f (4) (ξ) 4

f (x + h) = f (x) + f ′ (x)h + h + h + h , ξ ∈ (x, x + h)
2 6 24

4f ′′′ (x) 3 2f (4) (ξ) 4

f (x − 2h) = f (x) − 2f ′ (x)h + 2f ′′ (x)h2 − h + h , ξ ∈ (x − h, x)
3 3
We can also write:
f ′′ (x) 2 f ′′′ (x) 3 f (4) (x) 4
f (x + h) = f (x) + f ′ (x)h + h + h + h + O(h5 )
2 6 24

Lecture 2 23 / 23