Chapter 1

Introduction

1.1 What is Numerical Analysis?

This is an introductory course of Numerical Analysis, which comprises the
design, analysis, and implementation of constructive methods and algorithms
for the solution of mathematical problems.
Numerical Analysis has vast applications both in Mathematics and in
modern Science and Technology. In the areas of the Physical and Life Sci-
ences, Numerical Analysis plays the role of a virtual laboratory by providing
accurate solutions to the mathematical models representing a given physical
or biological system in which the system’s parameters can be varied at will, in
a controlled way. The applications of Numerical Analysis also extend to more
modern areas such as data analysis, web search engines, social networks, and
basically anything where computation is involved.

1.2 An Illustrative Example: Approximating

a Definite Integral
The main principles and objectives of Numerical Analysis are better illus-
trated with concrete examples and this is the purpose of this chapter.
Consider the problem of calculating a definite integral
Z b
I[f ] = f (x)dx. (1.1)
a

3
4 CHAPTER 1. INTRODUCTION

In most cases we cannot find an exact value of I[f ] and very often we only
know the integrand f at finite number of points in [a, b]. The problem is
then to produce an approximation to I[f ] as accurate as we need and at a
reasonable computational cost.

1.2.1 An Approximation Principle

One of the central ideas in Numerical Analysis is to approximate a given
function or data by simpler functions which we can analytically evaluate,
integrate, di↵erentiate, etc. For example, we can approximate the integrand
f in [a, b] by the segment of the straight line, a linear polynomial p1 (x), that
passes through (a, f (a)) and (b, f (b)). That is

f (b) f (a)
f (x) ⇡ p1 (x) = f (a) + (x a). (1.2)
b a
and
Z b Z b
1
f (x)dx ⇡ p1 (x)dx = f (a)(b a) + [f (b) f (a)](b a)
a a 2 (1.3)
1
= [f (a) + f (b)](b a).
2
That is
Z b
b a
f (x)dx ⇡ [f (a) + f (b)]. (1.4)
a 2

The right hand side is known as the simple Trapezoidal Rule Quadrature. A
quadrature is a method to approximate an integral. How accurate is this
approximation? Clearly, if f is a linear polynomial or a constant then the
Trapezoidal Rule would give us the exact value of the integral, i.e. it would
be exact. The underlying question is: how well does a linear polynomial p1 ,
satisfying

p1 (a) = f (a), (1.5)

p1 (b) = f (b), (1.6)

approximate f on the interval [a, b]? We can almost guess the answer. The
approximation is exact at x = a and x = b because of (1.5)-(1.6) and it is
1.2. AN ILLUSTRATIVE EXAMPLE 5

exact for all polynomials of degree  1. This suggests that f (x) p1 (x) =
Cf 00 (⇠)(x a)(x b), where C is a constant. But where is f 00 evaluated
at? it cannot be at x for if it did f would be the solution of a second order
ODE and f is an arbitrary (but sufficiently smooth, C 2 [a, b] ) function so
it has to be at some undetermined point ⇠(x) in (a, b). Now, if we take the
particular case f (x) = x2 on [0, 1] then p1 (x) = x, f (x) p1 (x) = x(x 1),
and f 00 (x) = 2, which implies that C would have to be 1/2. So our conjecture
is
1
f (x) p1 (x) = f 00 (⇠(x))(x a)(x b). (1.7)
2
There is a beautiful 19th Century proof of this result by A. Cauchy. It goes
as follows. If x = a or x = b (1.7) holds trivially. So let us take x in (a, b)
and define the following function of a new variable t as
(t a)(t b)
(t) = f (t) p1 (t) [f (x) p1 (x)] . (1.8)
(x a)(x b)

Then , as a function of t, is C 2 [a, b] and (a) = (b) = (x) = 0. Since

(a) = (x) = 0 by Rolle’s theorem there is ⇠1 2 (a, x) such that 0 (⇠1 ) = 0
and similarly there is ⇠2 2 (x, b) such that 0 (⇠2 ) = 0. Because is C 2 [a, b] we
can apply Rolle’s theorem one more time, observing that 0 (⇠1 ) = 0 (⇠2 ) = 0,
to get that there is a point ⇠(x) between ⇠1 and ⇠2 such that 00 (⇠(x)) = 0.
Consequently,

00 2
0= (⇠(x)) = f 00 (⇠(x)) [f (x) p1 (x)] (1.9)
(x a)(x b)
and so
1
f (x) p1 (x) = f 00 (⇠(x))(x a)(x b), ⇠(x) 2 (a, b). ⇤ (1.10)
2
We can use (1.10) to find the accuracy of the simple Trapezoidal Rule. As-
suming the integrand f is C 2 [a, b]
Z b Z b Z
1 b 00
f (x)dx = p1 (x)dx + f (⇠(x))(x a)(x b)dx. (1.11)
a a 2 a
Now, (x a)(x b) does not change sign in [a, b] and f 00 is continuous so
by the Weighted Mean Value Theorem for Integrals we have that there is
6 CHAPTER 1. INTRODUCTION

⌘ 2 (a, b) such that

Z b Z b
f 00 (⇠(x))(x a)(x 00
b)dx = f (⌘) (x a)(x b)dx. (1.12)
a a

The last integral can be easily evaluated if we shift to the midpoint, i.e.,
changing variables to x = y + 12 (a + b) then
Z b Z b a " ✓ ◆2 #
2 b a 1
(x a)(x b)dx = y2 dy = (b a)3 . (1.13)
a b a
2
2 6

Collecting (1.11) and (1.13) we get

Z b
b a 1 00
f (x)dx = [f (a) + f (b)] f (⌘)(b a)3 , (1.14)
a 2 12
where ⌘ is some point in (a, b). So in the approximation
Z b
b a
f (x)dx ⇡ [f (a) + f (b)].
a 2
we make the error
1 00
E[f ] = f (⌘)(b a)3 . (1.15)
12

1.2.2 Divide and Conquer

The error (1.15) of the simple Trapezoidal Rule grows cubically with the
length of the interval of integration so it is natural to divide [a, b] into smaller
subintervals, apply the Trapezoidal Rule on each of them, and sum up the
result.
Let us divide [a, b] in N subintervals of equal length h = N1 (b a), deter-
mined by the points x0 = a, x1 = x0 + h, x2 = x0 + 2h, . . . , xN = x0 + N h = b,
then
Z b Z x1 Z x2 Z xN
f (x)dx = f (x)dx + f (x)dx + . . . + f (x)dx
a x0 x1 xN 1

X1 Z xj+1
N (1.16)
= f (x)dx.
j=0 xj
1.2. AN ILLUSTRATIVE EXAMPLE 7

But we know
Z xj+1
1 1 00
f (x)dx = [f (xj ) + f (xj+1 )]h f (⇠j )h3 (1.17)
xj 2 12

for some ⇠j 2 (xj , xj+1 ). Therefore, we get

Z  N 1
b
1 1 1 3 X 00
f (x)dx = h f (x0 ) + f (x1 ) + . . . + f (xN 1 ) + f (xN ) h f (⇠j ).
a 2 2 12 j=0

The first term on the right hand side is called the Composite Trapezoidal
Rule Quadrature (CTR):

1 1
Th [f ] := h f (x0 ) + f (x1 ) + . . . + f (xN 1 ) + f (xN ) . (1.18)
2 2

The error term is

N 1
" N 1
#
1 3 X 00 1 1 X 00
Eh [f ] = h f (⇠j ) = (b a)h2 f (⇠j ) , (1.19)
12 j=0 12 N j=0

where we have used that h = (b a)/N . The term in brackets is a mean

value of f 00 (it is easy to prove that it lies between the maximum and the
minimum of f 00 ). Since f 00 is assumed continuous (f 2 C 2 [a, b]) then by the
Intermediate Value Theorem, there is a point ⇠ 2 (a, b) such that f 00 (⇠) is
equal to the quantity in the brackets so we obtain that
1
Eh [f ] = (b a)h2 f 00 (⇠), (1.20)
12
for some ⇠ 2 (a, b).

1.2.3 Convergence and Rate of Convergence

We do not not know what the point ⇠ is in (1.20). If we knew, the error could
be evaluated and we would know the integral exactly, at least in principle,
because

I[f ] = Th [f ] + Eh [f ]. (1.21)
8 CHAPTER 1. INTRODUCTION

But (1.20) gives us two important properties of the approximation method

in question. First, (1.20) tell us that Eh [f ] ! 0 as h ! 0. That is, the
quadrature rule Th [f ] converges to the exact value of the integral as h ! 0
1
. Recall h = (b a)/N , so as we increase N our approximation to the
integral gets better and better. Second, (1.20) tells us how fast the approx-
imation converges, namely quadratically in h. This is the approximation’s
rate of convergence. If we double N (or equivalently halve h) then the
error decreases by a factor of 4. We also say that the error is order h2 and
write Eh [f ] = O(h2 ). The Big ‘O’ notation is used frequently in Numerical
Analysis.

Definition 1.1. We say that g(h) is order h↵ , and write g(h) = O(h↵ ), if
there is a constant C and h0 such that |g(h)|  Ch↵ for 0  h  h0 , i.e. for
sufficiently small h.

Example 1.1. Let’s check the Trapezoidal Rule approximation for an integral
we can compute exactly. Take f (x) = ex in [0, 1]. The exact value of the
integral is e 1. Observe how the error |I[ex ] T1/N [ex ]| decreases by a

Table 1.1: Composite Trapezoidal Rule for f (x) = ex in [0, 1].

N T1/N [ex ] |I[ex ] T1/N [ex ]| Decrease factor
4
16 1.718841128579994 5.593001209489579 ⇥ 10
32 1.718421660316327 1.398318572816137 ⇥ 10 4 0.250012206406039
64 1.718316786850094 3.495839104861176 ⇥ 10 5 0.250003051723810
128 1.718290568083478 8.739624432374526 ⇥ 10 6 0.250000762913303

factor of (approximately) 1/4 as N is doubled, in accordance to (1.20).

1.2.4 Error Correction

We can get an upper bound for the error using (1.20) and that f 00 is bounded
in [a, b], i.e. |f 00 (x)|  M2 for all x 2 [a, b] for some constant M2 . Then

1
|Eh [f ]|  (b a)h2 M2 . (1.22)
12
1
Neglecting round-o↵ errors introduced by finite precision number representation and
computer arithmetic.
1.2. AN ILLUSTRATIVE EXAMPLE 9

However, this bound does not in general provide an accurate estimate of the
error. It could grossly overestimate it. This can be seen from (1.19). As
N ! 1 the term in brackets converges to a mean value of f 00 , i.e.
N 1 Z b
1 X 00 1 1
f (⇠j ) ! f 00 (x)dx = [f 0 (b) f 0 (a)], (1.23)
N j=0 b a a b a

as N ! 1, which could be significantly smaller than the maximum of |f 00 |.

Take for example f (x) = e100x on [0, 1]. Then max |f 00 | = 10000e100 , whereas
the mean value (1.23) is equal to 100(e100 1) so the error bound (1.22)
overestimates the actual error by two orders of magnitude. Thus, (1.22) is
of little practical use.
Equation (1.19) and (1.23) suggest that asymptotically, that is for suffi-
ciently small h,

Eh [f ] = C2 h2 + R(h), (1.24)

where
1 0
C2 = [f (b) f 0 (a)] (1.25)
12
and R(h) goes to zero faster than h2 as h ! 0, i.e.

R(h)
lim = 0. (1.26)
h!0 h2

We say that R(h) = o(h2 ) (little ‘o’ h2 ).

Definition 1.2. A function g(h) is little ‘o’ h↵ if

g(h)
lim =0
h!0 h↵

and we write g(h) = o(h↵ ).

We then have

I[f ] = Th [f ] + C2 h2 + R(h). (1.27)

and, for sufficiently small h, C2 h2 is an approximation of the error. If it

is possible and computationally efficient to evaluate the first derivative of
10 CHAPTER 1. INTRODUCTION

f at the end points of the interval then we can compute directly C2 h2 and
use this leading order approximation of the error to obtain the improved
approximation
1 0
Teh [f ] = Th [f ] [f (b) f 0 (a)]h2 . (1.28)
12
This is called the (composite) Modified Trapezoidal Rule. It then follows from
(1.27) that error of this “corrected approximation” is R(h), which goes to
zero faster than h2 . In fact, we will prove later that the error of the Modified
Trapezoidal Rule is O(h4 ).
Often, we only have access to values of f and/or it is difficult to evaluate
f (a) and f 0 (b). Fortunately, we can compute a sufficiently good approxi-
0

mation of the leading order term of the error, C2 h2 , so that we can use the
same error correction idea that we did for the Modified Trapezoidal Rule.
Roughly speaking, the error can be estimated by comparing two approxima-
tions obtained with di↵erent h.
Consider (1.27). If we halve h we get
1
I[f ] = Th/2 [f ] + C2 h2 + R(h/2). (1.29)
4
Subtracting (1.29) from (1.27) we get
4 4
C2 h2 = Th/2 [f ] Th [f ] + (R(h/2) R(h)) . (1.30)
3 3
The last term on the right hand side is o(h2 ). Hence, for h sufficiently small,
we have
4
C2 h2 ⇡ Th/2 [f ] Th [f ] (1.31)
3
and this could provide a good, computable estimate for the error, i.e.
4
Eh [f ] ⇡ Th/2 [f ] Th [f ] . (1.32)
3
The key here is that h has to be sufficiently small to make the asymptotic
approximation (1.31) valid. We can check this by working backwards. If h
is sufficiently small, then evaluating (1.31) at h/2 we get
✓ ◆2
h 4
C2 ⇡ Th/4 [f ] Th/2 [f ] (1.33)
2 3
1.2. AN ILLUSTRATIVE EXAMPLE 11

and consequently the ratio

Th/2 [f ] Th [f ]
q(h) = (1.34)
Th/4 [f ] Th/2 [f ]
should be approximately 4. Thus, q(h) o↵ers a reliable, computable indicator
of whether or not h is sufficiently small for (1.32) to be an accurate estimate
of the error.
We can now use (1.31) and the idea of error correction to improve the
accuracy of Th [f ] with the following approximation 2
4
Sh [f ] := Th [f ] + Th/2 [f ] Th [f ] . (1.35)
3

1.2.5 Richardson Extrapolation

We can view the error correction procedure as a way to eliminate the
leading order (in h) contribution to the error. Multiplying (1.29) by 4 and
substracting (1.27) to the result we get
4Th/2 [f ] Th [f ] 4R(h/2) R(h)
I[f ] = + (1.36)
3 3
Note that Sh [f ] is exactly the first term in the right hand side of (1.36) and
that the last term converges to zero faster than h2 . This very useful and
general procedure in which the leading order component of the asymptotic
form of error is eliminated by a combination of two computations performed
with two di↵erent values of h is called Richardson’s Extrapolation.
Example 1.2. Consider again f (x) = ex in [0, 1]. With h = 1/16 we get
✓ ◆
1 T1/32 [ex ] T1/16 [ex ]
q = ⇡ 3.9998 (1.37)
16 T1/64 [ex ] T1/32 [ex ]
and the improved approximation is
4
S1/16 [ex ] = T1/16 [ex ] +
T1/32 [ex ] T1/16 [ex ] = 1.718281837561771 (1.38)
3
which gives us nearly 8 digits of accuracy (error ⇡ 9.1 ⇥ 10 9 ). S1/32 gives
us an error ⇡ 5.7 ⇥ 10 10 . It decreased by approximately a factor of 1/16.
This would correspond to fourth order rate of convergence. We will see in
Chapter 8 that indeed this is the case.
2
The symbol := means equal by definition.
12 CHAPTER 1. INTRODUCTION

It appears that Sh [f ] gives us superior accuracy to that of Th [f ] but at

roughly twice the computational cost. If we group together the common
terms in Th [f ] and Th/2 [f ] we can compute Sh [f ] at about the same compu-
tational cost as that of Th/2 [f ]:
" 2N
#
h 1 X1 1
4Th/2 [f ] Th [f ] = 4 f (a) + f (a + jh/2) + f (b)
2 2 j=1
2
" N
#
1 X1 1
h f (a) + f (a + jh) + f (b)
2 j=1
2
" N
#
h X1 N
X1
= f (a) + f (b) + 2 f (a + kh) + 4 f (a + kh/2) .
2 k=1 k=1

Therefore
" N
#
h X1 N
X1
Sh [f ] = f (a) + 2 f (a + kh) + 4 f (a + kh/2) + f (b) . (1.39)
6 k=1 k=1

The resulting quadrature formula Sh [f ] is known as the Composite Simpson’s

Rule and, as we will see in Chapter 8, can be derived by approximating the
integrand by quadratic polynomials. Thus, based on cost and accuracy, the
Composite Simpson’s Rule would be preferable to the Composite Trapezoidal
Rule, with one important exception: periodic smooth integrands integrated
over their period.

Example 1.3. Consider the integral

Z 2⇡
dx
I[1/(2 + sin x)] = . (1.40)
0 2 + sin x

Using Complex Variables techniques

p (Residues) the exact integral can be com-
puted and I[1/(2 + sin x)] = 2⇡/ 3. Note that the integrand is smooth (has
an infinite number of continuous derivatives) and periodic in [0, 2⇡]. If we
use the Composite Trapezoidal Rule to find approximations to this integral
we obtain the results show in Table 1.2.
The approximations converge amazingly fast. With N = 32, we already
reached machine precision (with double precision we get about 16 digits).
1.3. SUPER-ALGEBRAIC CONVERGENCE 13

Table 1.2: Composite Trapezoidal Rule for f (x) = 1/(2 + sin x) in [0, 2⇡].
N T2⇡/N [1/(2 + sin x)] |I[1/(2 + sin x)] T2⇡/N [1/(2 + sin x)]|
8 3.627791516645356 1.927881769203665 ⇥ 10 4
16 3.627598733591013 5.122577029226250 ⇥ 10 9
32 3.627598728468435 4.440892098500626 ⇥ 10 16

1.3 Super-Algebraic Convergence of the CTR

for Smooth Periodic Integrands
Integrals of periodic integrands appear in many applications, most notably,
in Fourier Analysis.
Consider the definite integral
Z 2⇡
I[f ] = f (x)dx,
0

where the integrand f is periodic in [0, 2⇡] and has m > 1 continuous deriva-
tives, i.e. f 2 C m [0, 2⇡] and f (x + 2⇡) = f (x) for all x. Due to periodicity
we can work in any interval of length 2⇡ and if the function has a di↵erent
period, with a simple change of variables, we can reduce the problem to one
in [0, 2⇡].
Consider the equally spaced points in [0, 2⇡], xj = jh for j = 0, 1, . . . , N
and h = 2⇡/N . Because f is periodic f (x0 = 0) = f (xN = 2⇡) and the CTR
becomes
 N
X1
f (x0 ) f (xN )
Th [f ] = h + f (x1 ) + . . . + f (xN 1 ) + =h f (xj ). (1.41)
2 2 j=0

Being f smooth and periodic in [0, 2⇡], it has a uniformly convergent Fourier
Series:
1
a0 X
f (x) = + (ak cos kx + bk sin kx) (1.42)
2 k=1

where
Z
1 2⇡
ak = f (x) cos kx dx, k = 0, 1, . . . (1.43)
⇡ 0
Z
1 2⇡
bk = f (x) sin kx dx, k = 1, 2, . . . (1.44)
⇡ 0
14 CHAPTER 1. INTRODUCTION

Using the Euler formula3 .

eix = cos x + i sin x (1.45)

we can write

eix + e ix
cos x = , (1.46)
2
eix e ix
sin x = (1.47)
2i
and the Fourier series can be conveniently expressed in complex form in terms
of functions eikx for k = 0, ±1, ±2, . . . so that (1.42) becomes
1
X
f (x) = ck eikx , (1.48)
k= 1

where
Z 2⇡
1 ikx
ck = f (x)e dx. (1.49)
2⇡ 0

We are assuming that f is real-valued so the complex Fourier coefficients

satisfy c̄k = c k , where c̄k is the complex conjugate of ck . We have the
relation 2c0 = a0 and 2ck = ak ibk for k = ±1, ±2, . . ., between the complex
and real Fourier coefficients.
Using (1.48) in (1.41) we get

N 1
!
X1 X
Th [f ] = h ck eikxj . (1.50)
j=0 k= 1

Justified by the uniform convergence of the series we can exchange the finite
and the infinite sums to get
1 N
X1 2⇡
2⇡ X
Th [f ] = ck eik N j . (1.51)
N k= 1 j=0

3 2
i = 1 and if c = a + ib, with a, b 2 R, then its complex conjugate c̄ = a ib.
1.3. SUPER-ALGEBRAIC CONVERGENCE 15

But
N
X1 X1 ⇣
N ⌘j
ik 2⇡ j ik 2⇡
e N = e N . (1.52)
j=0 j=0

2⇡
Note that eik N = 1 precisely when k is an integer multiple of N , i.e. k = lN ,
l 2 Z and if so
X1 ⇣
N
2⇡
⌘j
eik N =N for k = lN . (1.53)
j=0

Otherwise, if k 6= lN , then
⇣ ⌘N
ik 2⇡
X1 ⇣
N
2⇡
⌘j 1 e N

eik N = ⇣ 2⇡ ⌘ = 0 for k 6= lN (1.54)

j=0 1 eik N

Using (1.53) and (1.54) we thus get that

1
X
Th [f ] = 2⇡ clN . (1.55)
l= 1

On the other hand

Z 2⇡
1 1
c0 = f (x)dx = I[f ]. (1.56)
2⇡ 0 2⇡
Therefore

Th [f ] = I[f ] + 2⇡ [cN + c N + c2N + c 2N + . . .] , (1.57)

that is

|Th [f ] I[f ]|  2⇡ [|cN | + |c N| + |c2N | + |c 2N | + . . .] , (1.58)

So now, the relevant question is how fast the Fourier coefficients clN of f
decay with N . The answer is tied to the smoothness of f . Doing integration
by parts in the formula (4.11) for the Fourier coefficients of f we have
Z 2⇡
1 1 2⇡
ck = f 0 (x)e ikx dx f (x)e ikx 0 k 6= 0 (1.59)
2⇡ ik 0
16 CHAPTER 1. INTRODUCTION

ikx
and the last term vanishes due to the periodicity of f (x)e . Hence,
Z
1 1 2⇡ 0
ck = f (x)e ikx dx k=6 0. (1.60)
2⇡ ik 0
Integrating by parts m times we obtain
✓ ◆m Z 2⇡
1 1
ck = f (m) (x)e ikx
dx k 6= 0, (1.61)
2⇡ ik 0

where f (m) is the m-th derivative of f . Therefore, for f 2 C m [0, 2⇡] and
periodic
Am
|ck |  , (1.62)
|k|m

where Am is a constant (depending only on m). Using this in (1.58) we get


2 2 2
|Th [f ] I[f ]|  2⇡Am m
+ m
+ + ...
N (2N ) (3N )m
 (1.63)
4⇡Am 1 1
= 1 + m + m + ... ,
Nm 2 3

and so for m > 1 we can conclude that

Cm
|Th [f ] I[f ]|  . (1.64)
Nm
Thus, in this particular case, the rate of convergence of the CTR at equally
spaced points is not fixed (to 2). It depends on the number of derivatives
of f and we say that the accuracy and convergence of the approximation is
spectral. Note that if f is smooth, i.e. f 2 C 1 [0, 2⇡] and periodic, the CTR
converges to the exact integral at a rate faster than any power of 1/N (or
h)! This is called super-algebraic convergence.