August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica...

ws-book9x6Book2 page i

Publishers page

i
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page ii

Publishers page

ii
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page iii

Publishers page

iii
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page iv

Publishers page

iv
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page v

To Adil Yaqub and all my other teachers

v
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page vi
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page vii

Preface

This book is an introduction to numerical analysis. It is intended for third

or fourth year undergraduates or beginning master level students. The re-
quired background includes multivariate calculus and linear algebra. Some
knowledge of real analysis is recommended. In terms of programming, some
programming experience would be a plus but is not required. Indeed, the
book is self-contained in this respect. But, the treatment is likely too fast
for some beginners.
The programming environment is Mathematica, a 4GL with an advanced
symbolic manipulation component. We make an effort to keep the program-
ming component simple. In particular, we use as little of the programming
language as possible. We are currently using Version 10. However, this
material has been developed and used over a period of years. There are
some programming changes or additions included in more recent versions.
But none of these affect the programs included in this text. With Version
10, Mathematica is also referred to as Wolfram Language.
The purpose is to introduce numerical analysis. Since the post WWII
period, [Grcar (2011)], numerical analysis has been the mathematics sup-
porting large calculations carried out on a computer. Hence, a course in
this topic must include computational exercises and these should be large
enough or sufficiently complex to warrant the use of a computer. It is bet-
ter still if there are realistic problems; ones that the student can imagine
would arise in an actual application.
One dominant application of numerical techniques concerns simulat-
ing processes represented by differential equations. In this setting, we are
charged with estimating the existent but unknown solution to a differential
equation. We have organized the topics of this book to introduce some of
the classical approaches to this problem. In particular, we develop finite

vii
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page viii

viii Elements of Numerical Analysis with Mathematica

difference method (FDM) for a parabolic equation in one spatial variable,

including explicit, implicit and Crank-Nicolson FDM. In addition, we touch
on stability. In another direction, we present some of the elementary tech-
niques used to simulate the solution of an ordinary differential equation.
Monte Carlo method is another means to simulate a differential equation.
We present the basics of this method.
Another area of current interest is certain Big Data applications. The
solution procedures for problems that arise in this area include solving
large linear systems of equations. Generally, these systems are too large for
Gauss-Jordan elimination. We will develop two basic solutions procedures
including Gauss-Seidel. In this context, we introduce the student to the
basics of Krylov subspaces. Another aspect of Big Data applications is
optimization, multivariate max/min problems. This is also an important
part of our development. We include both the greatest descent and Hessian
variants.
We present the techniques of numerical analysis together with the sup-
porting theory. Much of this book is organized in traditional definition,
theorem, proof format. With a small number of exceptions, the theory is
self-contained within this text and prerequisites. In the exceptional cases,
the necessary supporting material is identified and referenced. But, full un-
derstanding of the material requires both knowledge of the mathematical
foundations and hands on experience. In this regard, we include examples
and exercises that are beyond what can be done with pencil and paper. In
this way, we emphsasize the natural link between numerical analysis and
computing.
We present the material of the text in the given order. It is intended
to be a one semester course. Programming in Mathematica is sufficiently
intuitive for students with math or engineering background that little or
no programming background is necessary. In fact, the readers experience
manually solving problems provides the necessary foundation toward pro-
gramming sophisticated processes. At the same time, we find it useful to
schedule computer lab time for the first few weeks of the semester. In this
facility, we can provide one-on-one programming support. By mid-semester
all students are on an even footing programming-wise.
For most students this class is a rewarding experience. In this setting,
they are able to solve problems that are realistic in scale and complexity.
To this point, their experience is often restricted to problems that can be
executed easily with pencil and paper.
This text arose from our need of a beginning numerical analysis text
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page ix

Preface ix

that was sufficiently mathematical for our students and supported Mathe-
matica as the programming platform. During that time, our students have
seen topics come and go as we settled on a stable course. Without their par-
ticipation this text could not have been written. Special acknowledgement
goes to those who helped me understand how to present this material. In
particular this includes Scott Irwin, Yevgeniy Milman, Andrew Hofstrand,
Evan Curcio, Gregory Javens and James Kluz.

John Loustau
Hunter College (CUNY)
New York, 2015
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page x
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page xi

Contents

Preface vii

1. Beginnings 1
1.1 The Programming Basics for Mathematica . . . . . . . . . 2
1.2 Errors in Computation . . . . . . . . . . . . . . . . . . . . 6
1.3 Newtons Method . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . 14

2. Linear Systems and Optimization 19

2.1 Linear Systems of Equations . . . . . . . . . . . . . . . . . 20
2.2 The Norm and Spectral Radius of a Linear Transformation 27
2.3 Large Matrix Techniques . . . . . . . . . . . . . . . . . . . 32
2.4 Functions of Several Variables: Finding Roots and Extrema 37

3. Interpolating and Fitting 43

3.1 Polynomial Interpolation . . . . . . . . . . . . . . . . . . . 44
3.2 Bezier Interpolation . . . . . . . . . . . . . . . . . . . . . 53
3.3 Least Squares Fitting . . . . . . . . . . . . . . . . . . . . . 56
3.4 Cubic Splines and B-Splines . . . . . . . . . . . . . . . . . 62
3.5 Hermite Interpolation . . . . . . . . . . . . . . . . . . . . 71

4. Numerical Differentiation 75
4.1 Finite Differences and Vector Fields . . . . . . . . . . . . 76
4.2 Finite Difference Method, Explicit or Forward Euler . . . 81
4.3 Neumann Stabiity Analysis . . . . . . . . . . . . . . . . . 85
4.4 Finite Difference Method, Implicit and Crank Nicolson . . 91

xi
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page xii

xii Elements of Numerical Analysis with Mathematica

5. Numerical Integration 97
5.1 Trapezoid Method and Simpsons Rule . . . . . . . . . . . 98
5.2 Midpoint Method . . . . . . . . . . . . . . . . . . . . . . . 101
5.3 Gaussian Quadrature . . . . . . . . . . . . . . . . . . . . . 105
5.4 Comments on Numerical Integration . . . . . . . . . . . . 110

6. Numerical Ordinary Differential Equations 117

6.1 First order ODE Techniques, Forward Euler and Corrector
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2 Midpoint Method with an Application to Mathematical
Oncology . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.3 Shooting Method with an Application to Cooling Fins . . 125
6.4 The Method of Lines, Revisiting the Heat Equation . . . . 128

Bibliography 131
Index 133
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 1

Chapter 1

Beginnings

Introduction

This chapter provides a brief introduction to programming for those who

have never programmed. For those with programming experience, this is
the introduction to Mathematica. One of the several advantages to using
a 4GL such as Mathematica is that it makes numerical methods accessible
to all students with multivariate calculus and linear algebra. Indeed, most
students catch on very quickly to programming in Mathematica and are
doing complicated programs well before the end of the semester course.
To support the learning process, there are tutorials available by selecting
Help from the system menu and then Documentation Center. The first
item available in documentation contains the basic tutorials. For those
who prefer hard copy references, there are several textbooks available from
online booksellers.
A second feature of this chapter is to introduce the reader to the quirks
of Mathematica. Mathematica was originally developed by mathematicians,
and therefore it has a mathematicians point of view. If your background
is with C++ or another of the computer scientist developed programming
products, you will find Mathematica to be similar on the surface but sig-
nificantly different at lower levels. If you have never programmed in an
interpreted 4GL, then you have something to get used to.
In this chapter, we introduce the terminology associated to computer
error. Computers must represent decimal values in a finite number of signif-
icant digits. Therefore, the representation is often only close to the actual
value. For instance 1/3 is not 0.3333333. During normal arithmetic op-
erations, the error inherent in the representation is sometimes magnified.
In extreme cases, this may yield ridiculous results. When using any com-

1
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 2

2 Elements of Numerical Analysis with Mathematica

puter system you most always be cognizant of the potential of error in your
calculations. We will see an example of this in Section 1.2.
We next look at Newtons method. Most calculus courses include New-
tons method for finding roots of differentiable functions. If you have done
a Newtons method problem with pencil and paper, you know that doing
two or three iterations of the process is a nightmare. Even the simplest
cases are not the sort of thing most students want to do. Now, we see that
it is easy to program. In this regard, it is an excellent problem for the be-
ginning student. In addition, Mathematica provides a built in function that
performs Newtons method. It is empowering for the student to compare
his results to the output produced by Mathematica. We follow Newtons
method by the secant method to find the root of a function. This provides
the student with the first example of an error estimating procedure.
By the end of the chapter, the student should be able to program the
basic arithmetic operations, access the standard mathematical functions,
program loops and execute conditional statements (if ... then ... else ...). A
special feature of Mathematica is the graphics engine. With minimal effort,
the student can display sophisticated graphical output. By the end of this
chapter the student will be able to use the basic 2D graphics commands.

1.1 The Programming Basics for Mathematica

We use Mathematica Version 10. Each year when the university renews
its license, the version changes. In the past, programs for one version are
either fully upgradable to the subsequent version or Wolfram provides a
program that upgrades program code written for one version to the next.
During this semester you will be programming in Mathematica. To
begin with, you will learn to be able to program the following.

(1) The basic arithmetic operations (addition, subtraction, multiplication,

division, exponentiation and roots.)
(2) Define a function
(3) Loops (Do Loop or While Loop)
(4) Conditionals (If ... then ... else )
(5) Basic graphics (point plot, function plot, parametric plot)

We begin with item 1. You add two numbers, 3 + 5, subtract 3 5,

multiply 3 5 and divide 3/5. Alternatively, if a = 3 and b = 5, then
a + b, a b, a b and a/b have exactly the same result. For exponents,
ab = 243 and ba = 125. The usual math functions cosine, Cos[x], sine,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 3

Beginnings 3

Sin[x], exponential, Exp[x] and so forth all begin with a capital letter. The
argument is enclosed in square brackets.
To start, bring up Mathematica and select new notebook. Now type a
line or two of program code. For instance any two of the statements in the
prior paragraph. To execute the code, you hold the shif t key down and
then press enter.
Consider the polynomial in 2 variables, f (x, y) = (x + y)2 2xy y 2 .
You define this in a Mathematica program with the following statement,

f[x_,y_] = (x+y)^2 - 2*x*y - y^2

You can find descriptions of each of these in the first two or three chapters
of most any programming text for Mathematica. In addition Mathematica
includes a programming tutorial accessible via the Help Menu.
There are some comments that need to be made.
A. Error messages in Mathematica are cryptic at best. After a while
you will begin to understand what they mean and use them to debug your
program. But this will take some experience. On the other hand there
are circumstances where you might expect to receive an error or warning
message but none is generated. For instgance,

If [x == 0,
x = 5
];

will test the value of x. If it is zero, then it will be set to 5. On the other
hand,

If [x = 0,
x = 5
];

sets x to zero and then to 5, no matter what value x may have.

B. When multiplying two variables denoted by a and b, you may write
a b or a b (with a space between). But if you write ab (without a space),
then Mathematica will think that ab is a new variable and not the product
of two variables. Because of how Mathematica spaces input, it is not always
easy to distinguish between a b and ab. You will save yourself a lot of time
if you always use the asterisk for multiplication.
C. You may be used to using the three grouping symbols, parentheses,(
), square brackets,[ ], and braces, {}, interchangeably. In Mathematica you
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 4

4 Elements of Numerical Analysis with Mathematica

may not do this. Parenthesis may only be used in computations for group-
ing. Square brackets are only used around the independent variables of a
function, while braces are only used for vectors and matrices. For instance
each of the following expressions will cause an error in a Mathematica pro-
gram:
[a + b]2 , f (x), (x, y).
The correct expressions are
(a + b)2 , f [x], {x, y}.
D. When defining a function in your program, you always follow the in-
dependent variable(s) with an underscore. This is how Mathematica iden-
tifies the independent variable. Later when you reference the function, you
must not use the underscore. For instance the following statements defines
a function as xex , evaluates the function at x = 1 and then defines a
second function as the derivative of the first.

f [x ] = x Exp[x];

y = f [1];

g[x ] = D[f [x], x];

Once you have used a letter or group of letters for the independent
variable in a function definition, it is best to not use it for any other purpose.
For instance, we have used x as the independent variable for both f and
g. We can now write f [4] to evaluate f at 4 and we can write a = 4, f [a].
But if we write x = 4 then we will have a clash. On the one hand x is the
independent variable for f , and on the other hand, it is a constant 4.
E. All reserved words in Mathematica begin with upper case letters.
When you define a function or a variable, it is best to use names that begin
with lower case letters. This way functions and variables you define will not
clash with the ones Mathematica has reserved. For instance, P i is 3.14....,
and I is Sqrt[1]. Any attempt to use these symbols for any other purpose
will at the very least cause your output to be strange.
F. There are two ways to indicate the end of a line of program code.
First, you may simply hit the return key and go to the next line. Second,
you may end the line with a semicolon. The result upon execution is lightly
different. In the first case the result of the line of code is printed when the
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 5

Beginnings 5

code is executed. In the second the output of the line of code is suppressed.
For instance, if you enter
a=3+5
and execute (press shift+enter), Mathematica will return the value, 8. On
the other hand if you type
a = 3 + 5;
and execute, then there is no printed output. In any event, the calculation
does occur and the result is stored in a. Any subsequent calculation can
acess the result by referencing a.
G. It is best to have only one line of program code per physical line
on the page. For short programs, violating this rule should not cause any
problems. For long and involved programs, debugging is often a serious
problem. If you have several lines of code on the same physical line, may
have trouble noticing a particular line of code that is causing an error. For
instance,
z = x + y;
x = z + 1;
is preferred,
z = x + y; x = z + 1;
H. Mathematica has distinct computational procedures for integer arith-
metic and decimal arithmetic. Integer calculations take much longer to ex-
ecute. But the result expressed as fractions is exact. Decimal calculations
are much faster but there is round off error. (See Section 2.) For instance, if
all the data for a program is whole numbers and is entered without decimal
point, then Mathematical will assume that the calculations are integer and
proceed. (See problmes 1 and 2 below.
I. We did not use := when defining the funciton f (x, y). There are
technical differences between the two symbols used to define a function.
Different authors will suggest one or the other. Out take on this is that
:= is used when defininig a module, a function defined as a sub-program
and accessed at several different locations in your program. Otherwise, to
decfine a simple function as we have done, it is best to use =. That said, it
is unlikely that you will see the difference in the contexts that arise here.

Exercises:
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 6

6 Elements of Numerical Analysis with Mathematica

1. Write a program that defines the function f (x) = x2 + x2 and

evaluates f at x = 2 and x = 3. Use the Mathematica function Print to
display the output.

2. Repeat Exercise 1 for x = 2.0 and x = 3.0.

3. Define g = f 0 , the derivative of f . Evaluate g at 5 and 5.0.

1.2 Errors in Computation

Errors arise from several sources. There are the errors in data collection
and data tabulation. This sort of data error needs to be avoided as much
as possible. Usually this is accomplished by quality assurance procedures
implemented at the team management level. This is not our concern in
numerical methods. Programming errors are also a quality control issue.
These errors are avoided by following good practices of software engineering.
For our own programs, we are best advised to be as simple as possible:
simple in program design, simple in coding. A mundane program is much
easier to control and modify than a brilliant but somewhat opaque one.
The simple one may take longer to code or longer to execute, but within
bounds is still preferable.
There are errors that arise because of the processes we use and the
equipment that we execute on. Both are errors due to the discrete nature
of the digital computer. These errors cannot be prevented and hence must
be controlled via error estimation.
First, the computer cannot hold an infinite decimal. Hence, the decimal
representation of fractions such as 1/3 and 2/3 are inherently incorrect.
Further, subsequent computations using these numbers are incorrect. A
small error in the decimal representation of a number when carried for-
ward through an iterated process may accumulate and result in an error of
considerable size. For instance, when solving a large linear system of equa-
tions, an error introduced in the upper left corner will iterate through the
Gauss-Jordan process causing a significant error in the lower right corner.
Another type of error arises from discrete processes. For instance, sup-
pose you have an unknown function f (x). Suppose also that you know that
f (1) = 1 and f (1.1) = 1.21. Then it is reasonable to estimate the derivative
by the Newton quotient
df 1.21 1 0.21
(1) = = 2.1.
dx 1.1 1 0.1
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 7

Beginnings 7

If in fact, f (x) = x2 , then our estimated derivative is off by 0.1. But without
knowing the actual function, we have no choice but to use the estimation.
We are faced with one of two alternatives, doing nothing or proceeding
with values that we expect are flawed. The only reasonable alternative is
to proceed with errors provided we can estimate the error.
This is a special case of a more general problem. Suppose we want
to compute a value y, but in fact, we can only compute the values at a
sequence yn that converges to y. Since we can never do our computation
all the way to the limit, then we must have a means to estimate the nth ,
x xn .
We formalize the error in the following definition.
Definition 1.2.1. Suppose that there is a computation and that estimates
a value x with the computed value x , then e = x x is called the error .
In turn, |x x | is called the absolute error , If x 6= 0, then (x x )/x is
called the relative error . In this case, the relative absolute error is given
by |x x
|/|x|.
It is reasonable to ask why we should care about e if we already know
x. Indeed, if we know x, there is no need to for a numerical technique to
estimate x with x . In numerical analysis, the basic assumption is that x
is not computeable but estimatelble. Therefore, it is useful to have precise
definitions for these terms, as there are situations where we can estimate
the error e without knowing the actual value x. Indeed, each numerical
process should include a procedure to estimate the error. It can be argued
that any procedure that does not include an error estimate is of no value.
What is the purpose of executing a computation, if we have no idea whether
or not the computed data approximates the actual value?
A second comment is in order. It is preferable to use the relative or
relative absolute error. This is because these values are dimensionless. For
instance, consider the example of the derivative of the squaring function.
If the data is given in meters, then e = 0.1meters. If the data were instead
displayed in kilometers then e = 0.001 and or centimeters, e = 10. Even
though the error is the same, the impression is different. However, for
relative error the value is 0.1, is independent of the unit of measurement.
When this is the case, we say that the data is dimensionless.
In Exercise 1 below, you are asked to execute a simple calculation which
should always yield the same result independent of the input data. In this
problem you are asked to use several different values of x. Unexpectedly,
the results will vary across a broad spectrum of possible answers. In a
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 8

8 Elements of Numerical Analysis with Mathematica

simple calculation like this it is possible to look at how the numbers are
represented and determine why the error occurs. But in any actual situation
the calculations are so complex that such an analysis is virtually impossible.
When executing a computation, you should always have a idea of what
the results should be. If you get impossible output and there is no error
in your program, then you may be looking at small errors in numerical
representation compounded over perhaps thousands of separate arithmetic
operations.

Exercises:
1. Consider the function f (x, y) = ((x + y)2 2xy y 2 )/x2 . We expect
that if x 6= 0, then f (x, y) = 1. Set y = 103 and compute f for x = 10.01 ,
10.02 , 10.03 , 10.04 , 10.05 , 10.06 , 10.07 , 10.08 . For each value of x
compute the absolute error.

2. Repeat Problem 1 with g(x, y) = (x + y)2 /x2 2xy/x2 y 2 /x2 . Why

are the results different? Is g[x, 103 ] the same function as f [x, 103 ]?

3. Repeat Problem 1 using x = 101 , 102 , 103 , 10(4 , 105 , 106 ,

107 , 108 . Why are the results different?

1.3 Newtons Method

Suppose that you have a function f (x) = y and want to find a root or a
zero for f . Recall that x
is a root of f provided f (
x) = 0. If f is continuous
and f (x1 ) > 0 and f (x2 ) < 0 then you know that f must have at least one
root between x1 and x2 . This result, The Intermediate Value Theorem, is
usually stated as early as Calculus 1 and most commonly proved in the first
semester of Real Analysis. [Rudin (1976)] There is an intuitively simple
but inefficient means to determine a good approximation for x based on
this theorem.
(1) Consider the midpoint of the interval [x1 , x2 ], (x1 + x2 )/2 = x
.
(2) If f (
x) = 0, then x
is a root. Exit.
(3) If f (
x) > 0, then replace x1 with x.
(4) If f (
x) < 0, then replace x2 with x.
(5) Return to Step 1.
The following Mathematica code segment demonstrates this process. We
use the fact that a b > 0 if and only if a and b have the same sign.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 9

Beginnings 9

test-root = x1;
test-value = 10^-5;
While[Abs[f[test-root]] > test-value,
test-root = (x1 + x2)/2;
If[f[test-root]*f[x1] >= 0,
x1 = test-root,
x2 = test-root
];
];
Print[test-root];

In this code fragment, we assume that f , x1 and x2 are already known.

Notice that we use 105 as the kickout threshold. In particular, as soon as
the absolute value of f at the current test-root is less than 105 , the process
stops and the approximate root is printed. Since the procedure may only
approximate the solution, then we do need a kickout threshold. the value
f (
x) is called the residual , and we say that we are using a residual kickout
test. Alternatively, if xn and xn+1 are two successive computed values, then
|xn xn+1 | < 105 may be used to end the computation. The idea behind
this kickout test is that the sequence of computed approxiomate roots is
convergent and hence Cauchy. In particular, |xn xn+1 | 0.
There are two statements in the Mathematica program that are new to
us. First there is an If Then Else statement. This is called a conditional
statement. The statement has a condition. And following the condition is a
block of program statements to be executed if the condition holds followed
by a comma and then a block of program statements to be executed if the
condition fails. In this case the condition is that the the sign of f at x1
and the current approximate root is the same. If the condition is satisfied,
then replace x1 by the current value. If it is false, then repalce x2 by the
current value.
There is also a While Loop. Following the While there is a condition.
The While loop will execute as long as the condition holds. When it fails,
the program will exit the loop. In this case the condition compares the
aboslute residual to the test value.
A more efficient means to locate an approximate root is called Newtons
method . You probably saw this in Calculus. In this case, we suppose that
f is differentiable and that we want to find a root for f near x1 . The idea is
that the tangent line for f at x1 is a good approximation for f near x1 . The
equation for the tangent is easily solved to loacte a root, that we call x2 . If
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 10

10 Elements of Numerical Analysis with Mathematica

x2 is closer to the root of f than x1 , then the method has been productive.

1 1

1 2 3 4 5 1 2 3 4 5

A
-1 -1

B
-2 -2

-3 -3

Fig. 1.1 Figure 1.3.1 f (x) = x cos(x) Fig. 1.2 Figure 1.3.2 f and the tangent
line at B = (2, f (2))

For instance, if f (x) = x cos(x), then there is a root at the point A near
x = 1.6. (See Figure 1.3.1)
If we start the process with x = 2, f 0 (2) 2.2347, f (2) 0.8233
and the tangent to f at 2 is given by h(x) = f 0 (2)(x 2) + f (2). Now
h crosses the x-axis at 1.62757. Figure 1.3.2 shows f together with the
tangent.
If we write this out formally, then starting at x1 , the tangent line has
slope f 0 (x1 ) and passes through the point (x1 , f (x1 )). Hence,
y f (x1 )
f 0 (x1 ) =
.
x x1
Setting y = 0 and solving for x, we get
f 0 (x1 )(x x1 ) = f (x1 ),
or
f (x1 )
x = x1 . (1.3.1)
f 0 (x1 )
Replacing x1 by x we get an iterative process that we can repeat until
the absolute value of |f (x1 )| is less then some threshold value. We will call
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 11

Beginnings 11

(1.3.1) the operative statement. The following steps provide an outline for
the program.

(1) Provide an initial estimate for the root, x1 .

(2) Set up a while loop, use a residual threshold test to kickout.
(3) Put the operative statement inside the loop.
(4) Close the loop.
(5) Print the result.

The initital estimate for the root is often called the seed.
Newtons method is implemented in Mathematica via the FindRoot com-
mand. For instance, f (x) = x cos(x) has a root between x = 1 and 3. The
following Mathematica statement will implement Newtons method to get
an approximate value for the root.

x-root = FindRoot[x*Ccos[x] == 0, {x, 2}];

If you do a search on FindRoot in Mathematica help, you will see that

there are options available to the programmer. Among the options is the
opportunity to set the residual kickout threshold or use the default kickout
value imposed by Mathematica. In addition, you may set the maximal num-
ber of iterations for the process or accept the Mathematica preset iteration
count limit. The idea is to stop the process after a set number of iterations
no matter the value of the residual.
We can add an iteration count limit to our basic Newtons method pro-
gram. To accomplish this, we will need to set two new variables, iter-limit
and iter-cnt. Suppose we want to stop the process after 10000 iterations
no matter what the value of the residual. We accomplsih this with a com-
pound conditional on the While statement. Then we need only add 5 new
statements to our program. In Mathematica this would look like,

While[f[x]>10^{-5} && iter-cnt < iter-limit,

This reads, while the residual is greater then 105 or the iterations is less
than 10000.

(1) Provide a seed, an initial estimate for the root, x1 .

(2) Set iter-limit = 10000.
(3) Set iter-cnt = 0
(4) Set up a while loop as outlined above. Use a residual threshold test to
kickout plus an iteration count limit.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 12

12 Elements of Numerical Analysis with Mathematica

(5) Put the operative statement inside the loop.

(6) Add 1 to iter-cnt
(7) Close the loop.
(8) Print the result.
(9) Print iter-cnt

Before passing on, several words of caution are in order. First, if

f 0 (x1 ) = 0 then the process will fail. Indeed, in this case, the tangent
line is parallel to the x-axis. Second, if f 0 (x1 ) is not zero, but nearly zero,
then x given by (1.3.1) will be far from x1 . Indeed, for f (x) = x cos(x),
x1 = 0.9, then the slope of f at x1 is approximately 0.0833 and the process
will locate the root near 8. (See Figure 1.3.3.) He, it is important that
f 0 (x1 ) be bounded away from zero.

6 1.0

4 0.8

2
C 0.6

2 4 6 8
0.4

-2

0.2
-4

-6 -1.0 -0.5 0.5 1.0

-8

Fig. 1.3 Figure 1.3.3 f and the tangent Fig. 1.4 Figure 1.3.4 A Newtons
line at C = (x1 , f (x1 )) method, cyclic order 2

Finally, it is possible that Newtons method will fail to find any ap-
proximate root. Indeed, the process may cycle. In particular, starting at
a value x1 you may pass on to a succession of values x2 , x3 , ..., xn only to
have xn = x1 . Once you are back to the original value, then the cycle is set
and further processing is useless. The following diagram shows a function
f where f (1) = f (1) = 1, f 0 (1) = (1/2) and f 0 (1) = 1/2. Hence,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 13

Beginnings 13

x1 = 1, x2 = 1, x3 = 1 and so forth. The example is constructed using

a Bezier curve. It is indeed a function but not the sort of thing one would
expect under normal circumstances. Later when we develop Bezier curves
we will revisit this example. It is now apparant why an iteration count
kickout is essential.

Exercises:
1. Consider the function f (x) = x2 . We know that f has a root at
zero. Suppose we select the seed to be 0.5 and the threshold to be 105 .
Execute a program in Mathematica to estimate the root. Since you already
know the outcome, this sort of example is a good means to verify that your
program is correct.

2. Plot the function f (x) = xex 0.16064.

a. Use FindRoot to locate a root near x = 3.
b. Write a program in Mathematica implementing Newtons method.
Use your program to approximate the root near x = 3. Set the maximal
number of iterations to 100 and the residual kickout threshold to 105 .
How many iterations does your program actually execute before it stops?
c. Redo (b) with a kickout threshold set to 102 . Using the result of
(a) as the actual and this result as the computed, calculate the relative
absolute error.

3. Figure 1.3.5 shows the graph of f (x) = x/(x2 + 1) together with the
point (1.5, f (1.5)).
a. Use FindRoot to solve f (x) = 0 starting at 1.5. What happens?
Why?
b. Write your own program to execute Newtons method starting at
x = 1.5. What is the output for the first 10 iterations?
c. Plot f along with the tangent at the 4th iteration. Put both plots
on the same axis. (Hint: Execute both plots separately and save the out-
put in a variable. Then execute the Show statement. The syntax of these
statements is exlained in the language help.)

4. Consider the function f (x) = cos(ex ).

a. Define the function and plot it on the interval [-1, 4].
b. Locate a possible root, identify the seed and approximate the root
via your implementation of Newtons method. Coimpare your estimate to
FindRoot.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 14

14 Elements of Numerical Analysis with Mathematica

0.4

0.2

-15 -10 -5 5 10 15

-0.2

-0.4

Fig. 1.5 Figure 1.3.5 f and the tangent line at C = (x1 , f (x1 ))

1.4 Secant Method

The secant method is a second technique to estimate the zero of a function.

It is closely related to Newtons method. On the one hand, it is less efficient,
while on the other, it is free of the anomalies which may arise with Newtons
method. In particular, it will always locate an approximate root and the
root will be in the intended vicinity. To describe the secant method, we
continue the notation begun in the last section.
For the secant method we begin with a known positive and negative
value for the function. As with Newtons method, we call these points the
seeds. Suppose that the function is given by f (x) = y. If f is continuous
and f (x1 ) > 0 and f (x2 ) < 0 then we know that f must have at least
one root between x1 and x2 . The secant method with seeds x1 and x2 will
estimate that root.
The method is simple. It proceeds by considering the line connecting
(x1 , f (x1 )) and (x2 , f (x2 )). We let x
denote the point where the line (or
secant) intersects the x-axis. If f ( x) = 0, then we have the root between
x1 and x2 . If f (x) > 0, then we replace x1 with x and proceed. Otherwise,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 15

Beginnings 15

we replace x2 with x .
For instance if f (x) = x cos(x), then there is a root between 1 and 2.
Setting x1 = 1 and x2 = 2, then the line is given by (x) = [(f (2)
f (1))/(2 1)](x 1) and x = 1.39364. The following diagram shows the
graph of f along with the secant.

0.5 0.5

0.5 1.0 1.5 2.0 2.5 3.0 0.5 1.0 1.5 2.0 2.5 3.0

-0.5 -0.5

-1.0 -1.0

-1.5 -1.5

-2.0 -2.0

-2.5 -2.5

-3.0 -3.0

Fig. 1.6 Figure 1.4.1: f with the se- Fig. 1.7 Figure 1.4.2: f concave down
cant joining (1, f (1)) and (2, f (2)) near the root

Returning to the general procedure, the points (x, y) on the secant must
satisfy

f (x2 ) f (x1 ) y f (x1 )

= .
x2 x1 x x1
Setting y = 0 and solving for x, we get
 x2 x1 
x = x1 f (x1 ) . (1.4.1)
f (x2 ) f (x1 )
Equation (1.4.1) is the operative statement in the sense that any program
that implements the secant method must include this statement. The basic
structure of a progeram for this mehtod method is the same as the program
for Newtons method. In large measure, we need only to replace (1.3.1) for
Newtons method with (1.4.1) for the secant method.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 16

16 Elements of Numerical Analysis with Mathematica

However, there is one wrinkle. In Newtons method we have a single

seed which is replaced at each iteration by the current approximate root.
For the secant method there are two seeds, x1 and x2 . When you compute
x using (1.4.1) we must replace one of seeds before you compute the next
x. In this case, you will need to evaluate f at the three points, x, x1 and
x2 . We know that f (x1 ) and f (x2 ) have different signs. If f (x) and f (x1 )
have the same sign, then replace x1 , if they have different sign (hence, f (x)
and f (x2 ) have the same sign), then replace x2 .
The secant method is in fact an approximate form of Newtons method.
You see this by rewriting (1.4.1) as

f (x1 )
x = x1 f (x2 )f (x1 )
. (1.4.2)
x2 x1
If |x2 x1 | is small, then the denominator on the right hand side,
(f (x2 ) f (x1 ))/(x2 x1 ) is very near to f 0 (x1 ). As the iterative process
proceeds, we should expect the successive values for x1 and x2 to converge
together. In this case the expression (1.4.2) for the approximate root via
the secant method will converge to the expression given in (2.1.1) when
using Newtons method.
We turn now to error estimation. Suppose f is decreasing and concave
down in the interval [x1 , x2 ], as is the case for the currnet example f (x) =
x cos(x). Let x denote the approximate root derived from Newtons method
and let x denote the approximate root derived from the secant method. It
is easy to see that x x and that the actual root must lie between. Figure
2.4.2 illustrates this for the example case. We state this result formally in
the following theorem.
Theorem 1.4.1. Consider a twice differentiable real valued function f de-
fined on an interval [a, b]. Suppose that f has a root at x in the interval but
no relative extrema or inflection points, then the following holds. Each New-
tons method estimate x and secant method estimate x x
, satisfies x x
or xx x .
Proof. There are four cases to consider. We will do the proof for the case
when f is decreasing and concave up. If a < x1 < x , then f (x1 ) > 0 and
= x1 f (x1 )/f 0 (x1 ) > x1 . We denote the tangent for f at x1 by h.
x
Since f is concave up, then h decreases faster than f in the interval [x1 , x
].
Therefore, xx .
On the other side, given seeds x1 < x2 , we know that f (x1 ) > 0,
f (x2 ) < 0. By definition, the secant estimate x
is no larger than x2 . Let
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 17

Beginnings 17

k denote the secant. Since, f is concave up, then for each x [x1 , x2 ],
f (x) < k(x). Therefore, f ( x
x) < 0. It follows that x x2 . This
completes the proof for this case.

Next, we derive our first example of an error estimating procedure. In

particular, |
xx| |
xx |. Similarly, |
xx| |
xx|. Hence, |
xx| is
an upper bound on the absolute error. In other words, we may use | xx |
to estimate the error without knowledge of x .

Theorem 1.4.2. Suppose that f is twice differentiable function on [a, b]

with a root in the interval. Furthermore, suppose that f has no extrema or
changes of concavity in the interval. If x
is the Newtons mdthod estimate
of the root and x
is the secant method estimate, then the absolute error for
either procedure is bounded by |xx|.

The secant method is also implemented in Mathematica via the Find-

Root command . For the secant method, we need to supply two seeds. For
instance, we know that f (x) = x cos(x) has a root between x = 1 and 2.
The following Mathematica statement will implement the secant method to
return an approximate value for the root.
x-root = FindRoot[ x Cos(x) == 0, {x, 1, 2} ];
Again when using this option of the FindRoot statement, you have access
to the iteration count and kick out threshold.

Exercises:

1. Consider f (x) = (x 1)2 2 which has a root at 1 + 2. Use the
secant method with seeds at 2 and 3 to estimate this root.

2. Execute Newtons method for the function in Exercise 1. Compute

the error estimate and verify that actual absolute error is less than the
estimated absolute error.

3. Consider the function f (x) = xex 0.16064.

a. Use FindRoot to locate a root between x = 2 and x = 3.
b. Write a program in Mathematica implementing the secant method.
Use your program to locate a root near x = 3. Set the maximal number of
iterations to 100 and the kickout threshold to 105 .
c. How many iterations does your program actually execute before it
stops? How does the secant method compare to Newtons method?
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 18

18 Elements of Numerical Analysis with Mathematica

d. Use the result of Exercise 1.b from the previous work on Newtons
method along with the result of the prior section to get a upper bound on
the absolute error.

4. Complete the proof of Theorem 3.4.1

5. For the example in Exercise 1, use ListPlot to display 5 Newtons

method estimates along with 5 secant method estimates. Use different sym-
bols for the two plots.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 19

Chapter 2

Linear Systems and Optimization

Introduction

In this chapter we develop the basic procedures necessary to solve linear

systems of equations and locate functional the extrema. We introduce both
standard techniques and their extensions for large matrices. This includes
methods for finding eigenvalues, solving linear systems of equations and
optimization problems. We resolve these questions in the context of a course
supported by a 4GL programming language. We have all solved max/min
problems and linear systems of equations with pencil and paper. For the
most part we do this for carefully selected textbook exercises. You may
have found these problems to be tedious. Now, we find that Mathematica
makes these problems effortless and routine. Even in cases with thousands
of unknowns or hundreds of thousands of unknowns.
In particular, we begin with the Mathematica implementation of Gauss-
Jordan elimination. Next we consider techniques for solving large linear
systems. In this context we develop Jacobi and Gauss-Seidel methods as
special cases of the Richardson method. We follow this with the power
method for finding eigenvalues and eigenvectors. At the same time, we
introduce Krylov subspaces including a brief discussion of CG or conjugate
gradient method.
Our point of view is to provide an introduction to some of the ideas
that are necessary to deal with large data sets. An important aspect of
this development is the operator norm introduced in the second section.
This result is arguably one of the two most important of this book. We
encounter the other in the next chapter.
During this development we have occassion to apply the Gelfand theo-
rem on the spectral radius of a matrix. We will reference this result without

19
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 20

20 Elements of Numerical Analysis with Mathematica

proof as the usual argument requires the Jordan canonical form of a matrix.
We end the chapter considering max/min problems in several variables.
Along the way we show that finding the roots of a function of several vari-
ables may be recast as a max/min problem. In addition, these procedures
can be used to solve linear systems with singular coefficient matrix. Both
questions relate to the material developed in the prior chapter.

2.1 Linear Systems of Equations

Here we develop procedures for solving a linear system of equations with

non-singular coefficient matrix. The case for singular coefficient matrices
will be presented later in the chapter.
First we need to consider the program code necessary to do matrix
arithmetic. The following statements define and display a four dimensional
column vector, v.

v = {1,2,3,4} ;
Print[MatrixForm[v]];

The next two stdatements define 4 by 4 matrices A and B. The fol-

lowing statement multiplies the two matrices, then prints the product, C,
formatted as a matrix. The final step multiplies A times v to get the 4-tuple
w. Finally, we multiply w by a scalar 5. Notice that we use a period to mul-
tiply matrices and matrices times vectors whereas we use the * to multiply
scalars and vectors. Even though the entries of these matrices and vectors
are integers we use at least one decimal point to ensure all processing will
be decimal processing.

A = {{1.,2,3,4},{2,3,4,5},{5,4,3,2},{4,3,2,1}};
B = {{0.,1,0,0},{1,0,0,0},{0,0,0,1},{0,0,1,0}};
C = A.B;
v = {1.,2,3,4};
Print[MatrixForm[C]];
w = A.v;
w = 5*w;

Some other useful linear algebra functions are T ranspose[A],

T ranspose[v], Inverse[A], IdentityM atrix[n], N orm[v] and Length[v].
The first one produces the transpose of the matrix A. The next one changes
the column vector v into a row vector. The next two produce the inverse of
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 21

Linear Systems and Optimization 21

A, and the n n identity matrix. N orm[v] returns the length of the vector
v while Length[v] returns 4 if v is a four-tuple. Finally, A[[i]][[j]] = A[[i, j]]
returns the ij th entry of A and v[[i]] is the ith entry of the vector v.
Consider a linear system Ax = b, where A is a non-singular nn matrix
and x, b Rn . The condition that A is non-singular assures us that the
system has an unique solution given by x = A1 b. In addition we know
that A is non-singular if and only if A is row equivalent to the nn identity,
In .
The standard process of solving a linear system of equations is called
Gauss-Jordan elimination. This method is implemented in Mathematica.
Suppose you have the linear system

123 x1 1
4 5 6 x2 = 2
789 x3 3
In order to solve this system in Mathematica, you define the coefficient
matrix and constant vector via

coefMat = {{1,2,3},{4,5,6},{7,8,9}};
conVec = {1,2,3};

And then solve the system via the statement.

solVec = LinearSolve[coefMat, conVec];

Along with Gauss-Jordan elimination (LinearSolve), Mathematica will

also return the LUDecomposition. In order to understand what Mathemat-
ica does when solving a linear system, we need to look at LUDecomposition.
At this stage, you need to recall a number of things about row equivalence
and elementary row operations as they apply to linear systems of equations.

(1) Matrices A and B are row equivalent provided there exist elementary
Qm
matrices E1 , ..., Em with B = ( i=1 Ei )A.
(2) Multiplication on the left by an elementary matrix implements the cor-
responding elementary row operation.
(3) There are three elementary row operations.
(4) The type-1 elementary row operation exchanges two rows of a matrix.
It is denoted by E(i,j) , indicating that rows i and j are exchanged.
(5) The type-2 elementary row operation multiplies a row by a nonzero
scalar. It is denoted by E(i) , indicating that row i is multiplied by .
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 22

22 Elements of Numerical Analysis with Mathematica

(6) The type-3 elementary row operation adds a scalar times one row to
another. It is denoted by E(i)+j , indicating that times row i is
added to row j.
(7) All elementary matrices are non-singular. Their inverses are given by
1 1 1
E(i,j) = E(i,j) . E(i) = E1 (i) and E(i)+j = E(i)+j . Notice that
the inverse of an elementary matrix is the elementary matrix that re-
verses the operation.
(8) If Ax = b and Bx = c are linear systems and D is a product of ele-
mentary matrices then B = DA, c = Db implies that the two systems
have the same solution. In addittion, for each type, the inverse is of
the same type.

Gauss-Jordan elimination hinges on the fact that if A were upper triangular,

then the system, Ax = b could be easily solved. Indeed, the row-echelon
form of A is upper triangular. The U in LU -Decomposition refers to this
upper triangular matrix. The problem of solving the linear system reduces
to finding an upper triangular matrix U , row equivalent to A.
Suppose A = [i,j ] and that 1,1 6= 0. Then it is a simple matter to
clear out the entries of the first column of A below the 1, 1 entry. For this
purpose we only need elementary operations of type-3. The corresponding
elementary matrices, E(1)+j are all lower triangular since each j > 1. Since
the product of lower triangular matrices is lower triangular, then we have
a lower triangular matrix L1 with L1 A = A1 = [ i,j ] where
i,1 = 0 for all
entries of column 1 below the 1, 1 entry. Furthermore, L1 is nonsingular.
Now we move to the second column. If (2,2) 6= 0, then we can clear
out below the 2, 2 position using only type-3 elementary operations. As
before, there is a lower triangular matrix L2 with L2 L1 A = L2 A1 = A2 .
As before, L2 L1 is lower triangular and nonsingular. If the 3, 3 entry of A3
is non-zero, then the process continues. Indeed, if at each step the next
diagonal entry is not zero, then we clear out below the diagonal element.
At the end we have

(Ln ...L2 L1 )A = U (2.1.1)

where each Li is lower triangular and nonsingular and U is upper triangular.

Multiplying (2.1.1) by (Ln ...L1 )1 = L1 1
1 ...Ln = L yields the relation
A = LU , where L is lower triangular (the inverse of a lower triangular
matrix is lower triangular) and U is upper triangular. This is the LU -
decomposition for A. As long as we do not encounter a zero on the main
diagonal, the process is a descrived.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 23

Linear Systems and Optimization 23

Now, we only need consider what to do if there is a diagonal entry that

is zero. In this case, if the i, i entry is zero. there must be a non-zero entry
below the diagonal. If this were not the case then A would be singular. See
Exercise 2. For instance if we began with

123
4 8 6
759
Then upon clearing the first column, we have

1 2 3
0 0 6 .
0 9 12
Now, if after applying a type-1 elementary operation, E(2,3) , we have the
upper triangular matrix

1 2 3
0 9 12 = E(2,3) (E7(1)+3 E4(1)+2 )A.
0 0 6
In particular, whenever we encounter a zero on the diagonal, then we must
introduce a type-1 elementary matrix which interchanges the rows to bring
a nonzero entry to the diagonal.
Looking more closely at the product of a type-1 and a type-3 we see
easily that
E(i,j) E(s)+t = E(s)+t E(i,j)
provided s, t are not elements of the set {i, j}. Otherwise
E(i,j) E(i)+t = E(j)+t E(i,j) (2.1.2)
and
E(i,j) E(s)+i = E(s)+j E(i,j) .
For Gauss-Jordan elimination, only the second case (2.1.2) can occur
with both i and j greater than s. Hence, we may restate (2.1.1) as
Ln ...L2 L1 Pm ...P2 P1 A = U, (2.1.3)
where the Pj are type-1 elementary matrices. This brings us to the following
theorem.
Theorem 2.1.1. If A is a non-singular n n matrix, then there exists a
lower triangular matrix L, an upper triangular matrix U and a nonsingular
matrix P with P A = LU . The upper triangular matrix U is the row echelon
form of A, L is a product of type-3 elementary matrices and P is a product
of type-1 elementary matrices.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 24

24 Elements of Numerical Analysis with Mathematica

The matrix, P , which is the product of type-1 matrices, is called a

permutation matrix. The idea here is simple. Each type-1 elementary op-
eration transposes two rows of the matrix A. In turn, the product composes
transpositions into a permutation of the rows. Hence, the term, permuta-
tion matrix.
For any nonsingular matrix A, this information is available via the Math-
ematica function LU Decomposition. For instance, for the matrix,

123
A = 4 8 6
759
LUDecomposition[A]
will return the following output.
{{{1,2,3},{7,-9,-12},{4,0,-6}},{1,3,2},1}.
The first item in the list is a 3 3 matrix. The upper triangular part of this
matrix is U. The lower triangular part (after placing 1s along the diagonal)
is L. Now we know that

1 2 3 100
U = 0 9 12 , L = 7 1 0 .
0 0 6 401
Next {1, 3, 2} is interpreted as the permutation that sends 1 1, 2 3,
3 2. Hence,

100
P = 0 0 1 .
010
The final entry in the output is called the condition number. This number is
useful when estimating the error. We develop this in the following section.

Exercises:
1. Apply LUDecompostion to the following matrices. Write out L, U
and P.Which matrices
are ill conditioned?
1. 1. 0
a. 1. 1. 3.
0. 1. 1.

1. 2. 1. 7.
2. 0. 1. 4.
b.
1. 0. 2. 5.
1. 2. 3. 11.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 25

Linear Systems and Optimization 25

1. 2. 3.
c. 1. 1. 1.
5. 7. 9.
2. Let A = [i,j ] be an n n matrix with real entries. Suppose that
there is an m with i,j = 0 for i m, j m and i,i 6= 0 for 1 i < m.
Prove that A is singular.
3. The Mathematica statement Eigensystem[A] returns n+1 vectors for
an n n matrix A. The entries of the first vector are the eigenvalues of A.
The remaining vectors are the corresponding eigenvectors. Apply Eigensys-
tem to the matrices listed in (1). Recall that a matrix is singular if it has
zero as an eigenvalue. When looking at computer output a number close to
zero should be considered zero. Which of the matrices in (1) are singular?

4. Prove the following for an n n triangular matrix,

a. the product of upper (lower) triangular matrices is upper (lower)
triangular,
b. the inverse of a triangular matrix is a triangular matrix of the same
type.

5. Build a 500 500 matrix A = [i,j ] with diagonal entries i,i = 1,

super diagonal entries i,i1 = 0.5 and subdiagonal entries i+1,i = 0.5.
(Hint: A = IdentityM atrix[500] IdentityM atrix[500] will return a zero
matrix of the desired size. From this point you only need to load Ai,j = 1
for each i and so forth.)
a. Execute LinearSolve for the linear system Ax = b where bi = 1 for
every i. Use the TimeUsed function to determine the required CPU time.
b. Repeat exercise a for a 100 100, ..., 400 400. Plot CPU time
against the matrix size.

6. In a forest two species of tree dominate, TA and TB . Tree TA is

endangered and rumored to be on the edge of extinction. The forest man-
ager is charged with determining the distribution of the two trees in the
forest over the coming years. From a recent inventory carried out for over
an extended period, the following data is known.
A. In a given year the manager can expect 1 percent of TA to die and 5
pecent of TB .
B. There is a 25 percent chance that a dead tree is replaced by the
endangered TA and a 75 pecent chance that it will be replaced by TB .
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 26

26 Elements of Numerical Analysis with Mathematica

C. All dead trees are replaced by one or the other type of tree. New
trees enter the forest only by replacement.
D. Currently, there are 10 TA and 990 TB .
E. Since all collected data is annual, the manager sets the unit of time
to be 1 year. Note that tA ( ) = total(TA ) at a given time, and tB ( ) is
set similarly.
To solve the problem the manager has done the following
For a given year, the number of dead trees is (0.01)tA ( ) + (0.05)tB ( ).
The number of replacements from the type TA is

(0.25)(0.01)tA ( ) + (0.25)(0.05)tB ( ).

The type TB replacements is

(0.75)(0.01)tA ( ) + (0.75)(0.05)tB ( )

.
Unfortunately the manager has resigned leaving you to finish the study.
He has left no contact information.
a. Write tA ( + 1) in terms of tA ( ) and tB ( ). (Hint: What factors
determine tA ( + 1). These are: no TA dies or there are deaths of either
type that are replaced by TA or there are deaths of TA that are replaced
by TB . Which factors are positive, which are negative?)
b. Write tB ( + 1) in terms of tA ( ) and tB ( ).
c. Recast the results of a and b in matrix form; that is find A such that
   
tA ( + 1) t ( )
=A A .
tB ( + 1) tB ( )
d. Use Eigensystem[A] to find the eigenvectors and eigenvalues of A.
Confirm that A has a diagonal representation. We next recast the problem
for a diagonal for of A.
e. Write
 
tA (0)
tB (0)
as a linear combination of the eigenvectors of A.
f. Write
 
tA (1)
tB (1)
using the result of e and the eigenvalues of A.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 27

Linear Systems and Optimization 27

g. Use the result of f to write a formula for

 
tA ( + 1)
tB ( + 1)

in terms of tA ( ) and tB ( ), the eigenvectors and the eigenvalues.

h. Will TA disappear from the forest? If so, when? Plot tA over the
next 25 years.
i. Is it reasonable to expect that the matrix A will not change in the
next 25 years?

2.2 The Norm and Spectral Radius of a Linear Transfor-

mation

We begin with the operator norm of the linear transformation. This concept
is key to estimating the error in Gauss-Jordan elimination. But that is only
the beginning. The operator norm is essential to understanding iterative
processes with linear transformations. Hence, it plays a critical role in large
matrix processes. Recast into infinite dimensional vector spaces, the linear
operators with bounded norm are central. For instance they play a key role
in quantum mechanics and ln linear partial differential equations.
We will develop the concept to real vector spaces. The same arguments
work for the compex case.
Recall the norm of a vector in Rn , kvk = ( i vi2 )1/2 . We extend this
P

idea to a lnear operator in the following definition.

Definition 2.2.1. The operator norm of a linear transformation A of a

finite dimensional vector V is the least constant M which satisfies kAvk
M kvk for any v in V . It is usual to write kAk for the operator norm.

Notice that the norm of A as defined here is independent of the matrix

representation.
Our initial task is to prove that kAk exists. Toward this end, we develop
notation. First let e1 , ..., en denote the standard basis vectors. Here, ei is
the n tuple with 1 in position i and zeros elsewhere. If v is a unit vector,
Pn Pn
v = i=1 i ei and 1 = kvk2 = i=1 2 , since the ei form an orthonormal
frame. Hence, for each i, |i | 1. We calculate
n
X n
X n
X
kAvk = kA( i ei )k = k i Aei k |i |kAei k.
i=1 i=1 i=1
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 28

28 Elements of Numerical Analysis with Mathematica

Taking m = maxi kAei k. we have proved that

kAvk nm, (2.2.1)
for the case of a unit vector v. We now prove the existence theorem for the
operator norm.

Theorem 2.2.1. Any linear transformation of a finite dimensional real or

complex vector space has finite operator norm.

Proof. We continue the notation. Given u 6= 0 in V , v = u/kuk is a unit

vector. By (2.2.1), kAvk K where K = mn. Hence, for nonzero u,
1

kAuk = kukkA u k = kukkAvk kukK. (2.2.2)
kuk
The norm is the least K that satisfies (2.2.2).

Note that the theorem proves the existnece of the norm without pro-
viding a means to calculete it. Rather, we a given an upper bound.
In this section, we have aready seen two functions that we call a norm,
the Euclidean norm of a vector in Rn and now the norm of A. These are
both examples of a more general concept.

Definition 2.2.2. Let V be a real or complex vector space. A norm on V

is a function k.k taking values in R such that
(1) for any nonzero vector, kvk > 0,
(2) for any scalar , kvk = ||kvk,
(3) for u and v in V , ku + vk kuk + kvk.
In this case V is called a normed linear space.

Property (3) is called the triangle inequality.

The following theorem states that the operator norm is a norm.

Theorem 2.2.2. The vector space of linear transformations of the space

V with the operator norm is a normed linear space.

Proof. See Exercise 2.

A norm gives rise to a metric and with a metric we can talk about
convergence, open sets and continuous functions.

Definition 2.2.3. Given a set X, a metric is a real valued function d :

X X R satirfying the following.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 29

Linear Systems and Optimization 29

(1) For distinct x and y in X, d(x, y) > 0 and d(x, x) = 0.

(2) For x and y in X, d(x, y) = d(y, x).
(3) For x, y and z, d(x, y) + d(y, z) > d(x, z).

The last property is also called the triangle inequality . It is not difficult
to prove (see Exercise 8.) that kxyk = d(x, y) defines a metric on a normed
linear space. Before moving on, note that as a consequence of Theorem
2.2.1, linear transformations are continuous. We state this formally in the
following theorem.

Theorem 2.2.3. A linear transformation A defined on a real vector space

Rn is uniformly continuous.

Proof. Take  > 0 and select = /kAk. By Theorem 2.2.1,

kAu Avk = kA(u v)k kAkku vk < ,
provided ku vk < .

We now turn to the error analysis for the solution to a linear system.
Consider the linear system Av = b. We denote the solution by v and the
computed solution v. In turn, we set b = A
v . Now, we compute the relative
normed error,
kv vk kA1 (b b)k kb bk
= kA1 k .
kvk kvk kvk
Next, we multiply top and bottom by kbk and use kbk kAkkvk,
kv vk kb bkkbk
kA1 k
kvk kvkkbk

kb bkkvk kb bk
kAkkA1 k = kAkkA1 k .
kvkkbk kbk
The right hand expression is called the relative normed residual and the
coefficient C = kAkkA1 k is the condition number .
We state this result formally as a theorem.

Theorem 2.2.4. For the linear system Av = b, the relative normed error
is bounded by the condition number times the relative normed residual.

There is no problem to compute the relative normed residual. The

condition number (equivalently the operator norm) is another story. It is
related to the eigenvalues of A. We investigate this relationship in the
exercises.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 30

30 Elements of Numerical Analysis with Mathematica

If the condition number is very large, then the coefficient matrix for the
system is singular or nearly singular and the results returned by LinearSolve
are not considered reliable. When this is the case, Mathematica will return
a warning that the coefficient matrix is ill conditioned. But all of this
preseuposes that the matrices are not too large. For large matrices we
are on our own. Indeed, large nonsingular matrices may have very large
condition numbers. See Exercies 6,
Because of Theorem 2.2.3, we need to develop the operator norm in
order to estimate the condition number. To begin, note that kAkkBk
kABk. And there are cases where equality fails. (See Exercises 1, 3 and
4) Therefore, in order to know the condition number, you must know the
operator norm. Before proceeding, we need some terminology.

Definition 2.2.4. For a real or complex square matrix A, the spectral

radius is the maxi |i | where i are the eigenvalues of A. The spectral
radius of A is denoted (A).

Alternatively, we write (A) for the absolute value of the smallest eigen-
value. The proof of the following theorm is included in the exercises.

Theorem 2.2.5. Let A be an n n real nonsingular matrix. Then the

following hold.
i. If is an eigenvalue for A, then || kAk.
ii. In particular, (A) kAk and 1/(A) kA1 k.
iii. (A)/(A) C, the condition number for A.

Proof. See the Exercise 3.

For the case of a real symmetric matrix the situation is much more
tractable. Indeed, it is a happy circumstance that linear systems with
symmetric coefficieint matrices arise naturally. In particular, this is the
case for processes such as diffusion. The key here is that a real symmetric
matrix has n orthonormal eigenvectors.

Theorem 2.2.6. Let A be an n n real symmetric matrix, then the fol-

lowing hold.
i. The norm of A is equal to the spectral radius.
ii. kAk k = kAkk for any positive integer k.

Proof. See exercise 7

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 31

Linear Systems and Optimization 31

There is one more result that will be useful as we go forward. The proof
uses the Jordan canonical form. We state the theorem here without proof.
It is attributed to Gelfand. [Loustau (2016)]

Theorem 2.2.7. For any n n complex matrix,

lim kAk k1/k = (A) (2.2.3)

k

Exercises:
1. Prove that for real or complex matrices, kAkkBk kABk

2. Prove Theorem 2.2.2. (Hint: for part three consider k(A + B)vk =
kAv + Bvk kAvk + kBvk)

3. The following exercises lead to an estimator for the condition number

of a nonsingular matrix A.
a. Let v be an eigenvector for A with eigenvalue . Prove that ||Av|| =
||||v||. Prove that || ||A||.
b. Prove that is an eigenvalue for A if and only if 1/ is an eigenvalue
for A1 . (Hint: consider v = Av and multiply by A1 .)
c. Let (A) be the absolute value of the smallest eigenvalue of A. Prove
that (A) ||A|| and 1/(A) ||A1 ||.
d. Prove that (A)/(A) C, the condition number of A.

4. Prove that kA|k > 1 while (A) = 1, where

 
11
A= .
01

5. Repeat Exercise 5 of Section 2.1. But now execute LU decomposition.

Look at the condition number as a function of the matrix size. What is
happening? Are these matrices nonsingular? Do they have a zero eigen-
value?

6. For the matrices in Exercise 5, set the frist rowk to (1, 0, 0, ..., 0) and
the last row to (0, 0, ..., 1). Check the condition number. Also use Eigen-
values to verify that the matrices are nonsingular.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 32

32 Elements of Numerical Analysis with Mathematica

7. Prove Theorem 2.2.5 as follows. Supose that A is nn real symmetirc

with orthonormal eigenvectors v1 , ..., vn .
P
a. Let v be a unit vector with v = i i vi where the vi are orthonormal
P 2
eigenvectors. Prove that i i = 1.
b. Prove that (A) kAk, conclude that (A) = kAk.
c. Prove that is an eigenvalue for A if and only if k is an eigenvalue
for Ak .
d. Prove that kAk k = kAkk .

8. Prove that given a normed linear space, d(u, v) = ku vk defines a

metric.

2.3 Large Matrix Techniques

In this section we consider techniques for solving the linear system Ax = b

when A is smply too large to complete a Gauss-Jordan elimination. Stated
otherwise, we expect that A is nonsingular but it is too large to compute
A1 . This does not necessarily mean that the matrix is too large to process.
Recall that Gauss-Jordan elimination accumulates round off error at the
lower right. Hence, the larger the matrix, the less reliable the process. In
turn, the unreliability is reflected in the considtion number. See Exercies 5
and 6 of Section 2.2.
We begin with a paradigm due to Richardson that best summarizes
large matrix processing. It is called residual correction method.
Suppose that B is our guess for the inverse. We set x1 = Bb, e1 =
x1 x and r1 = b Ax1 , the first approximate solution, first error and
first residual. Continuing, we have the (k + 1)st solution, xk+1 = xk + Brk ,
error, ek+1 = xk+1 x and residual, rk+1 = b Axk+1 . If we expand the
(k + 1)st error we have

ek+1 = xk+1 x = xk + Brk x =

xk + B(b Axk ) x = ek + Bb BAxk =

(I BA)ek = (I BA)k e1 .

The idea behind the Richardson procedure is now apparent. If B is a good

estimate for A1 then we expect I BA to be small, in the sense that its
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 33

Linear Systems and Optimization 33

norm is less than one. In this case, we can prove that xk x. Indeed, it
suffices to prove that ek 0.
kek k = k(I BA)k1 e1 k kI BAkk1 ke1 k 0,
using Exercise 1 of the prior section. However, we can do better.

Theorem 2.3.1. Consider the linear system Ax = b and suppose that B

is an approximate inverse for A such that (I BA) 1, then there an
integer j such that the subsequence xnk+1 x for any k > j. Furthermore,
any two convergent subsequences have the same limit.

Proof. By Theorem 2.2.5, limk k(I BA)k k1/k < 1. Therefore, there is
an integer j such that k(I BA)k ]k < 1 for every k j. Indeed, the k th
root of a positive number is less than 1 only if the number is less than 1.
Now if k > j, then kenk+1 k = k(I BA)nk kke1 k k(I BA)k kn ke1 k 0
as n . The final assertion holds as any two convergent subsequences
have a common convergent subequsence.

As consequence of Theorem 2.3.1, the spectral radius of I BA is

critical. At the end of this section we introduce a technique to estimate the
spectral radius of a matrix.
We look at two specific techniques. The first is creditied to Jacobi.
Continuing with the notation of the theorem, suppose that the diagonal
entries i,i of A are not zero. Then set B so that Bi,j = 0, i 6= j and
Bi,j = 1/i,i . Most often the Jacobi method is used when A is diagonally
dominant, but there is no necessity for that.
As an example, consider the following. Suppose A is a 50 50 matrix.
This is so small that there is no difficulty computing the actual solution via
Gauss-Jordan elimination, and then the actual error for the Jacobi process.
For instance, set A to be tridiagonal with entries i,i = 0.5 on the
diagonal, i,i1 = i,i+1 = 0.25 on the super diagonal and sub diagonal and
zeros elsewhere. Next, take b to have ith entry equal to 1/i. The spectral
radius if I BA is approximately 0.998 < 1. We exepct the Jacobi method
to converge slowly. Indeed, this is the case. Starting with x0 = (1, 1, ..., 1)
we have relative normed error of 1.101438 after 50 iterations, 0.41144 after
500 iterations and 8 105 at 5,000 iterations.
Another form of the Richardson paradym is the Gauss-Seidel method.
In general, it is much faster than the Jacobi. In this case we consider L to
be the lower triangular part of A with the diagonal included. It is easy to
invert L. Indeed, if the diagonal entries of A are nonzero, then a sequence of
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 34

34 Elements of Numerical Analysis with Mathematica

type 3 elementary operations will yeild a row equivalent diagonal matrix.

We follow these with type 2 operation to get the identiy matrix. Now
executing the same opeartions on the identity matrix produces the inverse
of L1 = B. It is important to have a procedure for L1 that does not use
M athematica procedures.
Repeating the prior example for this case, the spectral radius of I BA
is 0.996. Nearly the same as for the Jacobi method. However, this time the
relative normed error is 0.6438 for 50 iterations, 0.1102 for 500 and 4109
for 5,000. These results are orders of magnitude better. It is not surprising
that the Gauss-Seidel method is preferred.
We need to be careful when implementing these techniques. If we could
execute ordinary matrix products, then we would be able to execute Gauss-
Jordan elimination and arrive at the solution of the linear system in the
usual manner. Given we cannot compute BA using oridnary procedures
we must take advantage of the probable sparse nature of the matrices.
For isntance, B is a diangonal matrix in one case and lower triangualr in
the other. In the first case BA = C where each Ci,j = Bi,i Ai,j and in
Pi
the Gauss-Seidel case, Ci,j = k=1 Bi,k Ak,j . In each case it is better to
write the code directly, rather than depend on a general purpose matrix
multiplication. Continuing, take advatnage of the zero entries of (I BA)
by computing (I BA)k as (I BA)(I BA)k1 .
Of course these itnerative processes require a stop criterion. There are
two standard procedures. The first, stops the iteration when the relative
normed residual, kb Axn k/kbk, is smaller than some predermined thresh-
old. The second, uses the relative normed difference between two successive
solution estimates, kxn xn+1 k/kxn k.
There is a simple procedure to estimate the spectral radius of a real ma-
trix. It is called the power method.. The method is remarkably simple. We
begin with an initial estimated eigenvector x0 . Then xn+1 = Axn /kAxn k.
As with most anything else the method is not universally applicable. With
the following theorem we state applicability criteria for the power iteration
method. We omit the proof aAs the theorem requires the Jordan canonical
form. We note that the proof is similar to the one for the Gelfand result of
the prior section.

Theorem 2.3.2. Let A be an n n matrix with spectral radius (A) as-

sociated to eigenvalue and eigenvector x . Suppose that is not the
eigenvalue for two independent eigenvectors. If x0 is a vector with nonzero
x component, then the sequence {xn : xn = xn1 /kxn1 k} converges to
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 35

Linear Systems and Optimization 35

x and {Axn } converges to x .

In the exercises, we develop a proof for the case of a symmetric matrix

A.
We end this section with a brief discussion of Krylov subspaces. The
idea behind Krylov subpaces is to generalize the idea of a iterative matirx
multiplicaiton process. This turns out to be useful as there are very effi-
cient linear solve procedures that are best developed with Krylov subspaces.
Conjugate gradient is one example. The basic idea is that the sequence of
approximate solutions {xj } is a subset of a finite dimensional vector space.
We suppose that kI BAk < 1. We claim the if xk = xk1 + BAxk1 is
a linear combination of x0 , ..., xk1 then the solution to the linear system is
also a linear combination of these vectors. Toward this end, we set W to the
linear span of x0 , ..., xk1 and prove that W is invariant under BA. Indeed,
since each xi = xi1 +BAxi1 , then BAxi lies in W for i = 1, ..., k2. Since
xk lies in W , then BAxk1 = xk xk1 lies in W . Hence, BA(W ) W .
Next, we look at xk+1 = xk + BAxk . Since W is BA invariant, then xk+1
is also in W . Now, it is immediate that {xj } W . Since W is finite
dimensional, then it is closed and the solution to the linear system also lies
in W .
Suppose k = dimW , then the solution to the linear system is a combi-
nation of k known vectors. Hence, what was an n n problem is now a
k k problem. Depending on the setting, this may be significant. In the
literature, W is called the k th Krylov subspace. We look at a particular
case in Exercise 8.
However, this is not the end of the story. Since we are computing
elements of a squence that coverges to the solution, then kxn xn+1 k 0.
Hence, it may well be that xn and xn+1 are independent but due to round
off error, they appear dependent. We get around the problem by replacing
each xn with a vector orthogonal to all xm , m < n. In the literature, this
is done with a clever modification of the Gram-Schmidt process. Given the
orthonormal frams for W , we have the basis needed to solve the new linear
system.

Exercises:
1. Consider the example with the tridiagonal coefficient matrix from
the text. Execute the Jacobi method using the stop threshold 105 .

2. Repeat Exercise 1 using the Gauss-Seidel method and 100 100 ma-
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 36

36 Elements of Numerical Analysis with Mathematica

trices. How does the size change affect the relative normed error at 50, 500
and 5000 iterations.

3. Repeat Exercise 2 using an exit threshold value of 105 . Compare

relative normed residual exit test to the relative step difference exit test.
Which one exits sooner?

4. Execute Gauss-Seidel for several symmetric matrices of various sizes.

Use Random in Mathematica to generate the matrices. Determine the de-
termine the approximate time and iteration counts.

5. Let L be a lower triangular matrix with inverse B. Write an expres-

sion for each Bi,j in terms of the entries of L. Be certain that you use a
mininal number of operations.

6. Use the power iteration method to estimate the spectral radius for
(I BA) for the two examples (Jacobi and Gauss-Seidel) develped in the
text. How many iterations are requried to complete the estimate given
above?.

7. The proof of Theorem 3.2.3 in the symmetirc case. Given a symmetric

A, there is a basis of eigenvector {v1 , ..., vm }. Let i denote the eigenvalue
for xi . In addition, we may suppose the |1 | is the spectral radius of A.
a. Prove that
Ak x 0
= xk+1 . (2.3.1)
kAk x0 k
P
b. Write x0 = 1 x1 + i=2 i xi .
c. Prove that
X k
i
Ak xk = k1 1 x1 +


k
i xi

i=2 1
d. Prove that
X k
i
1 x1 + i xi 1 x1
i=2
k1
as k
e. Prove that xk+1 x1 . (Hint: for any real, a/|a| = 1.)

8. Consider the Gauss-Seidel example developed in the section. Deter-

mine the dimenstion of the Krylov subspace.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 37

Linear Systems and Optimization 37

2.4 Functions of Several Variables: Finding Roots and Ex-

trema

The techniques we develop in this section are also referred to as Newtons

method since they use derivatives and a single initial estimate to establish
an iterative process to search for a root. In general terms, this is identical
to Newtons method introduced in Section 1.3.
As these procedures apply to differentiable functions f : Rn Rm , they
apply to linear systems which are not square or to square linear systems
whose coefficient matrix is singular. More generally, setting g = f.f , alter-
natively g = f 2 if m = 1, then the roots of f are minima of g. Hence, we
need only consider the problem of finding extrema in order to find roots.
The techniques developed in this section are applicable to optimal con-
trol theory and sensitivity analysis. Sensitivity analysis is of particular
interest. Here you define a function f which measures an outcome from
given independent (input) variables. However, the parameters necessary
to express f may not be known with certainty. For instance a formula in
finance may depend on the price volitility (the variance of a random vari-
able). But it is often the case that the variance, 2 , is not known exactly.
Sensitivity analysis attempts to determine how the outcome will vary with
changes in the estimate for 2 . In effect, the analyst is seeking to estimate
f / 2 .
In another direction, these minimization techniques apply to web search
technology. In that case, each person browsiing the web has a penalty func-
tion. This finction is determined by his/her prior tendencies. The browser
returns the list of web pages that minimizes the penalty fiunction. Of
course, this max/min problem has perhaps a hundred thousand veariables.
Do note that we have written this section for functions defined on R2 .
It is easy to extend these techniques to Rn .
We begin by looking at an example. Consider f (x, y) = x2 +y 2 mapping
R to R. (See Figure 2.4.1). The graph of f is a subset of R3 , and the single
2

minimum of f is at (0, 0). Suppose we start the search for a minimum at

(1, 2). If we think back to the method developed in Section 1.3, we want
a line tangent to the surface passing through (1, 2, 5) where 5 = f (1, 2).
Then we want to determine where interesects the xy-plane. That point
will be our next approximate minimum. If we repeat the process, we expect
that we will get better and better approximate roots.
The first problem is that we know the tangent plane at (1, 2, 5). It has
equation z 5 = f (1, 2).(x 1, y 2). But we do not know which line
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 38

38 Elements of Numerical Analysis with Mathematica

15

10
2
5

-2

0
-2

Fig. 2.1 Figure 2.4.1 f (x, y) = x2 + y 2

on the plane to use for . There are two standard procedures to determine
the direction vector . We develop one now and the second at the end of
the section. Recall a fact from multivariate calculus, the gradient points
in the direction of maximal descent. Hence, it seems reasonable to select
that direction for the line . In this case the technque is often called the
method of maximal descent.
We know that the gradient f = f /x(x0 , y0 ) is a vector in the xy-
plane, the domain of f , that determines the direction of maximal change
for f . So, it is reasonable to set = f /x(x0 , y0 ) and = f /y(x0 , y0 )
and consider the line (1, 2) + t(, ) = (1 + t, 2 + t) in the xy-plane. Next,
we define a function h : R R, h(t) = f (1 + t, 2 + t). We can now solve
for a max/min of h. This is a one variable calculus problem. Finding a
minimun for h should yield a value for f less than 5, the value at (1, 2).
Indeed, f (1, 2) = (2, 4), h(t) = (1 + 2t)2 + (2 + 4t)2 = 5 + 20t + 20t2 . The
derivative of h is 20 + 40t. It has extremum at t = 0.5. Now h(0.5) =
f (0, 0) = 0 < 5. Indeed, we recognize the origin as the minumum of f . And
we have arrived in one step.
We now state the general process for functions of several variables. Sup-
pose we seek a minimum of f mapping Rn to R.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 39

Linear Systems and Optimization 39

(1) Compute the gradient of f at (x0 , y0 ), f (x0 , y0 ) = (, ).

(2) Set h(t) = f (x0 + t, y0 + t).
(3) Solve the single variable calculus problem for h to yield t0 ..
(4) Set (x1 , y1 ) = (x0 + t0 , y0 + t0 ).
(5) If f (x0 , y0 ) < f (x1 , y1 ) , then exit (the process has failed).
(6) If the iteration count exceeds the maximum, exit (the process has
failed).
(7) If |f (x1 , y1 ) f (x0 , y0 )| is sufficiently small, exit (possible success).
(8) Go back to Step 1 using (x1 , y1 ) as the seed.

It is intereseting to note that maximal descent is sufficient for the mini-

mization problems that occur in big data and machine learning applications.
To introduce the second technique, we look at another example. Sup-
pose f (x, y) = cos2 (x)ey + 1. The minima for the function occur at the
odd multiples of x = /2. (See Figure 2.4.2). If we start the search at
(0, 1), then t0 = 35.7979, (x0 , y0 ) = (0, 96.2817) and we are way out
on the negative y-axis. Even though the value of f is nearly zero (about
1.4 1047 , no further processing will take us any closer to an actual min-
imum. Hence, maximal descent has failed for this case.

400
6

200

4
0

-5

2
0

5
0

Fig. 2.2 Figure 2.4.2 f (x, y) = cos2 (x)ey

There are many alternate choices for the direction vector for . One
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 40

40 Elements of Numerical Analysis with Mathematica

choice is similar to the secant method. In this case we begin with the
Taylor expansion for f .
1
f (x + s) = f (x) + f (x).s + sT H(x)s + R2 (2.4.1)
2
where sT denotes the transpose of s, H is the Hessian of f and R2 is the
the remainder term. Recall that the Hessian is the matrix whose entries are
2 f /xi xj . Because of the use of the Hessian, this technique is referred
to as the Hessian method.
If we suppose that f (s) = f (x + 2), then according to Rolles theorem,
we would expect a local extrema between x and x + s. Hence, = s is the
search direction. If we take the remainder term to be zero and we recast
(2.4.1),
1
H(x)s = f (x). (2.4.2)
2
Therefore, we solve for s. Since (2.4.2) is a linear system with coefficient
matrix H(x), then we can find provided H(x) is nonsingular. Finally,
to describe the Hessian method, we need only replace statement 1 by the
following
(1) Compute (, ) as the solution to the linear system 0.5H(x)s =
f (x).
As mentioned at the beginning of the section, if f takes values in Rm
then g = f.f is real valued and the roots of f are now extrema for g. Hence,
we can use the techniques developed here to solve the general problem
f (x) = 0. We present examples in the exercises.

Exercises:
1. Use maximal descent to find a minimmum for f (x, y) = x2 + xy + y 2 .
Use (2, 1) as the search starting point.

2. Use the Hessian method to find a minimmum for f (x, y) = x2 + xy +

2
y . Use (2, 1) as the search starting point.

3. Let f (x, y) = (x + y, x + y). Solve f = 0 using (2, 1) as the initial

estimate. Note that f is a singular linear transformation. When solving
this problem you are solving a linear system with a singular coefficient ma-
trix.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 41

Linear Systems and Optimization 41

4. Consider the linear transformation

4 2 3 5 4x 2y + 3z 5w

x
3 3 3 8 y 3x + 3y + 5z 8w
6 1 4 3 . z = 6x y + 4z + 3w
L(x, y, z, w) =

4 2 3 5 w 4x + 2y 3z + 5w
a. Use LUDecomposition to determine if A is singular or non-singular.
(Do not forget to introduce a decimal point to the data.) How does this
impact the problem of solving an equation of the form L(x, y, z, w) =
(x0 , y0 , z0 , w0 )?
b. Use the maximal descent method to solve L(x, y, z, w) = (1, 1, 1, 1).
Use (5, 5, 5, 5) for the initial estimate.
Use at least 35 iterations.
Use 105 as the tolerance in Step 7.
Make certain to use two if statements, one for Step 5 and one for
Step 7.
c. Redo Part b using (1, 2, 3, 4) as the initial estimate.
d. Why is it possible for the solution to b and c to be different?
e. Prove that if v is the slution to b and v is the solution to c, then v v
solves L(x, y, z, w) = (0, 0, 0, 0). (What is the kernel of a linear transforma-
tion?)

f. Use LinearSolve to get a solution to L(x, y, z, w) = (1, 1, 1, 1). Is

this solution trusted? Why? What was the condition number from Part a?
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 42
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 43

Chapter 3

Interpolating and Fitting

Introduction

We introduce the following terminology. Suppose we are given a set of n

points P1 , ..., Pn in the plane, R2 , we may want to find a curve (function)
which passes through the points (interpolating) or a curve which passes
near to the points (fitting). If we want the curve to pass through the
points, then we may have to accept anomalies on the curve. If we are
willing to accept a curve that only passes near the points, then we may
place stronger restrictions on the curve. In this chapter we see how this
give and take materializes.
Of the several techniques there is no best of all, no method that gives
best results under all circumstances. The spline, with applications in com-
puter graphics, visualization, robotics and statistics, is perhaps the most
widely used. The spline curve is twice continuously differentiable, depends
only on point data and faithfully reflects the tendencies of the input data.
On the other hand, among the techniques we present, splines have the most
complex mathematical foundation. For all of these reasons, we a correct
mathematical development of cubic splines.
In another direction, polynomial interpolation is the oldest of the tech-
niques. It has the most developed theory and is widely used as a tech-
nique for approximating integrals and approximating solutions to differen-
tial equations. For this reason, it is arguably the most important
Least squares fitting in the linear case provides the numerical technique
used for linear regression. Furthermore, least squares fitting often arises
in the literature as a generalization of polynomial interpolation. In this
context, it is a technique for estimating the error for finite element method.
Another technique is Bezier interpolation. This procedure was devel-

43
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 44

44 Elements of Numerical Analysis with Mathematica

oped originally to be used by engineers when resolving artist designs. In

particular, Bezier curves were developed as a tool to help an engineer derive
three dimensional coordinates from a designers concept drawing.
The final technique is Hermite interpolation. In this case we are charged
with finding a polynomial interpolation that both approximates the func-
tion but also its derivative. The Hermite interpolation provides the under-
lying mathematical foundation for Gaussian quadrature.
Before proceeding we mention the theorem of Weierstrass, any contin-
uous function on a closed interval is the uniform limit of a sequence of
polynomial functions. This is a remarkable result. And it is very old as
it was proved 1885. The proof however does not explain what the poly-
nomials are. It was not until 1912 that Bernstein identified a sequence
of polynomials. Even though the Berstein polynomials are determined by
the continuous function, they do not interpolate the target. Further, the
convergence is very slow. A reasonable approximation of a function with
Bernstein polynomials often requires Bernstein polynomials of degree two
or three thousand. This is not a useful alternative to the techniques we are
about to develop.

3.1 Polynomial Interpolation

In this section we introduce the idea of a polynomial interpolation. We

start with an unknown function f with known values at points xi in ths
domain and we contruct an interpolating polynomial p that agrees with f at
these locations. The idea is that if p agrees with f at designated locations
then we can use p in place of f . But if f is not continuous at xi , then the
values of f at the location may not relate to values of f at nearby points.
Therefore, in order to talk about interpolation, we must have a continuous
function. Throughout this section we suppose that f is contuous on its
domain.
We begin by looking at the Taylor expansion of a function. Consider
the function f (x) = xex 1. Plotting this function on the interval [1, 4]:

f[x_] = x*Exp[-x] - 1;
Plot[f[x], {x,1,4}];

shows a decreasing function with an inflection point.

Thinking of this curve as being more or less cubic, we can develop
the cubic Taylor polynomial interpolation for f expanded at the midpoint,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 45

Interpolating and Fitting 45

-0.65 -0.65

-0.70 -0.70

-0.75 -0.75

-0.80 -0.80

-0.85 -0.85

-0.90 -0.90

1.5 2.0 2.5 3.0 3.5 4.0 1.5 2.0 2.5 3.0 3.5 4.0

Fig. 3.1 Figure 3.1.1: f together with Fig. 3.2 Figure 3.1.2: f concave down
the Talor expansion at x = 2.5 near the root

x = 2.5.
df
(2.5)(x 2.5)+
g(x) = f (2.5) +
dx
1 d2 f 1 d3 f
2
(2.5)(x 2.5)2 + (2.5)(x 2.5)3 .
2! dx 3! dx3
When developing g you will need to compute the derivatives of
f . Recall that the derivatives of f are computed in Mathematica via
D[f [x], x], D[f [x], x, x] and so forth. If you plot f and g on the same axis
you will see that the cubic Taylor polynomial provides a remarkably good
approximation of this function. Figure 3.1.2 shows the graph of g together
with the graph of f . Notice that the graph of g is above f on the left and
below on the right.
A numerical measurement of the goodness of fit is given by the L2 norm
of f g,
Z 4

1/2
kf gk2 = (f g)2 dx .
1
This is called the norm interpolation error . In turn, the mean norm
interpolation error is
Z 4
1
1/2
(f g)2 dx .
41 1
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 46

46 Elements of Numerical Analysis with Mathematica

The finite Taylor expansion produces a high quality one point interpo-
lation provided we know the original function. However, suppose we have
points and no function, then we will need a different approach.

Definition 3.1.1. Consider points P0 , ..., Pn in R2 , Pi = (xi , yi ). The

polynomial interpolation is a polynomial p of degree n that interpolates the
n + 1 points in the sense that p(xi ) = yi .
Pn
Now, if p(x) = i=0 i xi , then to determine p we must find the coeffi-
cients 0 , ..., n . Notice that we can write the polynomial as a row vector
times a column vector,

0
n
1 X
(x0 , x1 , ..., xn ) i xi .

=
...
i=0
n
Our requirement for p is that it interpolate the n + 1 points. Hence, we
have for each i,

0
1
p(xi ) = (x0i , x1i , ..., xn )

... = yi

n
Collecting these equations we then get the following matrix equation

1 n
0 y0
1 x1 ... x1 1 y1
. . ... . =
... ...
1 x1n+1 ... xnn+1
n yn
where x0i = 1. This is a linear system of equations where the xi and yi
are known while the i are unknown. Hence, we can use the LinearSolve
function in Mathematica to find the coefficients of p provided the coefficient
matrix is non-singular. The matrix is called a Vandermonde matrix. It is
always nonsingular provided the xi are distinct.

Theorem 3.1.1. The Vandermonde matrix

1 x11 ... xn1

. . ... .
1 x1n+1 ... xnn+1
is nonsingular, provided the scalars xi , i = 0, 1, ..., n are distinct.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 47

Interpolating and Fitting 47

Proof. The Vandermonde matrix is singular only if the columns are de-
pendent. In particular, only if there are scalars 0 , ..., n not all zero with
1 n
1 x1 x1 0
0 ... + 1
... + ... + n ... = ...

1 n
1 xn+1 xn+1 0
Hence, for each i = 1, 2, ..., n + 1 we have
0 + 1 x1i + ... + n xni .
We have demonstrated a polynomial 0 + 1 x1 + ... + n xn that is not zero,
has degree n and therefore has at most n distinct roots. However, we just
showed that it has n+1 distinct roots, x1 , ..., xn+1 . As this is impossible, we
are led to the conclusion that the Vandermonde matrix is nonsingular.

There is another way to do polynomial interpolation. The outcome is

the same, but nevertheless, the approach does provide insight. As in the
previous case we begin with n + 1 points in R2 , denoted P1 , ..., Pn+1 with
Pi = (xi , yi ). For each i, we set
(x x1 )(x x2 )...(x xi1 )(x xi+1 )...(x xn+1 )
li (x) = =
(xi x1 )(xi x2 )...(xi xi1 )(xi xi+1 )...(xi xn+1 )
Q
(x xj )
Q j6=i .
j6=i i xj )
(x
It is not difficult to see that the polynomials li (x) have degree n, satisfy
Pn1
li (xi ) = 1 and li (xj ) = 0 whenever j 6= i. Moreover, q(x) = i=1 yi li (x)
interpolates the given points. (See Exercise 5.) The polynomials li (x)
are called Lagrange polynomials. We now see that the two polynomial
interpolations, p derived from the Vandermonde matrix and q derived from
the Lagrange polynomials are in fact the same.

Theorem 3.1.2. Given a continuous function f and interpolation points

P1 , ..., Pn1 , suppose that the interoplation derived from the Vandermonde
matrix is given by p and the interpolation derived from the Lagrange poly-
nomials by q, then p(x) = q(x).

Proof. We begin by setting r = p q. Hence, r is a degree n polynomial.

Since p(xi ) = yi = q(xi ) for each i = 1, 2, ..., n + 1, then r has n + 1 roots,
x1 , ..., xn+1 . But if r is not identically zero, then it can have at most n
roots. Therefore, r = 0 and p = q.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 48

48 Elements of Numerical Analysis with Mathematica

It is possible that you must use Lagrange polynomials to compute the

polynomial interpoladtion of a function. In particular, there are cases where
the Vandermonde matrix procedure does not work. Suppose that two of
the x-axis locations xi and xj are very close together. Then it would ap-
pear to Mathematica that two of the rows of the Vandermonde matrix are
equal or nearly equal. In this case the condition number will be large and
LinearSolve will not return reliable results. Nevertheless, it is still possible
to get the interpolation via Lagrange polynomials.
If the points Pi lie on the graph of a function f , then it is natural to
ask how well does p approximate f . If f has at least n + 1 continuous
derivatives then we can estimate the error, e(x) = f (x) p(x). Recall that
with this hypothesis, then the error for the Taylor interpolation will have
a known bound.

Theorem 3.1.3. Suppose that f is a real valued function defined on an

interval [a, b] and suppose that f has at least n + 1 continuous derivatives.
Further, take a x1 < ... < xn+1 b, with f (xi ) = yi . If p is the
polynomial interpolation of the points (xi , yi ), then the error e(x) = f (x)
p(x) is given by
f (n+1) () Y
e(x) = (x xi ), (3.1.1)
(n + 1)! i
for some = x in (a, b) depending on x. In particular,
M
|e(x)| (b a)n+1 , (3.1.2)
(n + 1)!
where M is the maximal value of f (n+1) on the interval.
Q
Proof. We define g(x) = e(x)/ i (x xi ), so that e(x) = f (x) p(x) =
Q
i (x xi )g(x). Next, take in [a, b] distinct from the xi and set
Y
h(x) = f (x) p(x) (x xi )g().
i
Note that we cannot be certain that g is defined at the xi , however our
choice of assures us that h is defined on [a, b] with n + 1 continuous
derivatives.
Q
Now, each xi is a root of h and in addition h() = e() i ( xi )g() =
0. Hence, h has n+2 roots in the interval [a, b]. Furthermore, h is continuous
on the closed interval and differentiable on the open interval (a, b). Hence,
we may apply Rolles theorem to the interval between each pair of successive
roots and conclude that between each pair of roots there is a root of the
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 49

Interpolating and Fitting 49

derivative of h. Hence, dh/dx has at least n + 1 roots on the interval (a, b).
Repeating this argument, d2 h/dx2 has at least n roots in (a, b). Continuing,
the k th derivative of h has at least n + 2 k roots. So that the n + 1st
derivative has at least 1 root. We denote this root by = , since
depends on our choice of . Now
dn+1 Y
0 = h(n+1) () = f (n+1) () p(n+1) () g() n+1 (x xi )|x= .
dx i

But p is degree n, so p(n+1) = 0. Also dn+1 /dxn+1 i (x xi ) = (n + 1)!,

Q

no matter what is. Therefore,

f (n+1) () Y
e() = ( xi ).
(n + 1)! i
Finally, since h is defined for any x in the interval, then this last expression
for the error is satisfied for all x.
For the final statement on the bound for the error magnitude we note
that since f is n + 1 times continuously differentiable, then f (n+1) is con-
tinuous and hence has maximum value on the interval.

Numerical integration is based on polynomial interpolation. Hence, the

interpolation error is also the numerical integration error. In turn, polyno-
mial interpolation is also an important feature in approximating the solu-
tion of a partial differential equation. Hence, interpolation error arrises in
that context. On the other hand, the estimate for the error magnitude is of
little use if we do not have information about f . Indeed, it is not difficult
to find functions where M is very large. Nor is it difficult to find functions
where the error is large. The following example is a case in point.
Returning to the function f (x) = xex 1 and the four points Pi =
(xi , yi ), x = 1, 2, 3 and 4. The polynomial interpolation, p(x), of the points
will again provide an approximation of f by a cubic polynomial. As in the
case of the Taylor interpolation, it is remarkably close to f . On the other
hand consider the function f (x) = 1/(1 + x2 ). In this case, pick a finite
sequence of points along the graph of f , which are symmetric about the y-
axis. Use these points to produce a polynomial interpolation of the f . (See
Exercise 3 below.) The problem is that the polynomial looks nothing like
the function. Further, the more points you choose the less the polynomial
resembles f . Looking at the graph if f we see that the function seems not to
be a polynomial function. (Note the asymptotic behavior. It is not easy to
find a polynomial can reproduce this type of behavior.) Hence, we should
not expect that there is a polynomial function that interpolates it well.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 50

50 Elements of Numerical Analysis with Mathematica

There is another problem with polynomial interpolation. Consider again

the function f (x) = 1/(1 + x2 ) and select four points P1 = (4, 1/17), P2 =
(2, 1/5), P3 = (2, 1/5), P4 = (4, 1/17) from the graph of f . Next select
P = (0, y) where y [0.2, 0.3]. Figure 3.3 shows the resulting polynomials
for three values of y. Suppose that the location of the points came from
some measuring or sampling process, then small errors (as in this case) may
yield significantly different results. Looking at the resulting curves we see
that shape of the curves is different. Further the change in y is magnified 20
times at p(5). This is an inherent problem with polynomial interpolation.
The technical term for the problem is that polynomial interpolation lacks
local control . In a subsequent section we develop spline curves. These
curves were developed precisely to resolve the local control problem.

0.30 0.25

0.25 0.20

0.20 0.15

0.10
0.15
0.05
0.10

-4 -2 2 4
-4 -2 2 4
0.2

0.1

-4 -2 2 4

-0.1

Fig. 3.3 Figure 3.1.3 Three alternate images y = 0.3, 0.25, 0.2; p(5) = 0.11, 0.04, 1.9

In spite of the problem we just noted, polynomial interpolation is an

important and productive tool for numerically solving differential equations.
When this technique is used special care is taken to ameliorate the problem
we see in Figure 3.1.3.
Because the Taylor expansion requires more information than is usually
available, it is often ignored as an interpolation technique. However, there
is an important application, which should not be ignored. In the next
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 51

Interpolating and Fitting 51

section we will develop a class of parametric cubic interpolations. Consider

the setting where (t) = (1 (t), 2 (t)) in R2 and each i is an ordinary
cubic polynomial. When represents a function, then it possible to solve
x = 1 (t) for t and then substitute this in 2 to yield = (x, f (x)).
However, the resulting function is rarely integrable. On the other hand,
you can get values for f and its derivatives. Hence, you can write the cubic
Taylor expansion for f and this is easily integrated.
Finally, in Exercise 7 we introduce the idea of piecewise polynomial in-
terpolation. The basic idea of polynomial interpolation is that the more
points that we interpolate, then the better the polynomial will approximate
the original function. However, as we add more and more points then the
degree of the polynomial increases. In piecewise polynomial interpolation,
we subdivide the interval into smaller and smaller subintervals while inter-
polating the function by polynomials of fixed degree on each subinterval.

Exercises:
1. Compute the norm error and the mean norm error for the function
f (x) = xex 1 and its cubic Taylor expansion about x = 2.5. Use the
interval [1, 4].

2. For f (x) = xex 1,

a. Compute the polynomial interpolation p for the Pi v = (xi , yi ), x =
1, 2, 3 and 4.
b. Plot the graph of f and p on the same axes for the interval [1, 4].
c. Compute the norm error and mean norm error.
d. Is p better or worse than the cubic Taylor interplant.
e. Use Theorem 3.1.3 to estimate the maximal absolute error for the
interpolation p.

3. Compute the polynomial interpolation of the points (xi , 1/(1 + x2i )

for xi = 2, 1, 0, 1, 2. Plot the polynomial against the graph of f (x) =
1/(1 + x2 ). Compute the norm error and mean norm error.

4. Repeat Exercise 3 with additional points on the x-axis and 3 and -3,
4 and -4. Does this produce a better approximation of the function f ?
5. Consider the points Pi = (xi , yi ), i = 1, 2, ..., n + 1, in the real plane
and the corresponding Lagrange polynomials li .
a. Prove that for each i, li is a degree n polynomial with li (xj ) = 0 if
i 6= j, and li (xi ) = 1.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 52

52 Elements of Numerical Analysis with Mathematica

Pn+1
b. Prove that q(x) = i=1 y1 Li (x) interpolates the given points.

6. Use Theorem 3.1.3 to estimate the maximal absolute error in 3 and

in 4. Does adding the additional points increase or decrease the error esti-
mate?

7. Consider the function f (x) = 1/(1 + x2 ) on the interval [4, 4].

a. Determine the maximal value for = f (3) (x) on the interval.
b. Divide the interval into 40 subintervals of length 0.2 = (4 - ( -4))/40.
In particular determine 4 = a0 < a1 < ... < a40 with each ak+1 ak = 0.2.
c. Compute the second degree polynomial interpolation of f on the
subinterval [ak , ak+1 ] using the three values ak , (ak + ak+1 )/2 and ak+1 .
d. Plot the result of Part c and overlay the plot of f .
e. Use Theorem 3.1.3 to prove that the absolute error |e(x)| is bounded
by /6(0.2)4 0, where is the value computed in Part a.
f. Prove that as the number of subintervals goes to , then error con-
verges to zero.

8. Researchers reporting in a chemical engineering journal reported

the following data on tungsten production as a function of temperature
measured in degrees Kelvin.

t 700 800 900 1000

f (t) 0.071 0.084 0.097 0.111

They determined that the data fit to the following function (to 3 decimal
places accuracy),

f (t) = 0.02424(t/303.16)1.27591 .

a. Use a cubic polynomial to interpolate the given data. Use this

polynomial to estimate the values at t = 750, 850 and 950. (Because the
values of t are large, the Vandermonde matrix will appear to Mathematica
as ill conditioned. You resolve this by using Lagrange polynomials.)
b. Assuming that f is the correct predictor for tungsten, determine the
mean absolute error for the three estimates in Part a.
c. Again assuming that f is correct, use Theorem 3.1.3 to calculate the
estimated error for the cubic polynomial interpolation as a function of t.
Then determine the estimated mean absolute error for the three values of
t.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 53

Interpolating and Fitting 53

d. Is the actual mean absolute error smaller than the estimated mean
absolute error?

3.2 Bezier Interpolation

Bezier interpolation arose around 1960 to solve a problem in the manufac-

turing industry. When a new product is begun, a designer will produce a
rendering. Engineers will then produce specifications from the designers
drawing. In the automobile or aircraft industries, the engineers task was
indeed difficult. The designer would produce a concept drawing of the car
or aircraft. From this drawing, the engineers would have to specify the
requirements for the sheets of metal for the exterior, the necessary frame
and then they could infer the shapes and sizes of the spaces for the pas-
senger compartments, the engine compartment, etc. The task was nearly
impossible causing cost overruns and time delays.
The tools at their disposal were primitive. Often they would produce
a wooden model of the object, then slice the wooden object with a saw to
create a sequence of cross sections. Next the cross sections where projected
on a large screen at actual size, then these images were traced and measured
to yield the data necessary for construction. This was the context when
Bezier interpolation was introduced.
The Bezier curve is a parametric cubic curve based on 4 guide points
B1 , B2 , B3 and B4 . The curve has end points at B1 and B4 , and tangent
lines at these end points passing through B2 and B3 . If we designate the
parametric curve in R2 as (t) = (1 (t), 2 (t)) with t in [0, 1], then these
requirements may be stated as
d d
(0) = B1 , (1) = B4 ; (0) = 3(B2 B1 ), (1) = 3(B4 B3 ).
dt dt
Bezier, an engineer in the French automobile industry came up with
following procedure. It is based a geometric construction. First, fix four
points, B1 , B2 , B3 and B4 , called guide points. Then connect the four guide
points with line segments. (See Figure 3.2.1a.) Next fix a real t in the
interval [0, 1]. Using t, locate the point B1,1 = B1 + t(B2 B1 ) on the
line segment connecting B1 to B2 . Similarly, locate B1,2 and B1,3 between
B2 , B3 and B3 and B4 , respectively, and connect these points with line
segments. (See Figure 3.2.1b) Repeat the process with the three points
B1,1 , B1,2 and B1,3 to derive two additional points B2,1 and B2,2 on the
segments connecting B1,1 to B1,2 and B1,2 to B1,3 . Finally, we set (t) =
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 54

54 Elements of Numerical Analysis with Mathematica

B2,1 + t(B2.2 B2,1 ). (See Figure 3.2.1c) If we write (t) in terms of the
original four points we have the usual representation for the Bezier curve.
(t) = (1 t)3 B1 + 3t(1 t)2 B2 + 3t2 (1 t)B3 + (t3 )B4 . (3.2.1)

B3
1.4 1.4

1.2 1.2 B12

1.0
1.0 B2
B13
0.8
0.8
0.6
B11
0.6
0.4
0.4 B4
0.2
0.2
B1 1 2 3 4 5 6

1 2 3 4 5 6

Fig. 3.4 Figure 3.2.1a: Four guide Fig. 3.5 Figure 3.2.1b: 2nd -level
points with segments points with line segments, t = 0.6.

Letting t vary in the unit interval, (3.2.1) describes a parametric cubic.

Figure 3.2.1d shows the plot of (t). The associated coordinate functions
are given by
1 (t) = (1 t)3 x1 + 3t2 (1 t)x2 + 3t(1 t)2 x3 + t3 x4 ,

2 (t) = (1 t)3 y1 + 3t2 (1 t)y2 + 3t(1 t)2 y3 + t3 y4 ,

where Bi = (xi , yi ).
Just as in the case of the polynomial interpolation, we require four points
to do a cubic interpolation. However, in this case the necessary information
includes only two points on the curve (the starting point and ending point)
and the slope of the curve at these two points. More complicated curves
can be constructed by piecing successive Bezier curves together.
Return to the function f (x) = xex 1. We can derive the Bezier
interpolation of f by setting B1 = (1, f (1)) and B4 = (4, f (4)). We use the
derivative of f at 1 and 4 to determine the other two guide points. Since the
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 55

Interpolating and Fitting 55

1.4 1.4

1.2
1.2
1.0 B23
B21 B22
1.0
0.8

0.6 0.8

0.4 0.6

0.2
0.4

1 2 3 4 5 6
0.2

1 2 3 4 5 6

Fig. 3.6 Figure 3.2.1c: 3r d-level points Fig. 3.7 Figure 3.2.1d: The Bezier
with line segments curve (t)

points on the function graph are given by (x, f (x)), then the tangent vectors
to the graph are d/dx(x, f (x)) = (1, f 0 (x)). Hence, the tangent vector at
B1 is (1, f 0 (1)). Since this vector must also satisfy (1, f 0 (1)) = 3(B2 B2 ),
we have B2 = B1 + 1/3(1, f 0 (1)). Similarly, B3 = B4 1/3(1, f 0 (4)). As in
the previous cases, the Bezier interpolation of f is a good approximation
of the original curve.
In Section 1.3 we showed a curve for which Newtons method failed
because the process cycled, the third estimated root was equal to the first,
the fourth equal to second and so forth. We created this curve using a Bezier
curve. We started with B1 = (1, 1) and B4 = (1, 1). Next, we wanted the
slope at B1 to be 1/2 and equal to 1/2 at B4 , so that Newtons method
would return points (1, 0) and (1, 0). Using the technique described above,
we have
(1, 1/2) = 3(B2 B1 ); (1, 1/2) = 3(B4 B3 ).
One purpose of interpolating points is to use the interpolating function
to compute the integral of a unknown function inferred from the points. We
will see later, that the parametric form of the Bezier curve is significantly
more difficult to deal with than the polynomial or Taylor interpolation.

Exercises:
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 56

56 Elements of Numerical Analysis with Mathematica

1. Use the Bezier technique to interpolate f (x) = xe x1 for x [1, 4].

Plot f and on the same axes.

2. Interpolate f (x) = 1/(x2 + 1) between 2 and 2. Use one Bezier

curve between 2 and 0 and another between 0 and 2.

3. Do the illustration at the end of Section 1.3. If the curve is given by

(t) = (1 (t), 2 (t)), then use the Mathematica statement, Solve[1 (t) ==
x, t], to solve for x in terms of t. Insert the result into 2 (t). The result
will give you an expression for the curve in the form f (x) = y. Use Expand
to fully resolve f . Plot f .

4. In the chapter introduction, we mentioned the Bernstein Polynomials.

Given a set of points P0 , P1 , ..., Pn with Pi = (xi , yi ) in the plane, then the
nth Berstein polynomial is given my
n
X n!
pn (t) = (1 t)i tni Pi .
i=0
i!(n i)!

a. Prove that the cubic Bezier curve (t) defined on four points is
identical with the third Berstein polynomial.
b. Prove that for any n, pn (0) = P0 and pn (1) = Pn .
c. Use parts a, b to define a generalization of cubic Bezier curves.

3.3 Least Squares Fitting

We begin with a word of caution. Least square fitting in the linear case
arises also in the context of linear regression. This is more of a coincidence
than anything else. It is true that in both cases a line is fit to a finite set
of points. In addition, the line arises from the same minimization process.
Beyond that the processes are different and distinct. Least squares fitting
in the numerical methods context is a procedure that begins with a set
of points, and then guides the researcher to a polynomial, which seems
to fit well to the points. On the other hand linear regression begins with
a set of points sampled from a distribution and includes assumptions on
the distribution and the sample. Then a line is inferred. In particular,
the line is derived by minimizing the variance of a related distribution.
Furthermore, statistics are returned indicating confidence intervals for the
slope and y-intercept of the line. In addition, a general statistic is returned,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 57

Interpolating and Fitting 57

which indicates whether linear regression was a reasonable approach to the

data.
In short, least squares fitting is a process that begins with a set of points
and returns a best fitting polynomial. Regression is a statistical process
that applies to a sample from a distribution, fits a line to the sample and
returns statistical information about the reliability of the process. In this
course we are concerned only with the former.
We begin with points P1 , ..., Pn ; Pi = (xi , yi ). We are expecting to find
a line, y = mx + b, which best fits the point set. In order to proceed we
must define best fits. Indeed, the term least squares refers to the following
definition of best fits. Suppose we were to calculate the vertical distance
from each of the points to the line and then total the squares of all the
distances. We will say that a line best fits the point set, if this number
(sum of squared verticle displacements) is minimal. Notice that we have
described a calculus max/min problem.
With this description in mind, we write out the term for the total cal-
culated displacement as a function of the slope and y-intercept of the line.
Now, the vertical distance from Pi = (xi , yi ) to the line y = mx + b, is
|yi (mxi + b)|. Since we are heading toward a calculus style max/min
process, the absolute value is inconvenient. Hence, square each of these
terms and then add to get a function with independent variables m and
b.
X
(m, b) = (yi mxi + b)2 .
i
The next step is to apply standard max/min techniques to . To begin,
we differentiate the dependent variable with respect to the independent
variable.

X X
= 2(yi (mxi + b))(xi ) = 2( (mx2i + (b yi )xi ),
m i i

X X
= 2(yi (mxi + b))(xi ) = 2 (yi (mxi + b)).
b i i
Setting these two terms to zero and reorganizing them just a little, we get
X X X X
0= (mx2i + (b yi )xi ) = m x2i + b xi xi yi
i i i i
X X X
0= (yi (mxi + b)) = m xi nb + yi
i i i
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 58

58 Elements of Numerical Analysis with Mathematica

Now these two equations can be recast as a 2 by 2 linear system with

unknowns m and b.
X X
X

xi yi = x2i m + xi b
i i i

X X

yi = xi m + nb.
i i

Or in matrix notation,
P 2 P    P 
x i xi m xi yi
Pi i = Pi
i xi n b i yi

Next set
 
x1 ... xn
A= .
1 ... 1
Then it is immediate that

 x 1
x1 ... xn 1
 P 2 P 
x i xi ,
AAT = . . = Pi i
1 ... 1 i xi n
xn 1
and

y1 P 
xy
A . = Pi i i .
i yi
yn
Hence we may rewrite the 2 by 2 system as

  y1
m
AAT = A . .
b
yn
This form of the linear system is most suitable for our calculations. It is
straight forward to prove that the coefficient matrix, A(AT ), is necessarily
non-singular provided that no two xi are equal.
To get a feel for how this looks, consider the following example. Suppose
we have points P1 = (5, 3), P2 = (4, 2), P3 = (2, 7), P4 = (0, 0), P5 =
(1, 5), P6 = (3, 3), P7 = (5, 5). The following figures shows the points and
the resulting least squares line.
If we had asked for a quadratic polynomial that best fit the point set,
then we would be looking for three coefficients, a, b and c. Setting up the
problem as in the linear case to get (a, b, c), differentiating with respect
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 59

Interpolating and Fitting 59

-4 -2 2 4

Fig. 3.8 Figure 3.3.1: even data points with least square fit

to the three variables, setting the resulting terms to zero and solving we
would get the following linear system.

a y1
AAT b = A . ,
c yn
where
2
x1 ... x2n

A = x1 ... xn
1 ... 1
There are similar expressions for the cubic least squares problem, etc.
The data shown in Figure 3.3.1 would seem to be cubic (see Exercise 1
below).
At the top of the section we mentioned that least squares fitting was
separate and distinct from linear regression. Before ending the section we
add some details to that statement. The setting for linear regression starts
with two random variables, X and Y , together with the hypothesis that Y
is a linear function of X. In particular, we are supposing that Y = aX + b,
where the parameters a and b are to be determined. Then the process is to
select the parameters so as to minimize the variance of Y X. When you do
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 60

60 Elements of Numerical Analysis with Mathematica

this calculation against sample data (supposing that the sample was done
with replacement), the process is exactly the degree 1 least squares fitting.
However, within the statistical context, the process returns values that
measure the correctness of the hypothesis and provide confidence intervals
for the two parameters. These are ideas special to statistical reqression not
shared with numerical analysis.
In the numerical analysis context there is no means to measure the cor-
rectness of the fit and no confidence intervals for the parameters. However,
least squares fitting is used to approximate the solution to a partial differ-
ential equation. In this case, the points that drive the least squares fitting
arise from numerical processes. We will have a means to measure how well
these values approximate the actual values and then use the least squares
process to fill between the known data.
We end this section with an important application. Exponential growth
is common in biology as well as the other sciences. For instance bacterial
growth is exponential. Epidemics show expontential growth during their
early stages. Exponential growth is characterized by the statement that
the rate of change of population size is proportional to the current size. In
particular, if f (t) represents the number of organisms in a bacteria growth
at time t, then rate of change for f is proportional to the value of f means
that
R T df /dt = f (t). Hence, by integrating both sides, we get f (T ) =
t
0 f (t)dt or f (t) = e , where = and = f (0).
Next, we turn this situation upside down. Suppose we have pairs (ti , yi )
of data, which because of the setting we know to be related via an exponen-
tial, yi = eti , but we do not know and . We can solve this problem
with least squares fitting. We write y = t and take the log of both sides.
This yields, log[y] = log[] + t. In this from, log[y] is a linear function of
t. Hence, we have the technique. First, we take the log of the yi , then fit
these values to the ti using a linear least squares fitting. The result is a
line, y = at + b. And b = and = ea . Exercise 6 is an example of this
sort of problem.

Exercises:
1. Fit the data P1 = (5, 3), P2 = (4, 2), P3 = (2, 7), P4 =
(0, 0), P5 = (1, 5), P6 = (3, 3), P7 = (5, 5) to a line (Figure 3.3.1), a
quadratic and a cubic. In each case, calculate the sum of squares from
the curve to the points. Which curve gives the best fit.

2. For the linear case prove that AAT is non-singular provided that
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 61

Interpolating and Fitting 61

there is a pair i and j, with xi 6= xj , as follows.

a. Let (x1 , ..., xn ) and (y1 , ..., yn ) be elements of Rn . Prove that
( i xi yi )2 = ( i x2i )( i yi )2 i<j (xi yj xj yi )2 .
P P P P

b. Set each yi = 1 in Part a and conclude that ( i xi )2 = n i x2i

P P
2
P
i<j (xi xj ) .
c. Prove that AAT is singular if and only if ( i xi )2 = n i x2i
P P

d. Conclude that AAT is nonsingular provided there is a pair i and j,

with xi 6= xj .

3. State and prove a result analogous to Exercise 2 for the case of the
quadratic least squares fitting.

4. Suppose that P1 , ..., Pn are points in R3 , Pi = (xi , yi , zi ) and let that

f (x, y) = a + bx + cy + dxy. Derive a procedure to determine a, b, c and d
so that i k(xi , yi , f (xi , yi )) (xi , yi zi )k2 is minimized. Note that we use
P

k(xi , yi , zi )k to denote the usual Euclidean length (x2i + yi2 + zi2 )1/2 .

5. Prove for the least squares fit of n + 1 points to an nth degree poly-
nomial A is the transpose of the Vandermonde matrix. Conclude the least
squares fitting for this case is equivalent to the polynomial interpolation
process described in Section 3.1.

6. The following data is known to be related via an exponential, y =

et . Use the procedure described in the section to identify and .

t 13.3522 10.6354 14.1693 9.67667 16.0862

y 10.8308 14.5888 31.5388 47.2135 53.374

t 6.221 4.93898 11.4164 9.13563 13.7273

y 1.66611 2.41672 4.70308 4.62067 7.95687

t 18.2355 13.243 16.2672 20.2108 15.1733

y 67.8773 98.8846 155.689 274.55 494.796

t 22.8416 23.3023 21.7688 23.9842 25.4667

y 754.759 758.462 1540.63 2059.3 2380.1

Plot the data along with y = e t on the same axis.

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 62

62 Elements of Numerical Analysis with Mathematica

7. This exercise is an extension of Exercise 8, Section 1. Recall that we

had a function
 t 1.27591
f (t) = 0.02424
303.16
that predicted the amount of tungsten production as a function of temper-
ature. Temperature is measured in degrees Kelvin. The following data is
an extension of the data given before.
a. Plot the data. By inspection determine the degree of least squares
that would fit the data best (degree 1, 2, or 3). Explain your decision.
b. Execute the least squares fitting and use it to predict the three values,
f (750). f (850) and f (950).
c. Determine the mean absolute error and compare the result to the
previous result. Which method was better?

t 300 400 500 600 700 800

f (t) 0.024 0.035 0.046 0.058 0.071 0.084

t 900 1000 1100 1200 1300 1400

f (t) 0.097 0.111 0.126 0.14 0.155 0.17

t 1500 1600 1700 1800 1900 2000

f (t) 0.186 0.202 0.219 0.235 0.252 0.269

3.4 Cubic Splines and B-Splines

The term spline refers to a large class of curves. Some interpolate the given
points while others fit the data. They are commonly used in many areas
of application, including statistics, probability, engineering and computer
graphics. One reason for their wide use is that splines exhibit local control.
Another advantage to cubic spline interpolation or fitting over polynomial
interpolation is that the curve can simulate asymptotic behavior. For this
reason these curves are often used in probability theory to approximate a
density function.
In this section we develop two classes of spline, the classical cubic spline
[Su and Liu (1989)] and the B-spline. In addition, cubic Hermite interpo-
lation introduced in conjunction to Gaussian quadrature is often referred to
as cubic orthogonal spline interpolation. We see this interpolation method
in the next section. The Bezier curve is a parametric curve. However, the
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 63

Interpolating and Fitting 63

spline is formed from several segments where each segment is a parametric

cubic. We state precisely,

Definition 3.4.1. Consider [a, b] R with a partition a = t0 < t1 < ... <
tn = b. A spline defined on [a, b] is a paramedic curve taking values in
Rm such that
a. for each i = 1, ..., n 1, restricted to (ti1 , ti ) is a parametric cubic,
denoted i ,
b. i (ti ) = i+1 (ti ), i = 1, ..., n 2,
c. is twice differentiable at each ti , i = 1, 2, ..., n 2.
The points (ti ) are called the knot points and the i are called the
segments.

The splines developed in this section will be twice continuously differen-

tiable at the knot points. Before continuing, we remark that Bezier curves
are splines with a single segment.
The original use of the term spline arose in drafting. A spline was a thin
wooden strip that a draftsman bends and pins on his table to provide a firm
edge against which he could draw a curve. As we shall see, the parametric
cubic is derived from the physical properties of the wooden object. The
result is a linear system with size equal to the number of points to be
interpolated.
We begin with the mathematical analysis of the draftsmans spline.
Specifically, this spline is a thin elastic strip (traditionally made of wood),
which is attached to a drafting table by pins and bent by weights (called
ducks). Considering the spline as a beam and the ducks as concentrated
loads on the beam, then the deformed beam satisfies the Bernoulli-Euler
equation
EIk(x) = M (x), (3.4.1)
where EI is a constant measures rigidity, M (x) is the bending moment and
k(x) denotes the curvature of the beam. We further suppose that M (x) is
a linear function of x. Letting (x) denote the arc represented by the bent
beam, then the curvature is given by
00
k(x) = .
(1 + 02 )3/2
We will need the following assumptions.
Assumption 1: the beam deflections are small, the tangent vector sat-
isfies k 0 k  1. Equivalently stated, 0 is negligible.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 64

64 Elements of Numerical Analysis with Mathematica

Hence, we may suppose that k(x) = ( 00 ). Since M 00 = 0, then the

fourth derivative (4) = 0. These leads to following.
Assumption 2: is twice continuously differentiable at each duck and
cubic between the ducks.
We are now in a position to solve for . In particular, we supose that
the physical spline has pins wtih x-coordinates at each xi , a = x0 < x1 <
... < xn = b. Furthermore, we require = i on [xi1 , xi ], where each i
is a parametric cubic, and is twice continuously differentiable at each xi .
Finally, we set 0 (xi ) = mi and 00 (xi ) = Mi .
Since the second derivative of i is linear, then we have
Mi1 (xi x) Mi (x xi1 )
i00 (x) = + , (3.4.2)
i i
where i = xi xi1 . Indeed, the right hand side of (3.4.2) is the unique
linear function which is Mi1 at xi1 and Mi at xi . Integrating (3.4.2)
twice we get successively
Mi1 (xi x)2 Mi (x xi1 )2
i0 (x) = + + C,
2i 2i

Mi1 (xi x)3 Mi (x xi1 )3

i (x) = + + Cx + D. (3.4.3)
6i 6i
Evaluating (3.4.3) at xi1 and xi we get
Mi1 (xi xi1 )3
yi1 = + Cxi1 + D,
6i

Mi (xi xi1 )3
yi = + Cxi + D,
6i
were we write yi = (xi ). Subtracting yi1 from yi we have
Mi (xi xi1 )3 Mi1 (xi xi1 )3
yi yi1 = + C(xi xi1 ).
6i 6i
Dividing through by i = xi xi1 yields,
yi yi1 i
= (Mi Mi1 ) + C.
i 6
Now solving for C and substituting into (3.4.3) we get
(xi x)2 (x xi1 )2
i0 (x) = Mi1 + Mi +
2i 2i
yi yi1 i
(Mi Mi1 )
i 6
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 65

Interpolating and Fitting 65

Using similar but significantly more complicated calculations we get

yi1 i yi i
D= xi + Mi1 xi xi1 + Mi xi1
i 6 i 6
Hence, we have
Mi1 (xi x)3 Mi (x xi1 )3 yi yi1
i (x) = + +
6i 6i i

i yi1 i yi i
(Mi Mi1 ) x+ xi + Mi1 xi xi1 + Mi xi1 . (3.4.4)
6 i 6 i 6
Note that the unknowns in equation (3.4.4) are Mi1 and Mi , and that
the equations are linear in these two variables. That is, (3.4.4) describes
a linear system of equations. We now organize this equation to get an
expression for the second derivatives (the Mi ) without reference to x, the
independent variable, and then write the resulting linear system in matrix
form.
First, we write (3.4.4) is a slightly more compact form,
Mi1 (xi x)3 Mi (x xi1 )3
i (x) = + +
6i 6i

yi i
yi1 i

Mi (x xi1 ) + Mi1 (xi x).
i 6 i 6
And for i+1 we get
Mi (xi+1 x)3 Mi+1 (x xi )3
i+1 (x) = + +
6i+1 6i+1

yi+1 i+1
yi i+1

Mi+1 (x xi ) + Mi (xi+1 x).
i+1 6 i+1 6
Now we differentiate these last two expressions with respect to x, eval-
uate the derivative at xi and use i0 (xi ) = i+1
0
(xi ) to get
i yi i yi1 i
Mi + Mi + Mi1 =
2 i 6 i 6

i+1 yi+1 i+1 yi i+1

Mi + Mi+1 + Mi .
2 i+1 6 i+1 6
Now separating the terms, we have
i i i+1 i+1 yi+1 yi yi+1 yi
Mi1 + Mi + Mi + Mi+1 = .
6 3 3 6 i+1 i
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 66

66 Elements of Numerical Analysis with Mathematica

Next, we multiply by 6 and divide by (i+1 + i ),

i i
Mi1 + 2Mi + Mi+1 =
i+1 + i i+1 + i

6 i+1 yi yi+1 yi 
y
. (3.4.5)
i+1 + i i+1 i
Equation (3.4.5) represents a linear system of equations. There are
n + 1 unknowns, Mj , j = 0, ..., n and n 1 equations associated to the
spline segments, i , i = 1, ..., n 1. Setting boundary values at M0 and
Mn adds two more equations to the system,
M0 = 0 , Mn = n . (3.4.6)
If we denote the fight hand side of (3.4.5) by di and set i /(i+1 +i ) = i ,
i+1 /(i+1 + i ) = i , then (3.4.5) and (3.4.6) yield the following lower
triangular linear system in matrix form.

1 0 0 ... 0 0 0 0 M0 0
2 2 0 ... 0 0 0 0 M1 d1


2 ... 0 0 0 0 M2 d2

2 3
0 2 ... 0 0 0 0 M3 d3

3
... ... ... ... ... ... ... ... ... = ... (3.4.7)


0 0 0 ... n3 2 n2 0 Mn3 dn3

0 0 0 ... 0 n2 2 n1 Mn2 dn2

0 0 0 ... 0 0 n1 2 Mn1 dn1
0 0 0 ... 0 0 0 1 Mn n
The solution to (3.4.7) is a peicewise polynomial function. It is not
a polynimial, but rather the join of polynimial segments. Further, the
spline curve interpolates the given set of points. Hence, we have a process
similar to polynomial interpolation. However, based on the physical model,
splines will have local control. In particular, the problem that we identified
wtih polynomial interpolation cannot happen here. However, the size of
the linear system in (3.4.7) depends on the number of knot points or spline
guide points. There is an alternative called B-spline. B-splines are piecewise
cubics that exhibit local control and do not require that we solve a linear
system. The downside that they do not interpolate the guide points.
The basic idea is to remove the physical context from the spline while
keeping the basic properties. In particular, we want a priecewise cubic poly-
nomial that is twice continuously differentiable where the segments join. In
particular, suppose we have a list of distinct points, Pi = (xi , yi ), i = 0, ..., n.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 67

Interpolating and Fitting 67

Suppose further that xi xi+1 . Now seek a function defined on [x1 , xn ]

with the following properties. Each segment i equal to on the subinter-
(j) (j)
val [xi , xi+1 ] is a cubic polynomial so that for each i, i (xi ) = i+1 (xi )
for each j = 0, 1, 2. In short, the function is locally a cubic polynomial
and globally twice continuously differentiable.
We carry this line of thought a step further. We know that P3 , the set
of all polynomials of degree less than of equal to three, is a four dimensional
vector space. Does P3 have a basis p1 , p2 , p3 , p4 with the property. For any
x1 , x2 , x3 , x4 and x5 in R,
(j) (j) (j) (j)
x1 p1 (1) + x2 p2 (1) + x3 p3 (1) + x4 p4 (1) =

(j) (j) (j) (j)

x2 p1 (0) + x3 p2 (0) + x4 p3 (0) + x5 p4 (0), (3.4.8)
for derivatives j = 0, 1, 2? If the answer is affirmative, then we may set
P P
1 (t) = i xi pi (t) and 2 (t) = i xi+1 pi (t) and join 1 and 2 to form a
piecewise cubic, twice continuously differentiable function on [0, 2].
P3 k
If we set pi (X) = k=0 ik X , then the three constraints given in
(3.4.8) become
4
X
(xi xi+1 )i0 + xi i1 + xi i2 + xi i3 = 0 (3.4.9)
i=1

4
X
(xi xi+1 )i1 + 2xi i2 + 3xi i3 = 0 (3.4.10)
i=1

4
X
(xi xi+1 )i2 + 6xi i3 = 0 (3.4.11)
i=1

Since these equations may hold for any xi , we set x1 = x2 = x3 = x4 = 0

and x5 = 1 in (3.4.9) to yield 40 = 0. In turn (3.4.10) implies that
41 = 0 and (3.4.11) implies that 42 = 0. Therefore, p4 (X) = 43 X 3 .
Continuing in this manner, we derive p1 (X) = 40 (X 3 + 3X 2 3X + 1),
p2 (X) = 41 (3X 3 6X 2 + 4), p3 (X) = 42 (3X 3 + 3X 2 + 3X + 1). (See
Exercise 7.) We normalize by setting each 4i = 1/6. With this choice,
P
i pi = 1. These four polynomials are called the B-spline basis functions.
The B-spline basis functions may be used to define a powerful fitting
process. Given a list of points, Pi = (xi , yi ), i = 1, ..., n, we write,
4
X 1
j (t) = pi (t)Pj+i1 = (t3 + 3t2 3t + 1)Pj + (3t3 6t2 + 4)Pj+1 +
i=1
6
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 68

68 Elements of Numerical Analysis with Mathematica

(3t3 + 3t2 + 3t + 1)Pj+2 + t3 Pj+3 ,


(3.4.12)

where t [0, 1] and j n 3. We have the following theorem.

Theorem 3.4.1. Given points Pi = (xi , yi ), i = 1, ..., n, the function (t)

defined by joining the segments defined in (3.4.12) is a parametric function.
Further, is piecewise cubic and twice continuously differentiable.

Proof. The result follows from the properties of the B-spline basis func-
tions.

We call the B-spline fit for the given point set.

We carry this a step further and suppose that f is a function with values
f (xi ) = yi , and set = maxi i . Wtih this notation we state the following
theorem. The proof is beyond the level of this book.

Theorem 3.4.2. Suppose f is a smooth function defined on a real interval

[a, b]. Suppose that a = x0 x1 ... xn = b is a partiiton of the domain
denoted by P. We take Pi = (xi , f (x) ) and write P for the associated
B-spline fit. It follows that P f as 0

Proof. See [Loustau (2016)].

In particular, the B-spline fit for the function is not an interplation but it
does approximate the function.
Finally, we consider local control for B-splines. First, note that each
point on a sgment i is a linear combination of the four guide points for
the segment. Furthermore, as the sum of the B-spline basis functions is
1, then sum of the coefficients in the linear combination is also 1. Hence,
the curve lies within the convex hull of the four guide points. Note that
the convex hull of a point set is the smallest convex set that contains the
points. For 4 points, it is the convex quadralateral or triangle that contains
the points. The following diagram illustrates the convex hull property. The
spline segment is generated by the four points.
In summary, B-splines are parametric cubic curves which fit the given
set of points, guide points. Each B-spline segment is determined by four
of the guide points. For instance, segment j is determined by the four
guide points, Pj , Pj1 , Pj+2 , Pj+3 . In practice each segment traces a curve
very close to its four determining guide points. The B-spline curve (the
union of the segments) is twice continuously differentiable. This high level
of smoothness is the reason that B-splines are so often used.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 69

Interpolating and Fitting 69

2.0
P2
1.5

1.0
P4

0.5

P1
1 2 3 4 5

-0.5
P3
-1.0
Figure 3.4.1: The B-spline segment and the convex hull.

Fig. 3.9 Figure 3.4.1: The B-spline segment and the convex hull

In addition, you can predict the effect on the curve that arises from
the change in one of the guide points. This is because of the convex hull
property. Any given guide point is included in the calculation of at most
four curve segments, or equivalently at most four of the bounding convex
hulls. Therefore, if you change a guide point, then you can predict the
change in the curve by looking at how the convex hulls change. Figures
3.4.2 shows a two segment B-spline. The convex hull for the first segment
is formed by P1 , P2 , P3 , P4 . For the second segment the convex hull is
determined by P2 , P3 , P5 , P4 . In Figure 3.4.3, we have moved P3 .
Smootheness, local control and convergence are two strong advantage
for B-splines over polynomials. Since there is no need to solve a large linear
system, then B-splines are preferred over cubic splines. Indeed, unless exact
interpolation is absolutely required, B-splines are often the technique of
choice.

Exercises:
1. Plot the four B-spline basis functions.

2. Compute the B-spline fit to the points (xi , 1/(1 + x2i ) for xi =
4, 2, 1, 0, 1, 2, 4. Plot the parametric B-spline against the graph of
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 70

70 Elements of Numerical Analysis with Mathematica

2.0 2.0
P2 P2
1.5 1.5

1.0 1.0
P4 P4

0.5 0.5

P1 P5 P1 P5
1 2 3 4 5 6 1 2 3 4 5 6
A A
-0.5 -0.5
P3 P3
-1.0 -1.0
Figure 4.3.2: Two B-spline segments with knot point A Figure 3.4.3: Two B-splines, P3 has been moved to the left

Fig. 3.10 Figure3.4.2: Two B-spline Fig. 3.11 Figure 3.4.3: Two B-splines,
segments with knot point A P3 has been moved to the left

f (x) = 1/(1 + x2 ). Compare your output with Exercise 3 of Section 3.1.

3. In Exercise 3, the fourth guide point is (0, 1). Leaving the other
guide points unchanged, repeat Exercise 3 using (0, 1.1) for the fourth
guide point. Compare the output to the curve in Exerciese 2.

5. Repeat Exercise 3 with fourth guide point equal to (0, 1.2) or (0.0.9)
or (0, 0.8).

6. Repeat Exercise 7 of Section 3 with the same data. This time fit
B-Splines to the data and plot the curve on the same graph with the data.
Use the B-Spline to predict values for f (750). f (850) and f (950). Deter-
mine the mean absolute error and compare the result to the result using
least squares fitting. Which method was better?

7. Complete the derivation of the B-Spline basis functions.

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 71

Interpolating and Fitting 71

3.5 Hermite Interpolation

Hermite interpolation goes one step beyond polynomial interpolation. In

Section 3.1, we considered a function f f and a finite set of onterpolation
points xi , i = 0, 1, ..., n in the domain. In this setting we determined a
poltynomial p that matched f at the given points. Now we assume that f
is differentiable and we require that f (xi ) = p(xi ) and f 0 (xi ) = p0 (xi ) at
each of the interpolation points. The resulting polynomial will be called
the Hermite interpolation
We will see in the Chapter 5 that Hermite interpolation gives rise to
Gaussian quadrature, the generally accepted gold standard of numerical
integration techniques. But that is just one of the applications. In the
case of piecewise interpolation (introduced in Section 3.1) the resuling in-
terpolant will be C 2 , This has implcations for finite element method in one
spatial dimension or for rectangular higher dimensional domains. In an-
other direction, the need to model both the function and its derivative leads
to discontinuous Galerkin method. There are at least two more application
of Hermite interpolation in the area of numercial differential equations. One
is OSC or orthogonal spline colocation and the other is Hermitian method,
an interesting technique designed specifically for non-lineasr equation in
one dimension.
We begin with the formal definition.

Definition 3.5.1. Suppose that f is a differentiable function on [a, b] and

take a x0 < x1 < ... < xn1 < xn b. The expression
n
X n
X
h(x) = f (xi )Hi (x) + f 0 (xi )Si (x)
i=0 i=0

is a Hermite interpolation of f provided the polynomials Hi and Si are

degree 2n + 1 and satisfy,
(i) Hi (xj ) = i,j , Si (xj ) = 0
(ii) dHi /dx(xj ) = 0, dSi /dx(xj ) = i,j .

Our first task is to show that the Hermite interpolation exists; that is,
polynomials exist that satisfy the conditions of definition 4.4.1. We start
with the n + 1, nth degree Lagrange polynomials associated to the points
xi , i = 0, .., n. Recall from Section 3.1,
Q
j6=i (x xi )
li (x) = Q .
j6=i (xj xi )
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 72

72 Elements of Numerical Analysis with Mathematica

Since li (xj ) = i,j , then the same is true for hi = li2 . Furthermore, if we set
Qn
p(x) = i=0 (x xi ), then
XY Y
p0 (x) = (x xj ); p0 (xi ) = (xi xj )
i j6=i j6=i

0
Hence, li (x) = p(x)/[(x xi )p (xi )]. We compute
p0 (x)
 
d d 1 p(x) 1 p(x) 1
hi = 2li (x) li (x) = 2 +
dx dx x xi p0 (xi ) (x xi )2 p0 (xi ) x xi p0 (xi )
0
Therefore, for j 6= i, hi (xj ) = 0.
At this point we have n+1 functions hi , which are polynomials of degree
2n. Next, we seek polynomials ui and vi satisfying the following,
0 0
a. degree ui = 1, ui (xi ) = 1, ui (xi ) = hi (xi );
0
b. degree vi = 1, vi (xi ) = 0, vi (xi ) = 1.
Indeed, in this case Hi = ui hi and Si = vi hi satisfy the conditions of
Definition 4.1.1. It is immediate that vi (x) = x xi is consistent with item
0
b above. In turn, we set ui (x) = (x xi ) + 1 where = hi (xi ). It is
now a simple matter to complete the proof of the following Theorem.

Theorem 3.5.1. Any differentiable function on an interval has Nermite

interpolation.

With existence in hand, the next task is to estimate the error.

Theorem 3.5.2. Suppose that f is 2n + 2 times continuously differentiable

on (a, b) and that h is the Hermite interpolation of f with respect to a
partition of [a, b]. If we denote the error, e(x) = f (x) h(x), then for each
x
, there is x in the interval with
f (2n+2) (x )
e(
x) = x) 2 .
p( (3.5.1)
(2n + 2)!

If M = kf (2n+2) k , then
M
|e(
x)| x)2 .
p( (3.5.2)
(2n + 2)!
Furthermore, if f is a polynomial of degree less than or equal to 2n + 1, then
f = h.

Proof. The proof of this is analogous to the derivation of the error esti-
matge for polynomial interpolation as developed in Section 3.1. As in that
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 73

Interpolating and Fitting 73

case, it is an application of Rolles theorem. We will set up the argument

and leave the final details to the reader. To begin, we set
e(x)
e(x) = f (x) h(x); g(x) = .
p(x)2
Next, we take distinct for any of the xi and define
k(x) = f (x) h(x) p(x)2 g()
Now, k has n + 2 distinct roots at the xi and at . By Rolles theorem,
k 0 has n + 1 distinct rootes each distinct from the roots of k. In addition,
since h is the Hermite interpolation, the xi are also roots of k. This gives
us at least 2n + 2 distinct roots for k 0 . Therefore, k (2n+2) has at least one
root. Now complete the proof as in Theorem 3.1.3.
Equation (3.5.2) is an easy consequence of (3.5.1). The final assertion
is immediate.

There is a special case of Hermite interpolation that is too important to

omit. The following 4 polynomials form a basis for the degree 3 polynomials
on [, ]
 2  3  2  3
x x x x
H0 = 3 2 , H1 = 3 2 ,

4
( x)2 ( x)3 (x )2 (x )3
S0 = 2
, S1 = + .
( ) ( )2
These polynomials are called the Hermite cubic polynomials The function
H0 satisfies, H0 () = 1, H0 () = 0 and both derivatives of H0 at the end
points are zero. Whereas, S0 vanishes at the end points, and the slope at
is 1 and the slope at is zero. H1 and S1 have analogous properties.
It is not difficult to prove that the 4 polynomials are linear independent.
Indeed, if aH0 + bH1 + cS0 + dH1 = 0 then
0 = aH0 () + bH1 () + cS0 () + dH1 () = aH0 () = a.
Similarly, b = c = d = 0 Wek use a dimension argument to conclude that
the four Hermite cubics form a basis for the degree three polynomials.
These degree three polynomials are connonly used to model the solution
to a second order PDE.

Exercises
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 74

74 Elements of Numerical Analysis with Mathematica

1. Verify that when n is 1, a = x0 and b = x1 , then the Hermite cubics

provide a Hermite interpolation of [a, b].

2. Complete the verification that the functions ui hi and vi hi satisfy the

requirements for a Hermite interpolation.

3. Complete the proof of Theorem 3.5.1.

4. Verify that the Hermite cubics determine a Hermite interpolation on

the interval [, ].

5. Plot the 4 Hermite cubics and verify by inspection that they satisfy
the requirements of Definiton 3.5.1.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 75

Chapter 4

Numerical Differentiation

Introduction

In this chapter we begin the study of the basic constructs of calculus from
the standpoint of numerical analysis. In particular, we look at procedures
to approximate the derivative of a function when we only know values of
the function but have no closed form or series representation. In the next
chapter we consider the integral.
We begin by introducing the finite difference as the basic technique to
approximate the derivative of a function from only knowledge of some of the
function values. For instance, if we know values of our function f at points
a and b, then (f (b) f (a))/(b a) is approximately equal to the derivative
provided b a is small. From the numerical analysis point of view, we want
an error estimate for any approximation. For this setting, if we use the
Taylor expansion to derive the expressions for the finite differences, then
the error estimation procedure is given.
Finite differences is the entry point into numerical processes to approx-
imate the solution of a differential equation. In the second section, we
introduce the finite difference method (FDM) to estimate solutions of the
one-dimensional heat equation. in this section we consider only explicit
FDM. On one hand the technique is successful. However, almost immedi-
ately we see that this technique is can give rise to unacceptable levels of
error. This leads us to consider von Neumann stability questions in Section
3. In Section 4, we introduce Crank Nicolson FDM. Stability is not an issue
for this procedure. This is often the method of choice for parabolic partial
differential equations (PDE). In the exercises we consider FDM as applied
to the order 1 wave equation. We see here that FDM for hyperbolic PDE
is very different. Little of the intuition gained from the parabolic case can

75
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 76

76 Elements of Numerical Analysis with Mathematica

be brought forward to the hyperbolic equations.

4.1 Finite Differences and Vector Fields

If we start with a function, f which is real valued and differ-0entiable at x0 ,

then the derivative is given by f 0 (x0 ) = limxx0 (f (x) f (x0 ))/(x x0 ).
Hence, for x near to x0 , the derivative at x0 is approximately equal to
(f (x) f (x0 ))/(x x0 ). This is the basic idea behind the finite difference.
Finite differences are useful because the researcher often knows values of a
function without specifically knowing the function. With finite differences,
the researcher can infer approximate values of the derivative of an unknown
function. We will see an example of this later in the section. Furthermore,
we see in subsequent sections that finite differences provide a valuable tool
for approximating the solution of a differential equation. We turn now to
the formal definiton.
Let f be real valued function defined on an interval [a, b]. For a positive
integer n set x = (b a)/n and consider a uniform partition a = x0 <
x1 < ... < xn = b, where for each i, xi xi1 = x. Suppose further that
f is three times continuously differentiable, so that we may write the order
three Taylor expansion for f at xi evaluated at xi+1 .
df 1 d2 f 2 1 d3 f
f (xi+1 ) = f (xi ) + (xi )x + (x i )x + (xi )x3 + O(x4 ).
dx 2 dx2 6 dx3
(4.1.1)
The term O(x4 ) is derived from the Taylor remainder term. In particular,
this term converges to zero order 4 as x goes to zero.
In turn, we have
df 1 d2 f 2 1 d3 f
f (xi1 ) = f (xi ) (xi )x + (x i )x (xi )x3 + O(x4 ).
dx 2 dx2 6 dx3
(4.1.2)
In (4.1.1), subtracting f (xi ) from both sides and dividing by x yields
f (xi+1 ) f (xi ) df 1 d2 f 1 d3 f
= (xi ) + 2
(xi )x + + (xi )x2 + O(x3 ) =
x dx 2 dx 6 dx3
df
(xi ) + O(x). (4.1.3)
dx
If we do the same to (4.1.2) we obtain
f (xi ) f (xi1 ) df
= (xi ) + O(x). (4.1.4)
x dx
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 77

Numerical Differentiation 77

Note that we only need f to be C 2 to derive Equations (4.1.3) and (4.1.4).

We state formally,

Definition 4.1.1. Consider a differentiable real valued function f defined

on the interval [a, b] with uniform partition a = x0 < x1 < ... < xn = b and
x = xi xi1 . The first forward difference and first backward difference
for f are given by
f (xi+1 ) f (xi ) f (xi ) f (xi1 )
; .
x x
The following theorem follows immediately from (4.1.3) and (4.1.4). We
continue the current notation.

Theorem 4.1.1. If f is twice continuously differentiable on [a, b] then the

first forward and backward differences at xi converge to f 0 (xi ) order x.

Next, we add (4.1.1) to (4.1.2f) and divide by 2 to get

f (xi+1 ) f (xi1 ) df 1 d3 f
= (xi ) + (xi )x2 + O(x3 ) =
2x dx 6 dx3
df
(xi ) + O(x2 ). (4.1.5)
dx
In turn, if we subtract (4.1.2) from (4.1.1) and divide by x we get
f (xi+1 ) 2f (xi ) + f (xi1 ) d2 f
2
= (xi ) + O(x2 ). (4.1.6)
x dx2
These two equations define the central differences approximating the first
and second derivatives of f .

Definition 4.1.2. Consider a twice differentiable real valued function f

defined on the interval [a, b] with uniform partition a = x0 < x1 < ... <
xn = b and x = xi xi1 . The first central difference and second central
difference for f are given by
f (xi+1 ) f (xi1 ) f (xi+1 ) 2f (xi ) + f (xi1 )
; .
2x x2
Equations (4.1.5) and (4.1.6) resolve the order of convergence for the central
differences. It is important to note that the central difference s are better
behaved that then the forward and backward differences. We will develop
this more in the exercises.

Theorem 4.1.2. If f is three times continuously differentiable on [a, b]

then the first central and second central differences at x1 converge to f 0 (xi )
and f 00 (xi ) order x2 .
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 78

78 Elements of Numerical Analysis with Mathematica

When defining the finite differences, we used a uniform partition along

the x-axis. Generally speaking, finite differences perform best when the
partition is uniform. But there are many other forms of the finite difference
which are used in particular circumstances. With the aid of these more
advanced techniques applied to certain special cases, researchers can achieve
good results with non-uniform partitions. For our purposes here, we will
restrict attention to the uniform case. For a more complete treatment, see
[Loustau (2016)].
Consider the following setting from computational fluid dynamics. Sup-
pose we have a partially obstructed channel as shown in Figure 4.1.1. The
upper and lower boundaries are the channel edges, the right and left mar-
gins are the inflow and outflow edges. We will suppose that the fluid moves
from left to right. By results from complex analysis there is a real valued
flow potential function defined on the region in D in Figure 4.1 that sat-
isfies the Laplace equation, 2 = 0. In this case, = (x , y ) defines a
vector field that represents a fluid flow through the channel. We know that
exists, but have no means to solve for it. Nevertheless, using advanced
numerical techniques, we can approximate this function at locations in D.
We do this for a uniform grid (xi , yj ), xi+1 xi = x, yj+1 yj = y. We
show the grid in Figure 4.1.2. There are about 230 locations in the grid.
Next, we use forward differences to approximate the vector field,
(xi+1 , yj ) (xi , yj ) (xi , yj+1 ) (xi , yj )

,
x y
The resulting vector field (see Figure 4.1.3) illustrates the fluid flow.
We used the Arrow function in Mathematica graphics. The length of the
vectors represents the speed of the flow at the location.
In Exercise 2, we use finite differenes to approximate the flow field about
cylinder. In this case, we know the flow potential function. Hence, for this
example we can compare the finite difference approximations to the actual.

In Exercise 3, we know the complex potential function but cannot get

the real part and compute the gradient. Finite differences provide a means
to do this computation.

Exercises:
1. Take f (x) = x2 and x = 0.1.
a. Compute the first forward, backward and central differences at x =
1, 2, ..., 100. Compare the ouput against the actual derivative. Compute
the mean absolute error for each case.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 79

Numerical Differentiation 79

Figure 4.1.2: The channel with uniform grid in $x$ and $y$ directions
Figure 4.1.1: The partially obstructed channe

Fig. 4.1 Figure4.1.1: The partially ob- Fig. 4.2 Figure 4.1.2: The channel
structed channel with uniform grid in x and y directions

b. Which finite difference percedure is better? Why?

c. Compute the second central difference at the given locations. Com-
pare the output against the actual second derivatives and compute the
mean absolute error.

2. Consider the following function, u(x, y) = x + x/(x2 + y 2 ) defined

on the domain D = {(x, y) : 2 x 2, 0 y 2, x2 + y 2 1}. We
can think of this domain as a partially obstructed channel. In this case the
obstruction is a semi-circle.
a. Graph the domain D.
b. Create a list of points in D so that Pi = (xi , yi ) is in the list provided
xi = 2 + 0.2m and yi = 0.2n for some positive integers m in {1, 2, ..., 19}
and n in {1, 2, ..., 9} . Before proceeding plot the list of points to be certain
you have all points in D whose x and y coordinates step by increments of
0.2 in both the x and y direction.
c. For each i, plot the vector (xi , yi ) + u(xi , yi ). Use the arrowhead
feature when drawing the vectors. You should see the vector field describ-
ing the flow of a non-viscous, incompressible fluid around a semi-circular
obstruction.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 80

80 Elements of Numerical Analysis with Mathematica

Figure 4.1.3: The flow field approximated with finite differences

Fig. 4.3 Figure4.3: The flow field approximated with finite differences

d. For the interior points of D, replace the gradient in c with the central
difference in the x and y directions, (u(xi +0.2, yi )u(xi 0.2, yi ), u(xi , yi +
0.2) u(xi , yi 0.2)). Plot the vector field.
e. Compare the output in c against the output in d. Compute the mean
normed error.

3. The function in this exercise arises in aerospace engineering. Consider

the following function f : C C, f (z) = z + 1/z defined on the complex
plane, z = x + iy. The inverse of f is given by
1 p
g(z) = (z + z 2 4)
2
Further, we can think of g as a function from R2 to R2 ,
1 p
g(x, y) = (x + iy + x2 y 2 + 2ixy 4) = (Re(f (x, y)), Im(f (x, y))),
2
the real and imaginary parts of f . In fluid dynamics it is inportant to know
the real part of the gradient of g for given locations in the domain.
a. Write a program that appriximates the gradient of g with central
differences.
b. Extend the program in Part a to take a given (x0 , y0 ) in the domain
of g and approximate the real part of the gradient at that location.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 81

Numerical Differentiation 81

4.2 Finite Difference Method, Explicit or Forward Euler

In this section we introduce finite difference method (FDM), one of basic

techniques used to approximate the solution to a partial differential equa-
tion, PDE. This a classical technique that became more popular with the
development of computers. The theory to suport it dates from the 1950-
1970, [Loustau (2016)]. Our treatment of the finite difference method given
here and in the following sections is by no means exhaustive. It will serve
to only convince the reader that finite differences are useful. The particular
form of FDM is referred to as explicit FDM or forward Euler FDM.
Consider the following common setting. Suppose an unknown function
is defined on an interval of the x-axis and a time interval, u = u(x, t),
u : [a, b] [0, 1] R and satisfies the following differential equation.
u 2u
= 2. (4.2.1)
t x
This equation is called the one dimensional heat equation , as it represents
the flow of heat across a thin rod. However, this equation arises in settings,
which have nothing to do with heat flow. Indeed, it represents diffusion
process. For now, we focus on the usual heat flow setting. In this context,
the values u = u(x, t) represent temperatures at the location x and time t.
Suppose we partition the intervals [a, b] and [0, 1] in a uniform manner,
a = x0 < x1 < ... < xn = b and 0 = t0 < t1 < ... < tN , x = xi xi1 and
t = tn tn1 . We can subsequently recast (4.2.1) in finite difference for-
mat using the forward difference for time and the second central difference
for space.
u(xi , tn+1 ) u(xi , tn ) u(xi+1 , tn ) 2u(xi , tn ) + u(xi1 , tn )
= + ,
t x2
where  is the cutoff error inherent in using finite differences for the deriva-
tives. If we set = t/x2 then we get the following expression,
u(xi , tn+1 ) = u(xi+1 , tn ) + (1 2)u(xi , tn ) + u(xi1 , tn ) + .
For the heat context, this latter expression may be phrased as, the temper-
ature at location xi and the next time tn+1 is a linear combination of the
temperatures at time the current time tn and location xi and its neighbors
with a small error. In other terms, we have a interative process that infers
temperatures in suceeding times from the current temperatures.
If we surpress the error term, then we should change the notation slightly
to reflect that the numbers are not exact, but finite difference approxima-
tions. In this context, it is standard to write uni for the vlaue u(xi , tn )
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 82

82 Elements of Numerical Analysis with Mathematica

computed with finite differences. With the new notation we write

un+1
i = uni+1 + (1 2)uni + uni1 . (4.2.2)

In addition, we write un to denote the nth time state, the column vector
whose entries are uni . In the notation of the topic, this rendering or choice
of finite differences is referred to as FTCS , for forward time, central space.
It is important that the problem has unique solution. In the literature
this is referred to setting a well posed problem. We generally assume that
the underlying physics is deterministic. The requirement that the problem
have an unique solution is consistent with this assumption. If we look at
(4.2.1) we realize that any multiple of a solution u is also a solution. We
can pin this down by designating the temperature at the boundary. In our
finite difference setting this is equivaluent to declaring prior knowledge of
the temperatures at the boundary. In particular we must designate un0 and
unk+1 for all n. In addition we will need the initial values, u0i for all locations
xi .
This question of uniqueness arises also in the discrete setting. Notice
that (4.2.2) defines a linear relation between states, un+1 = Aun . In matrix
form this relation is
n n+1
1 2 0 ... 0 0 0 u0 u0
1 2 ... 0 0 0 un un+1
1 1
0 1 2 ... 0 0 0 un un+1
2 2
... ... ... ... ... ... ... ... ... = ... .

n n+1
0 ... 1 2

0 0 0 uk1 uk1
n n+1
0 0 0 ... 1 2 uk uk
0 0 0 ... 0 1 2 unk+1 un+1
k+1
(4.2.3)
The matrix in (4.2.3) is nonsingular. It is symmetric and the eigenvalues
are known and nonzero for any k and any . (See [Loustau (2016)].)
Therefore, given an initial state u0 , each successive state is completely
determined, un = An u0 . Therefore, the 1D heat equation is inherently well
posed as a FDM problem. But just as noted, any multiple of this matrix
will also transform each state to the next and still be a discrete form of
Equation (4.2.1). We remove this ambuiguity by designating boundary
values, a = x0 and b = xk+1 . Generally, these values are known to us.
In this case, we set values that may or may not be time dependent. For
instance, if the boundary values at time tn+1 are n+1
0 and n+1
k+1 , then the
linear relation (4.2.3) is modified as
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 83

Numerical Differentiation 83

n+1 n+1
1 0 0 ... 0 0 0 0 0
1 2 ... 0 0 0 un un+1
1 1
0 1 2 ... 0 0 0 n n+1
u2 u2
... ... ... ... ... ... ... ... ... = ... . (4.2.4)

n n+1
0 ... 1 2

0 0 0 uk1 uk1
unk un+1

0 0 0 ... 1 2 k

0 0 0 ... 0 0 1 n+1
k+1 n+1
k+1

The transformation between the format (4.2.3) and (4.2.4) can be mathe-
matically justified. See for [Loustau (2016)].
It is most important to realize that equation (4.2.1) defines a linear
process that transforms the temperatures at one time state to the next. In
the discrete form we realize the linear process by the matris in (4.2.4).
Consider the following example. Suppose there is a thin rod initial which
is insulated along its length. Suppose that the temperature is initially zero
everywhere, and that the left end is suddenly heated and kept at 20 degrees.
Finally we set = 1/2. In notation we have set the spatial interval is
[5, 5], x = 0.1, t = 0.01 and = 1/2. For the initial setting take
u(x, 0) = 0 and for the boundary values take u(5, t) = 20 and u(5, t) = 0.
Using (4.2.4) we solve for approximate values of u along the interval. The
following plot shows that local temperatures after 10 time steps.
This is the basic idea behind the finite difference method for solving
differential equations. The explicit or forward Euler method and other
related techniques are remarkably successful (see Section 4). On the one
hand any differential equation may be rendered in FDM form. However, all
these techniques have their limitations. Compare the output from Exercises
1 and 2 or 5 and 6 below. The change in causes dramatic changes in the
output. Indeed, the output of Exercise 2 is impossible. The difficulty arises
because of computational error. In the next section we develop a technique
that demonstrates how is related to the error.
Problem 4 describes an application from cancer therapy. The specific
context is the delivery of chemotherapy across a cell membrane.

Exercises:
1. Execute the example from the text using forward Euler method.
Take = 1/2, the spatial interval as [5, 5], x = 0.1, t = 0.01. For
the initial state take u(xi , 0) = 0 for every i and for the boundary val-
ues take each u(5, tn ) = 20 and u(5, tn ) = 0. Plot the temperatures at
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 84

84 Elements of Numerical Analysis with Mathematica

20

15

10

-4 -2 2 4
Figure 4.2.1: Temperatures distribution after 10 iterations

Fig. 4.4 Figure4.2.1: The temperature disribution after 10 iterations

t = 0.03, 0.06, 0.08, 0.1.

2. Redo Problem 1 with = 2. Notice that the results are not well
behaved. The problem here is that the forward Euler method for this ge-
ometric setting is not stable then > 0.5. Stability of transient or time
dependent processes is covered in following section.

3. Redo Problem 1 with = 2 and t = 0.001.

4. Diffusion of a chemical through a permeable membrane is given

by u/t = 2 u/x2 , where u(t, x) represents the concentration of the
chemical at location x and time t. The constant is determined by the
membrane. This setting arises when modelling the absorption of a sub-
stance into a cell. We want to use FDM to model this diffusion through a
membrane and determine when stasis occurs.
Consider a 4 point model, x0 and x1 outside the cell and x2 , x3 in-
side the cell. Set = 0.05, x = 0.25, t = 0.01. Set up a FDM
model with x0 and x3 as boundary points. The initial condition will be
u(0, x0 ) = u(0, x1 ) = 0.075 and u(0, x2 ) = u(0, x3 ) = 0.025. We further
assume that the conentraction is constant at the bondary. We want to de-
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 85

Numerical Differentiation 85

termine t and u(t, x2 ) when stasis has been reached. For our purposes we
define stasis as u(tn+1 , x2 ) u(tn , x2 ) < 0.0001.

5. Consider the first order wave equation. Given u = u(t, x),

u u
= .
t x
a. Using forward differences, render this PDE in finite difference format.
In this case it is usual to write C = t/x. C is called the Courant
number
b. Execute FDM for the following setting.
= 300, Region: [0, 300], Time interval [0, 5],
Initial Conditions:
u(0, x) = 0, 0 x 50; 
u(0, x) = 100 sin (x50)
60 , 50 x 110;
u(0, x) = 0, 110 x 300.
Boundary Values: u(t, 0) = 0 and u(t, 300) = 0,
x = 5; t = 0.0015.
Use B-splines to display the output at time states 1, 3, 5.

6. Redo Problem 5 with = 300.

4.3 Neumann Stabiity Analysis

In the last section we developed a very general technique that seemed to

allow us to resolve any order 2 PDE. Then in the exercises we encountered
problems. Further there did not seem to be an apparant resolution. Cer-
tainly, if forward Euler FDM is going to be an important technique for us
to approximate the soluton to a PDE, we need to be able predict when the
calculations will yield reasonable results. And a step further, we want to
know if the data we compute approximates the values of the actual solu-
tion. Neumann stability analysis resolves the first equstion. We have more
to say on the second later in the section.

Definition 4.3.1. An FDM realization is Neumann stable provided there

is a positive integer n0 so that the time state vectors satisfy kun+1 k kun k,
for every n n0 .

The idea here is that if the state vectors are non-increasing in norm
then we prevent the wild fluctuations in output that we experienced in the
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 86

86 Elements of Numerical Analysis with Mathematica

prior exercises. In addition, there is a means to verify when the condition

holds. For this purpose, we use the diescrete Fourier interpolation.
We start with a uniform partition of an interval and a function f defined
on the interval. With a simple variable transformation, we may suppose
that the interval is [, ]. In particular, we have values f (xi ) for =
x0 < x1 < ... < xN 1 < . Notice that the functions on this finite set are
really just N -tuples (f (x0 ), f (x1 ), ..., f (xN 1 )). Hence, we may associate
the set of all functions defined on the partition to CN . From results in
linear algebra, we have a linear form on the space,
N
X 1
(f, g) = f (xi )g(xi ). (4.3.1)
i=0

In Chapter 2 we defined a norm on a vector space. However, the function

determines something weaker than a norm on the space of functions. The
following definition modifies Definition 2.2.2.

Definition 4.3.2. Let V be a real or complex vector space. A semi-norm

on V is a function k.k taking values in R such that
(1) for any nonzero vector, kvk 0,
(2) for any scalar , kvk = ||kvk,
(3) for u and v in V , ku + vk kuk + kvk.

Note that is not a norm on the space of fucntions defined on [, ]

as it is possible for f (xi ) = 0 while f 6= 0. The following theorem is a
standard linear algebra result.

Theorem 4.3.1. The linear form defines an Hermition form on Cn .

Further, kf k = ((f, f ))1/2 defines a semi-norm on Cn .

Proof. See [Loustau and Dillon (1993)].

Notice that we do not include xN = N x = = . We cannot include

this value because we do not want to assume or deny f () = f (). In
standard terminology, we do not assume that we have the function space
of periodic functions.
We now define discrete Fourier interpolation. Here and in subsequence
work, we use h = 2/N .

Definition 4.3.3. Let f be a complex valued function defined on the set

{xi : xi = ih, i = 0, 1, ..., N 1}. The discrete Fourier interpolation of f is
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 87

Numerical Differentiation 87

given by
M
X 1 1

f(x) = ck eikx , (4.3.2)

k=M0

where each ck = (f, eikx )/N , and M0 = M1 = N/2 for N even, M0 =

M1 = (N 1)/2 for N odd. The ck are called the (discrete) Fourier
coefficients forf .

The expression in (4.3.2) is a interpolation of f as a trigonometric poly-

nomial. In particular, eikx = cos(kx) + i sin(kx). Hence, by the multiple
angle indentities, cos kx and sin(kx) may be written in terms of the powers
of the cosine and sine. Therefore, the complex exponential is a polynomial
in sine and cosine.
To put this into context, if we were to start with a absolutely inte-
grable function f and form the Fourier transform and subsequently the
reverse transform, then the result would be a function equal to f almost
everywhere. In particular the discrete Fourier interpolation is the inverse
Fourier transform provided we replace the integration by the numerical
integral using the trapezoid form. (See Section 5.1.)
The following theorems state the basic properties of the discrete Fourier
interpolation. We do not prove all, as that would take us to far from our
focus. Indeed, we restrict attention to the properties that are critical for
FDM stability.

Lemma 4.3.1. For any integers j and k,

(eijx , eikx ) = N,

if (j k)/N is an integer and

(eijx , eikx ) = 0,

otherwise.

Proof. In any case, setting h = 2/N ,

N
X 1 N
X 1 N
X 1
(eijx , eikx ) = eijxi eikxi = ei(jk)ih = (ei(jk)h )i .
i=1 i=0 i=0

If (j k)/N is an integer, then since h = 2/N we have

ei(jk)h = ei(jk)2/N = (e2i )(jk)/N = 1(jk)/N = 1,

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 88

88 Elements of Numerical Analysis with Mathematica

and
N
X 1
(ei(jk)h )i = N.
i=0

On the other hand, if (j k)/N is not an integer, then ei(jk)h =

i2 (jk)/N
(e ) = , an N th root of unity. We apply the usual expression for
the partial geometric sum to get
N 1 N 1
X X 1 N
(ei(jk)h )i = i = = 0.
i=0 i=0
1

PM1 1
Theorem 4.3.2. If f (xj ) = k=M 0
dk eikxj for each j = 0, 1, ..., N 1,
then each dk = ck as given in Definition 4.3.2. Conversely, for each j,
PM1 1 ikxj
k=M0 ck e = f (xj ).

Proof. See[Loustau (2016)] or [Thomas (1999)] .

Our next result and its corollary provide the mathematical foundation
for Neumann stability analysis. The result identifies the norm of f as an
N -tuple and the norm of f defined by .

Theorem 4.3.3. (Perceval) For f defined on the given partition, f =

(f (x0 ), ..., f (xN 1 )), then kf k = kfk , where kfk is the semi-norm deter-
mined by , ((f, f))1/2 .

Proof. As a complex N -tuple, the norm squared of f is

N
X 1 N
X 1
kf k2 = |f (xi )|2 = f (xi )f (xi ) = (f, f)
i=0 i=0

by the definition of . The result follows as (f, f) = kfk

Next, suppose that un is the state vector of an FDM process and suppose
that un has discrete Fourier coefficients cni , then Theorem 4.3.2 may be
restated as follows.

Corollary 4.3.1. An FDM realization is Neumann stable provided for each

n and i, |cni | |cn+1
i |.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 89

Numerical Differentiation 89

Proof. From the basic properties of the discrete Fourier interpolation and
Theorem 2.3.2,
M
X1 1 M
X1 1

ku n+1 2
k = un+1 k2
k = ( cn+1
i eiix , cn+1
j eijx ) =
i=M0 j=M0

M
X1 1 M
X1 1

cn+1
i cn+1
j (eiix , eijx ) = N |cn+1
i |2
i,j=M0 i=M0

since (eiix , eijx ) = 0 if i 6= j and (eijx , eijx ) = N . This last statement

follows from Lemma 4.3.1 By hypothesis,
M
X1 1 M
X1 1

N |cn+1
i |2 N |cni |2 = k
un k2 = kun k2 .
i=M0 i=M0

The result follows.

We now return to the 1-D heat equation and the explicit formulation as
derived from (4.2.2).
un+1
i = uni+1 + (1 2)uni + uni1
If we write this equation as the discrete Fourier interpolation we get
M
X1 1

cn+1
k eikxi =
k=M0

M
X 1 1 M
X1 1 M
X1 1

cnk eikxi+1 + (1 2) cnk eikxi + cnk eikxi1 =

k=M0 k=M0 k=M0

M
X 1 1

(cnk eikx + (1 2)cnk + cnk eikx )eikix

k=M0
or
M
X 1 1 M
X1 1

cn+1
k eikix = (cnk eikx + (1 2)cnk + cnk eikx )eikix .
k=M0 k=M0

By equating corresponding Fourier coefficients (Theorem 2.3.1)

cn+1
j = cnj eijx + (1 2)cnj + cnj eijx
Now divide through by cnj and take the absolute value,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 90

90 Elements of Numerical Analysis with Mathematica

|cn+1
j |
n = |eijx + (1 2) + eijx |.
|cj |
Expressing the right hand side in terms of cosine and sine yields,
|cn+1
j |
n = |(cos(jx) + i sin(jx))+
|cj |

(1 2) + (cos(jx) i sin(jx))|,
or
|cn+1
j |
n = |2(cos(jx) + (1 2)| = |1 + 2(cos(jx) 1)|.
|cj |
Now stability will occur when the absolute value of the right hand side
is less than or equal to 1, or 1 1 2(1 cos(jx)) 1. Hence,
2 2(1 cos(jx)) 0. Therefore, Neumann stability is satisfied
provided 1 (1 cos(jx)). But the maximal value of 1 cos() is 2.
Hence, stability requires that 1/2. Appealing to Theorem 4.3.3, we
have proved the folloiwng theorem.

Theorem 4.3.4. The FDM rendering of the 1D heat equation is von Neu-
mann if an only if 0.5.

For the proof of Theorem 4.3.3, we were able to take advantage of some
simple properties of the trigonometry functions. Alternatively, we could
have applied basic calculus max/min procedures to determine values of
that satisfy 1 2(1 cos(jx)).
We see now that stability depends on the balance between x and
t. Generally, if you decrease x then you must decrease t so the it
accompdates for the square of x.
But stability is only one issue. We also must consider the error, en (xi ) =
u(tn , xi ) uni . We can see that this is realted to sdtability via the state
matrix A. For row i of A,
en (xi ) = u(tn , xi ) uni = u(tn , xi ) A(i) uin1 =

u(tn , xi ) A(i) u(tn1 , xi ) + A(i) u(tn1 , xi ) A(i) uin1 =

u(tn , xi ) A(i) u(tn1 , xi ) + A(i) [u(tn1 , xi ) un1

i ]=

u(tn , xi ) A(i) u(tn1 , xi ) + A(i) en1 (xi ) = cn (xi ) + A(i) en1 (xi ),
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 91

Numerical Differentiation 91

where cn (xi ) = u(tn , xi ) A(i) u(tn1 , xi ). If we continue the steps regress-

ing back to the initial stage, we may write in terms of the state vector,
en (x) = cn (x) + Aen1 (x) = ... =
cn (x) + Acn1 (x) + A2 cn2 (x) + ... + An1 c1 (x) + An e0 (x) =
cn (x) + Acn1 (x) + A2 cn2 (x) + ... + An1 c1 (x),
since the initial error is zero. Resolving this equation is more than we can
handle at this state. We can however indicate the general direction. In
this last equation there are two terms, the powers of A and the vectors cn .
It can be proved that if the FDM is stable, then the sequence An 0 in
operator norm. At least this is true for the 1-D heat equation. Further, if
u is C 2 then the sequence cn 0 as x and t 0. To prove this, we
use the Taylor expansions used to derive the finite dofference operators.
Hence, en 0 and we know that the values computed by FDM are
indeed good approximations of the actual values of u. For other equations
the situation is not so nice. These quesstions are for a more advanced
course.

Exercises:
1. Recall the frist order wave equation, given u = u(t, x),
u u
= .
t x
with FDM rendering, forward time and forward space (FTFS).
a. Implement FTFS for the first order wave.
b. Execute Neumann stability anlaysis for > 0
c. Execute Neumann stability analysis for < 0
d. What is the difference between the case for positive and negarive.

2. Forward time, central space provides an antermative FDM rendering

of the order 1 wave equation. Resolve stability for this FDM procedure.

4.4 Finite Difference Method, Implicit and Crank Nicolson

In this section we look at two alternative FDM approaches to the 1D heat

equation. In the first case, we look at backward time, central space, BTCS.
With the notaiton of Section 4.2,
un+1
i uni = un+1 n+1
i+1 2ui + un+1
i1 .
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 92

92 Elements of Numerical Analysis with Mathematica

After solving for uni we have the following expression analogous to (4.2.2).
uni = un+1 n+1
i+1 + (1 + 2)ui un+1
i1 . (4.4.1)
n+1
As in Section 2, we use (4.4.1) to form a linear relation Au = un . In
n+1 n
matrix form, after setting oundary values, this yields Bu =u ,
n+1 n+1
1 0 0 ... 0 0 0 0 0
1 + 2 ... 0 0 0 un+1 un
1 1
0 1 + 2 ... 0 0 0 un+1 un
2 2
... ... ... ... ... ... ... ... ... = ... .

n+1 n
0 ... 1 + 2

0 0 0 uk1 uk1
un+1
n
0 0 0 ... 1 + 2 k
uk
0 0 0 ... 0 0 1 n+1
k+1 n+1
k+1
(4.4.2)
The process is called backward Euler or implicit. As in the explicit case,
the matrix B is nonsinular. Indeed, we have formula of the eigenvalues of
B and none of them are zero or near to zero.
In this case, we know the nth and want to solve for the n + 1st state.
Hence, either we solve (4.4.2) as a linear system or we compute the inverse
of B and resolve un+1 = B 1 un . Hence, un+1 = B (n+1) u0 provided the
boundary values are time independent.
But first, we should consider stability. Analugus to the explicit case, we
resolve this question by recasting (4.4.1) in Fourier form.
M
X 1 1

cnk eikxi =
k=M0

M
X1 1 M
X1 1 M
X1 1

cn+1
k eikxi+1 + (1 + 2) cn+1
k eikxi cn+1
k eikxi1 =
k=M0 k=M0 k=M0

M
X 1 1

(cn+1
k eikx + (1 + 2)cn+1
k cn+1
k eikx )eikix
k=M0
or
M
X1 1

cnk eikix =
k=M0

M
X1 1

(cn+1
k eikx + (1 + 2)cn+1
k cn+1
k eikx )eikix .
k=M0
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 93

Numerical Differentiation 93

By equating corresponding Fourier coefficients (Theorem 2.3.1)

cnj = cn+1
j eijx + (1 2)cn+1
j + cn+1
j eijx
Now divide through by cn+1
j and take the absolute value,
|cnj |
= | eijx + (1 + 2) eijx |.
|cn+1
j |
Expressing the right hand side in terms of cosine and sine yields
|cnj |
= | (cos(jx) + i sin(jx))+
|cn+1
j |

(1 + 2) (cos(jx) i sin(jx))| =

|1 + 2(1 cos(jx))| 1,
since 1 cos(jx) 1 and > 0. Therefore, |un+1 |/|un | < 1 uncondition-
ally. We have proved the following theorem.

Theorem 4.4.1. Implicit FDM applied to the 1D heat equation is uncon-

ditionally stable.

At this stage, it is natural to ask why we bother with explicit FDM

when implicit is so well behaved. There are several reasons why implicit
FDM is not sufficient. First, if the PDE is non-linear, then explicit FDM
is often the approximating procedure of choice. (See [Loustau (2016)]).
Secondly, implicit often under estimates the result so badly that that it is
useless. Often researchers actully execute a few steps with explicit FDM
and then change over once the data is well initalized. The third reason is
the explicit and implicit combine to form Crank Nicolson method. This is
by far the most popular of the FDM time stepping procedures.
To begin Crank Nicolson, we write the time step as the average of the
implicit and explicit formulations,
1  1 n
un+1 uni = un+1 n+1
+ un+1 ui+1 2uni + uni1

i i+1 2ui i1 +
2 2
Next we introduce a ficticious intermediate time state, un+1/2 ,
n+1/2 n /2
(un+1
i ui ) + (ui 1 uni ) =

1  n+1  1 n
ui+1 2un+1 + un+1 ui+1 2uni + uni1 .

i i1 +
2 2
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 94

94 Elements of Numerical Analysis with Mathematica

We separate this equation into a two step process,

n+1/2 1
(un+1 ) = un+1 n+1
+ un+1

i ui i+1 2ui i1
2

n+1/2 1 n
uni ) = ui+1 2uni + uni1 .

(ui
2
The step un un+1/2 is explicit while the step un+1/2 nn+1 is implicit,
n+1/2 1
un+1 n+1
un+1

ui = i+1 + 2(1 + )ui i1 ,
2

1 n
n+1/2
ui+1 2(1 + )uni + uni1 .

ui =
2
The corresponding matrices are

1 0 0 ... 0 0 0
2(1 + ) ... 0 0 0

0 2(1 + ) ... 0 0 0
1

... ... ... ... ... ... ... ... ,

2
... 2(1 + )

0 0 0 0

0 0 0 ... 2(1 + )
0 0 0 ... 0 0 1
and

1 0 0 ... 0 0 0
2(1 + ) ... 0 0 0

0 2(1 + ) ... 0 0 0
1

... ... ... ... ... ... ... ... .

2
... 2(1 + )

0 0 0 0

0 0 0 ... 2(1 + )
0 0 0 ... 0 0 1

The inverse of the first yeilds the n + 1/2 state from the nth . The second
maps the n + 1/2 state to the n + 1st
Taken together they are unconditionally stable.

Theorem 4.4.2. Crank Nicolson FDM applied to the 1D heat equation is

unconditionally stable.

Proof. See Exercie 4.

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 95

Numerical Differentiation 95

Exercises:
1. Use implicit to execute Exercise 1 of Section 4.2.

2. Use Crank Nicolson to exeuct Exercise 1 of Section 4.2

3. Compare the three trreatments for the setting in Exercise 1 of SEc-

tion 4.2. Intrepret the result in light of the statment that Crank Nicolson
is the procedure of choice.

4. Prove Theorem 4.4.2.

a. Use the descrite Fourier interpolation to get an expression for
n+1/2
|cn+1
k | |ck |
,
n+1/2
|ck | |cnk |

b. Multiply the two expression to get and expression for |cn+1

k |/|cnk |.

5. Crank Nicolson for the first order wave equation.

a. Implement FTCS and BTCS for the first order wave equation.
b. Implement Crank Niicolson for the frist order wave equation.
c. Do the Neumann stabiity analysis for this process.
d. The result of Part c was |cn+1
k |/|cnk | = 1. Is such a procedure numer-
ically robust?
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 96
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 97

Chapter 5

Numerical Integration

Introduction

Numerical integration is a core topic of numerical analysis. These tech-

niques yield approximate integrals using only a few function values. Hence,
they can be applied to observational data, cases where we may have little
knoledge of the underlying function. Additionally there are cases where
we know the function, the integral may be intractable or worse, a func-
tion with no closed form antiderivative. For instance B-splines and Bezier
curves are known as parametric cubic, (x(t), y(t)). By solving x(t) for t
and substituting into the second you can represent the curve as a func-
tion graph (x, f (x)). Invariably, functions derived from a parametric cubic
in this fashion do not have closed form antiderivatives. In another direc-
tion, the integral that arises in arc length computations are also often not
resolvable.
We begin the chapter with three numerical integration techniques we
encounter in calculus, the trapezoid, midpoint and Simpsons methods. The
first two are used to develop the idea of the Riemann integral as the area
under a curve. Next, we introduce Simpsons rule which can be expressed as
a linear combination of the other two. We will see that both midpoint and
trapszoid are exact for linear kpolynomials. Simpsons method is exacat
for quadratics.
The quadrature procedures use polynomial interpolation to estimate
the integral. These techniques proceed as follows. Given a function f , we
use a procedure to determine p the polynomial interpolantion. Now, we
integrate p. This step is usually routine. As the polynomial interpolation
will have an error estimate, then the integral of the polynomial error is
the numerical integration error. Gaussian quadrature is the main event

97
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 98

98 Elements of Numerical Analysis with Mathematica

of this chapter. Generally speaking this is the gold standard of numerical

integration. The underlying interpolation is Hermite interpolation. Since
Hermitian interpolation at n+1 locations is exact for polynomials for degree
2n + 1, then Gaussian quadrature is also exact for polynomials of degree
2n + 1.
There is an important technique that uses Gaussian quadrature to re-
solve integrals over a triangle. We will develop this method the last section.
In this section was also includee weighted quadrature. We will focus on
Chebyshev quadrature. Finally, we will develop numerical integration for
functions in paramedtric form.
In the last section we look briefly at other specialized procedures.
Among these is Chebyshev quadrature.
Before
R bproceeding we note that Mathematica has two integration com-
mands, a f [x]dx is equivalent to Integrate[f [x], {x, a, b}]. In this case,
Mathematica will compute the integral if f by identifying the antideriva-
tive g and evaluating g(b) g(a). If you use N Integrate[f [x], {x, a, b}],
then the system executes Gaussian quadrature. Alternatively, if Mathe-
matica fails to find an anti-derivative or the function is one of the functions
known to not have an anti-derivative, then Mathematica will automatically
execute Gaussian quadrature. As noted we develop Gaussian quadrature
in Section 5.3.

5.1 Trapezoid Method and Simpsons Rule

Perhaps the comceptually simplist of the nmercial integration procedres is

the trapezoid method. We begn with a function f defined on an interval
[a, b] that is bounded, real valued and integrable. Furthermore, suppose
we have a partition of [a, b], a = x0 < x1 < ... < xn = b. If we know
f or at least have the values f (xi ), then we can easily approximate the
integral of f over [a, b]. Indeed, if f is positive on [a, b] and we join the
points (xi , f (xi )) with line segments, then the area under the resulting
polygon will approximate the area under f . Since the figure formed by
connecting the four points (xi1 , 0), (xi , 0), (xi , f (xi )) and (xi1 , f (xi1 )),
is a trapezoid, we will get the area under the polygon as the sum of areas
of trapezoids. (See Figure 5.2.1.)
Using the standard formula for the area of a trapezoid (the length of
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 99

Numerical Integration 99

1.0 1.5 2.0 2.5 3.0 3.5

0
Figure 5.1.1: Function, polygon and a single trapezoid

Fig. 5.1 Figure 5.1.1: The temperature disribution after 10 iterations

base times the average of the heights) we have

Z b n
X f (xi ) + f (xi1 )
f dx (xi xi1 ) (5.1.1)
a i=1
2
Notice that if f is negative on [a, b], then (5.1.1) will evaluate negatively
for each trapezoid corresponding to the integral of a negatively valued func-
tion. If f (xi ) = 0, then the corresponding trapezoid degenerates to a right
triangle with area (1/2)f (xi1 )(xi xi1 ). Hence, (5.5.1) continues to hold.
Finally, consider the case where f takes both positive and negative values.
From the previous remarks we need only consider the case where f (xi ) > 0
and f (xi1 ) < 0. In this case, there is no trapezoid, rather the figure
formed by the four points (xi , 0), (xi1 , 0), (xi , f (x) ) and (xi1 , f (xi1 )) is
the union of two disjoint right triangles. One triangle is above the x-axis
and the other is below. In this case, we take x as the point where the
line connecting (x1 , f (xi )) and (xi1 , f (xi1 )) meets the x-axis. We take
f (
x) = 0. Then, we may suppose that x in a point in the partition and
we have returned to the prior case. All together the right hand side of
Equation 5.1.1 provides an approximation for the integral on the left hand
side.

Definition 5.1.1. Suppose f is definited on an interval [a, b] with a parti-

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 100

100 Elements of Numerical Analysis with Mathematica

tion given above, then approximation of the intergral of f given by (5.1.1)

is called the trapezoid method.
The trapezoid method is intuitively obvious, easy to implement and
hence, a reasonable procedure. Later we will learn that it is not as accu-
rate as other less intuitive processes. Further, ther is no immediate error
estimation process. This is a critical flaw for us.
Next, we look at an alternative to the trapezoid method. Notice that
the area of the trapezoid returns the actual value of the integral when the
function is linear. But most functions that we consider have curved graphs,
not polygonal graphics. Simpsons Rule is characterized by the fact that
it returns the actual integral for certain quadratics. In other words, you
would use it when the graph of the function is more or less a parabola.
First we state Simpsons rule and then we will verify the claim.
Definition 5.1.2. For the interval [a h, a + h], the estimated integral for
f in the interval using Simpsons rule is
Z a+h
1 2 1 
f (x)dx 2h f (a + h) + f (a) + f (a h) . (5.1.2)
ah 6 3 6
If we have an interval [a, b] together with a partition, a = x0 < x1 < ... <
xn = b. Then Simpsons rule states that
Z b n
X 1 2 xi + xi1 1 
f (x)dx f (xi ) + f ( ) + f (xi1 ) (xi xi1 ).
a i=1
6 3 2 6
(5.1.3)
In the second statement we merely recast the technique in the same
format as the trapezoid method. Notice that 2h becomes (xi xi1 )
If f (x) = (x a)2 , then
1 2 1 1 1  2
2h f (a + h) + f (a) + f (a h) = 2h h2 + h2 = h3 ,

6 3 6 6 6 3
which is exactly the integral of f . Therefore, Simpsons Rule is exact in
the case of the quadratic polynomial Rcentered at the interval midpoint.
1
Consider the f (x) = x3 , so that 0 f (x)dx = 1/4. Now take x0 = 0,
x1 = 1/4, x2 = 1/2, x3 = 3/4, x4 = 1. Whereas, the trapezoid approxima-
tion is 0.265625, Simpspons rule evaluates it at 0.25 to 5 decimal places.
As mentioned, one of the important applications for numerical integra-
tion is computing arc length. In particular, if f (x) = y for x in the interval
[a, b], then the length of the graph of f is given by
Z b
[1 + f 0 (x)2 ]1/2 dx
a
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 101

Numerical Integration 101

But if the curve is given parametrically, (t) = (1 (t), 2 (t)), t [a, b], then
the arc length is known directly from the parametric formulation as
Z b
0 0
[1 (t)2 + 2 (t)2 ]1/2 dt.
a

Again, the presence of the square root often renders the integral intractable.

Exercises:
1. Let f (x) = xex 1 and set the interval to [1, 4].
a. Use integration by parts to compute the integral of f on the given
interval. Use Mathematica to evaluate the exponentials.
b. Use the Integrate command in Mathematica. Compare the result
with the result in (a).

2. Let f (x) = xex 1 and set the interval to [1, 4]. Consider the
partition 1 < 1.5 < 2 < 2.5 < 3 < 3.5 < 4.
a. Compute the integral of f using the trapezoid method for the given
partition.
b. Compute the integral of f using Simpsons rule for the given parti-
tion. Compare these results with those in 1.

5.2 Midpoint Method

The midpoint is remarkable both because of its simplicity and its accuracy.
As before we begin with a function f defined on an interval [a, b] and a
partition a = x0 < x1 < ... < xn = b. For the sub interval [xi1 , xi ], let
denote the midpoint of the interval and 2h the length. The interval now
becomes [ + h, + h] and the trapezoid approximation for the integral
is 2h(f ( + h) + f ( h))/2 = h(f (] + h) + f ( h)). If we were to
replace [f ( + h) + f ( h)]/2 by f (), then the method would be called
the midpoint rule.

Definition 5.2.1. For the interval [a h, a + h], the estimated integral for
f in the interval using midpoint rule is
Z a+h
f (x)dx 2hf (a) (5.2.1)
ah
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 102

102 Elements of Numerical Analysis with Mathematica

If we have an interval [a, b] together with a partition, a = x0 < x1 < ... <
xn = b, then midpoint rule states that
Z b Xn
f (x)dx f (i )(xi xi1 ), (5.2.2)
a i=1

where i deontes the midpoint of the sub interval [xi xi1 ].

Given a function f defined and intergrable on a clsed interval [a, b], we

write s(f, a, b) for the Simpsons rule integral estimate, m(f, a, b) for the
midpoint and t(f, a, b) for the trapezoid integral estimate. Then s(f, a, b) =
(2/3)t(f, a, b) + (4/3)m(f a, b).
If the function is concave up or concave down in the interval [a, b], then
the midpoint
Rb method is better than the trapezoid method. As expected, we
call | a f dx I(f, a, b)| is called the absolute numerical integration error
for the midpoint method.

Theorem 5.2.1. Suppose the f is concave up or concave down of the inter-

val [a, b], the absolute numerical integration error for the midpoint method
is smaller than the absolute error for the trapezoid method.

Proof. The proof is a simple argument using elementary geometry.

Suppose that the function is concave up in the interval. Consider Figure
3.1a. The error for the midpoint rule is the area between the horizontal
line and the curve. The area to the left of the point A, denoted by , is
positive and the area on the right is negative. Denoting the error to the
right as for > 0, we have the midpoint method error is .
If we include the tangent line to f at the midpoint (see Figure 5.2.1b),
then triangles ABC and ADE are congruent (use side-angle-side). Let be
the common area of the two triangles, then = + and = . Hence,
we may rewrite the midpoint error as = ( + ) = ( + ).
This last term is in fact the error if we had used the tangent line to compute
the approximate integral. Taking absolute values, the absolute midpoint
method error is + .
Figure 5.2.2a shows the trapezoid method error, + , along with the
tangent line at the interval midpoint. Next we construct line segments
from M and N parallel to the tangent. (See Figure 5.2.2b.) We now have
two parallelograms, P1 : EASM and P2 : ACN T . Since the triangles
M SQ and QN T are congruent (use angle-side-angle), then + is equal
to the part of P1 and P2 that lies above the curve. On the other hand the
absolute midpoint method error is equal to the portion of the area of the
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 103

Numerical Integration 103

1.4 1.4

1.2 1.2

1.0 1.0

0.8 0.8

D
0.6 0.6
A A B

0.4 0.4

E
0.2 0.2

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Fig. 5.2 Figure 5.2.1a: Midpoint Fig. 5.3 Figure 5.2.1b. Midpoint
method, method with tangent at the midpoint.

two parallelograms that lies below the curve. Since the curve is concave
up, then necessarily, + > + , and we have proved the theorem.

It is important to keep in mind that the midpoint method is easier to

implement than Trpezoid. And now that we can select the interval so that
is is better. In the next section we will see that midpoint method is also
the simplist case of Gaussian quadrature.

Exercises:
1. Let f (x) = xex 1 and set the interval to [1, 4]. Consider the
partition 1 < 1.5 < 2 < 2.5 < 3 < 3.5 < 4. Compute the integral of f using
the midpoint rule. Compare these results with those of Exercises 1 and 2
of Section 5.1.

2. The following is a modified version of an often used exercise from

Gearld. In this exercise we determine the heat generated by an exothermal
reaction. Suppose two substances are placed in a kiln. The first substance
is active producing an exothermal reaction. The second substance is inert.
Over a 25 minute period the temperature of the kiln goes from 86.2 to 126.5
degrees (Fahrenheit). At each minute the temperature of each substance
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 104

104 Elements of Numerical Analysis with Mathematica

1.4 1.4

1.2 1.2

1.0 1.0 Q

S C

0.8 0.8

0.6 0.6 M
A A

0.4 0.4

E
0.2 0.2

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Fig. 5.4 Figure 5.2.1c: The trapezoid Fig. 5.5 Figure 5.2.1d: with parallelo-
method, grams.

is measured and the temperature difference is recorded. The results are in

the following tables.

0 1 2 3 4 5 6 7 8
0.0 0.34 1.86 4.32 8.07 13.12 16.8 18.95 18.07

9 10 11 12 13 14 15 16
16.69 15.25 13.86 12.58 11.4 10.33 8.95 6.46

17 18 19 20 21 22 23 24 25
4.65 3.37 2.4 1.76 1.26 0.88 0.63 0.42 0.3

a. Plot the 26 points (t, (t)). Fit a cubic B-spline to the data and
display both plots on the same axis. In order to ensure that the curve
extends near to the domain end points, duplicate the first and last points
when generating the B-spline. We denote this curve , and the segments
of as i (s) = (1i (s), 2i (s)).
b. The temperature difference is caused by the reaction. The first t
for which (t) > 0 is the starting time of the reaction, denoted a. To find
the end time of the reaction plot the points (t, log(](t)). You will notice
that after a while the points seem to lie on a line. The value of t for which
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 105

Numerical Integration 105

the plot begins to appear linear is denoted b and is the end time for the
reaction. Rb
c. Estimate the integral a (t)dt using the trapezoid method method.
d. Estimate the same integral using the midpoint rule as follows. Sub-
divide the interval into 10 sub-intervals with length h = (ba)/10. On each
sub-interval estimate the integral via the midpoint rule. Use the points on
the B-spline to estimate values of . For instance, in order to get a value
(t), you will need to

determine the pertinent segment of ,

determine s so that 1i (s) = t,
evaluate 2i (s).
Rb
e. Recall from calculus that | a (t)dt| maxt[a,b] |(t)|(b a). Use
the B-spline to estimate the maximum of .
The value calculated in c is the reaction heat.

5.3 Gaussian Quadrature

The mathematics underlying Gassian quadrature is polynomial interpola-

tion. If f is a function with Rpolynomial
R b interpolation q and error e, then
b Rb
f = q + e and it follow that a f = a q + a e. Provided we know q then
Rb Rb
it is routine to calculate its integral. Hence, a f a q, and we can use
Rb
a
e to estimate the error. The interpolation of choice is a special case of
the Hermite interpolation.
We begin with some terminology. Recall that an inner product [Loustau
and Dillon (1993)] is a real valued function of two variables defined on
a vector space V so that for each u, v and w,

(u, v) = (v, u)
(u + v, w) = (u, w) + (v, w)
(u, u) > 0 for any u 6= 0.
Rb
In particular, (f, g) = a f gdx, defines a positive definite inner product
on the space of continuous functions of an interval (See Exercise 4.) This
is similar to the Hermitian form which arose in the context of Neumann
stability and the discrete Fourier interpolation in Section 4.3. At the time
the form was defined on the complex vector space Cn via certain function
values.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 106

106 Elements of Numerical Analysis with Mathematica

Returning to the current question, the interpolating polynimials are

called the Legendre polynomials. This set of polynomials forms a ba-
sis for the space Pn , the polynomials of degree no lorger than n. In
that sense they are similar to the Lagrange polynomials. Recall that the
Lagrange polynomials satisfy the relation li (xj ) = i,j for interpolation
points xi . The
R b determining characteristic for the Legendre polynomials is
(pi , pj ) = a pi pj = i,j . Hence, the Legendre polynomials are orthogonal
for this inner product. More generally, weRwill refer to any pair of integrable
b
functions on [a,b] as orthogonal provided a f g = 0. For instance, f (x) = x
and g(x) = 1 are orthogonal for the interval [1, 1].
We start with a function f defined on an interval [a, b] with a partition
a x0 < x1 < ... < xn b, and we consider the Hermite interpolatoin of
f.
n
X n
X
h(x) = f (xi )Hi (x) + f 0 (xi )Si (x), (5.3.1)
i=0 i=0

where each Hi and Si is a polynomial of degree 2n + 1 and Hi (xj ) = i,j ,

Si (xj ) = 0 while dHi /dx(xj ) = 0, dSi /dx(xj ) = i,j . In section 3.5 we
proved that the Hermite interpolation of f exists provided f is smooth.
Furthermore, we derived the error estimate. In particular, for the degree
Qn
n + 1 polynomial pn+1 (x) = i=0 (x xi )

f (2n+2) (x )
x) h(
f ( x) = e(
x) = x)2 ,
pn+1 ( (5.3.2)
(2n + 2)!
provided f is 2n + 1 times continuously differentiable.
The next step is to use the Hermite interpolation to estimate the integral
of f .

Definition 5.3.1. Let f be a C 2n+2 [a, b] function. Then the Hermitian

quadrature for the integral of f relative to a partition a x0 < x1 < ... <
xn b is
n
X n
X
i f (xi ) + i f 0 (xi ), (5.3.3)
i=0 i=0

where the coefficients are given by

Z b Z b Z b Z b
0
i = Hi = [1 2hi (xi )(x xi )]hi , i = Si = (x xi )hi ,
a a a a

where hi = p2i , The nonzero coefficients are called the weights.

August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 107

Numerical Integration 107

As a consequence of (5.3.1) and (5.3.2), we estimate the Hermitian

quadrature error.

Theorem 5.3.1. The error E for the Hermite quadrature is bounded by

Z b
M M
|E| p2n+1 (b a)2n+3 , (5.3.4)
(2n + 2)! a (2n + 2)!
where M denotes maxx[a,b] |f (2n+2) |. Furthermore, if f is a polynomial of
degree less than or equal to 2n + 1, then the estimate for the integral of f
given by (5.1.3) is exact.

Proof. It is only necessary to integrate the error estimate for Hermite

interpolation and then notice that |p2n+1 (x)| (b a)2n+2 , and f (2n+1) = 0
if f P2n+1

Next, we want to look at Hermitian quadrature with the additional

assumption that i = 0 for each i. We see that this requirement corresponds
to a particular choice of the xi . For this purpose we use the Lagrange
polynomilas. Recall that for each i = 0, 1, ..., n, the ith Lagrange polynomial
is given by
Q
j6=i (x xj )
li (x) = Q .
j6=i (xi xj )

These polynomials form a basis for Pn . In addition, li (x) =

pn+1 (x)/[p0n+1 (xi )(x xi )].
We calculate,
Z b Z b
0 = i = (x xi )hi = (x xi )li2 = (5.3.5)
a a

Z b  2 Z b
1 pn+1 (x) 1
(x xi ) = pn+1 (x)li (x),
a x xi p0n+1 (xi ) p0n+1 (xi ) a

since li = pn+1 (x)/[p0n+1 (xi )(x xi )]. Hence, the condition i = 0 implies
that pn+1 is orthogonal to the space of all polynomials of degree no larger
than n. We state this formally in the following theorem.

Theorem 5.3.2. Suppose that f C 2n+2 , and pn+1 is orthogonal to Pn ,

the polynomials of degree no larger then n, then the Hermitian quadrature
for f is given by weighted sum of function values,
Z b Z b Z b
X M
f= i f (xi ) + E; i = Hi ; |E| p2n+1 , (5.3.6)
a i=0 a (2n + 2)! a
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 108

108 Elements of Numerical Analysis with Mathematica

Proof. This result is a simple consequence of Theorem 5.3.1 and Equations

(5.3.3) and (5.3.5). Note that (5.3.5) actually states that pn+1 is orthogonal
to a basis for Pn . It is immediate that any vector orthogonal to a basis of
a space is orthogonal to the entire space.

The special case that arose in Theorem 5.3.2 is called Gaussian quadra-
ture.

Definition 5.3.2. The numerical inegration procedure that arises when

p0 , p1 , ..., pn+1 are orthogonal is called Gaussian quadrature.

The next step is to determine when pn+1 is orthogonal to the space Pn of

polynomials of degree no greater than n. As the Lagrange polynomials
R form
a basis for this space, then we need only consider the integrals li p = 0.
Rb Rb
Given 0 = a Si = a (x xi )li and setting f = lj in (5.3.6), we compute
Z b Xn Z b Z b Z b
lj = lj (xi ) Hi = Hj = uj hj =
a i=0 a a a

Z b Z b Z b Z b
d d
[1 hj (xj )(x xj )]lj2 = lj2 hj (xj ) Sj = lj2 .
a dx a dx a a
We have now proved the following corollary.
Qn
Corollary 5.3.1. The polynomial p(x) = i=0 (x xi ) is orthogonal to Pn
provided the Lagrange polynomials satisfy
Z b Z b
li = li2 . (5.3.7)
a a
Rb
In this case the weights are given by l
a i
>0

We see that the Gaussian quadrature is the special case of Hermite

quadrature associated to (5.3.6). and the polynomials pn+1 are the Legen-
dre polynomials. Gaussian quadrature with n points has the property that
it is exact for polynomials of degree 2n + 1. There is no numerical inte-
gration technique that is better. It all depends on the choice of the points
x0 , x1 , ..., the partition points. These points are commonly called Gaussian
quadrature points. In turn, to find the Gaussian quadrature points for a
given n we must identify the Legendre polynomial pn as these points are
the roots.
To identify the Gaussian quadrture points, we take p0 = 1 on [a, b] and
then use the Gram-Schmidt process to solve for each pn in succession. For
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 109

Numerical Integration 109

instance, for the interval [1, 1], p1 (x) = x and p2 (x) = x2 x + 1/6.
The corresponding Gaussian quadrature points for this interval are 0 for
n + 1 = 1 and 0.5 0.5/ 3 for n + 1 = 2.
Additionally the points and weights for the standard interval [1, 1]
are listed online or in pre computer era numerical analysis texts
[Hildebrand (1974)]. On the other hand, the Mathematica function
GaussianQuadrature[n, a, b] returns the Gaussian quadratgure points and
weights for the nth order quadrature on the interval [a, b]. In the literature
it is possible to find approximations for the Gaussian point for n in the tens
of thousands.

Exercises:
1. Given a set of Gauss points for an interval [a, b] determine the trans-
formation that will map them to the corresponding Gaussian quadrature
points another interval [, ].

2. Consider two point Gaussian quadrature on the interval [0, 4] and

the function fn = xn + 2x 3. For n = 4, 5, 6 R4
a. compute the Gaussian quadrature estimete for 0 f dx,
b. compute the actual error and the estimated error.

3. Let f (x) = xex 1 and set the interval to [1, 4]. Compute the inte-
gral of f using the two point Gaussan quadrature. Compare these results
with those of Exercieses 1 and 2 of Section 5.1 and Exercise 1 of Section
5.2.

4. Consider the vector space C 0 [a, b] of continuous real

R b valued functions
on the interval [a, b]. Prove that the function (f, g) = a f gdx is a positive
definite inner product on the vector space C 0 [a, b].

5. Suppose the interval [1, 1], p0 (x) = 1 and use the Gram-Schmidt
process to derive p1 and p2 .

6. Prove that one point Gaussian quadrature is identical to the mid-

point rule.

7. Prove that there are Gaussian quadrature points for every n. This
is not the case for some of the weighted qudratures introduced in the next
section.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 110

110 Elements of Numerical Analysis with Mathematica

5.4 Comments on Numerical Integration

In this section we resolve three unrelated numerical integration procedures.

We begin with weighted quadratures. This is a generalization of Gaussian
quadrature which addresses certain functions which do not conform well
to the general techniques. In particular, the functions with graphs that
do not appear to be the graph of a polynomial. In the seond, we look at
functions that are represented in parametdric form. We address the prob-
lem of appying numerical integration to these functions. The problem here
is that we must rewrite a parametric representation, (t) = (1 (t), 2 (t))
into the format (x, f (x)). Lastly we look at applying Gaussian quadrature
to rectangles and triangles. For the rectangle case, we can use Cartesian
products. For triangles, there is a transformation beteween rectangles and
triangles that enables change of variables.
There are many different realizations of quadrature. Gaussian quadra-
ture is only the simplist form. Generally speaking, Gaussian quadrature
works well for functions that are polynomial like. The idea is that Gaus-
sian quadrature is exact for polynomials up to a degree associated to n,
the number of points. More gernally, it is well-behaved for functions with
bounded derivatives {f (m) (x), m = 1, 2, ..}. Hence, the error term goes
to zero as n increases. Specifically, the more points used in the numerical
process, the better the approximation. Conversely, it is not difficult to de-
termine a function where the error bound does not converge to zero. In
another direction, f (x) = (x2 + 1)1 is assymptotic to the x-axis.; whereas,
polynomials are not. Hence, we say that this function is not polynomial-
like. There are other quadratures designed to approximate the integral for
a function that has properties not shared by polynoimials.
We begin this section with Chebyshev quadrature. In this case we are
concerned with a function f defined on an interval [1, 1] so that f 0 (x) is
not bounded as x 1 or x 1 or both. Stated otherwise, the function
has vertical tangent
at one or the other end point. It is reasonable to look
at g(x) = f (x) 1 x2 . We refer to g as the weighted form of f . With
the notation of the last section, we follow the direction presented in (5.3.3)
and (5.3.6) modified now for the weighted case. In particular, we want to
know when the i = 0 and in this case what can we say about E and i .
To begin, we have the following result.

Lemma 5.4.1. With the current notation, i = 0 if and only if pn+1 sat-
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 111

Numerical Integration 111

isfies
Z 1
p q
0= n+1 , (5.4.1)
1 1 x2
for any q Pn .

Proof. The proof is entirely analogous to the corresponding result for

Gaussian quadrature, Theorem 5.3.2.

Next, we resolve (5.4.1) with a variable substitution x cos(). It is

then routine to verify the foloowing statements.

Theorem 5.4.1. The Chebyshev quadrature satisfies

pn+1 (x) = Cn+1 cos((n + 1) cos1 (x)); E = f (2n+2) (),
22n+2 (2n + 2)!
(5.4.2)
where || < 1 and Cn+1 is a constant. The weights satisfy i = /(n + 1).

Proof. With the comment before the theorem statement, the proof is
routne.

If we compute the points xi we find that they are clusterd at the outer
edges of the interval. For instance, see Exercise 3.
Next, we turn to Laguerre quadrature. In this case we consider the half
line [0, ) and the weight ex

Theorem 5.4.2. The Laguerre quadrature satisfies

ex dn x n ((n + 1)!)2 (2n+2)
pn (x) = (e x ); E = f (), (5.4.3)
n! dxn (2n + 2)!
where 0 < < . The weights satisfy i = (i!)2 . The pn are called the
Laguerre polynomials.

Proof. See Exercise 5

Another important case is the Jacobi quadrature. This is a gener-

alization of the Chebyshev case. In this case the weighting function is
w(x) = (1 x) (1 + x) , , > 1
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 112

112 Elements of Numerical Analysis with Mathematica

In previous sections, we have developed procedures to approximate an un-

known function by parametric cubics. For these cases, the implementation
of the trapezoid method or any other of the numerical integration tech-
niques is considerably more difficult.
Consider the following setting. Suppose we have a function f (x) that
is known only at a few domain values. Suppose further that we use these
values to fit a B-spline segments k (t) = (1k (t), 2k (t)), t [0, 1] associated
to points on the graph of f . Notationally we designate the points (xi , f (xi )),
i = 0, 1, ..., n. Next, given points in a = x0 ... xn = b in the domain of
f , we want to find yi on the B-spline so that
Z b n
X yi yi1
f dx (xi xi1 )
a i=1
2

approximates the integral of f or equivalently the area under the B-spline?

We can rephase this quation as follows. Given a point (x, y) on the
B-spline, how can we find k and t so that (x, y) = (1k (t), 2k (t))? If we can
resolve this, then we casn write (xi , yi ) as (1ki (ti ), 2ki (ti )) and appriximate
the integral with
n
X ki (ti ) ki (ti1 )
2 1
(1ki (ti ) 1ki (ti1 )) (5.4.4)
i=1
2

Hence, we must solve for k, the segment identifier, and t, the specific pa-
rameter value.
To determine k, we must find k such that k (0) x k (1).
To locate t, we use Newtons method via the FindRoot function.

FindRoot[gamma^k[t] == x_i, {t, tau_i}];

where i is the estimate for ti required by the method. We claim that

i = ti1 is a reasonable estimate provided xi and xi1 lie in the same
B-spline segment. Alternatively, we take i = 0. We base these choices on
the tacit assumption that the differences xi xi1 are small.
Keep in mind that Newtons method must always be suspect, as it may
produce a ti , which is not between 0 and 1. Therefore, it is best to test
the value returned for reasonableness. In particular, we expect ti to be in
[0, 1], and we expect that t increases as x increases. The following pseudo
code implments this.

t_i = FindRoot[ x[t] == x_i, {t, \tau_i]}];

If [t_{i-1} < t_i || t_i > t_n,
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 113

Numerical Integration 113

exit and print an error statement,

otherwise continue with processing
];

We consider a third question. Suppose we want to integrate a function

over a triangle. This is an important equation for finite element method. In
that context, we may need to compute thousands of integrals over triangles
for a single FEM problem. We will need to do that efficiently and acurately.
The problem is that integrals over triangular regions are not easy.
The Gaussian quadrature developed in the prior section applies only
to intervals. But it is routine to extend to irectangles via Cartesian prod-
ucts. For instance two point quadrature on an interval becomes 4 point
quadrature on a rectangle. Given Gaussian quadrature points {x1 , x2 } on
the x-axis and {y1 , y2 } on the y-axis, we merely take the Cartesian prod-
uct {x1 , x2 } {y1 , y2 } to get Gaussian quadrature points for the rectangle,
[a, b][c, d]. The weights for the interval are 1/2 at both locations. The cor-
responding weights for the rectangle are 1/4. The idea is to apply Fubinis
theorm,
Z bZ d Z bZ d
f (x, y)dydx = [ f (x, y)dy]dx
a c a c

2 Z b 2 X
2
X 1 X 1
f (x, yj )dx f (xi , yj ). (5.4.5)
j=1
2 a i=1 j=1
4

Now, suppose you want to integrate over a triangular region E with

vertices (, ), (, ), (, ). The rational function in two variables
 
(1 + x)(1 y) 1 + y
(x, y) = (, ) = ,
4 2
maps the rectangle [1, 1] [1, 1] to the triangle (0, 0), (1, 0), (0, 1). Fur-
thermore, it is bijective on the interior of the rectangle. It is not bijective
on the boudnary, but that is not important for our purpose. Continuing,
the affine mapping A(x, y) = T (x, y) + (, ), where T is the linear trans-
formation given by the matrix
 

,

will map the triangle (0, 0), (1, 0), (0, 1) bijectively to the triangle E. Hence,
by combining A and , we have a bijection from the standard rectangle to
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 114

114 Elements of Numerical Analysis with Mathematica

any triangle. Therefore, employing the usual variable change procedure

[Marsden and Tromba (2003)], we are able rewrite
Z Z 1Z 1
f (, )dA = f A (x, y)J(A )dxdy.
E 1 1

For the final step, ow, approximate the rectanglular integral via the quadra-
ture technique (5.4.5).

Exercises:
1. Prove Lemma 5.4.1.

2. Complete the verification of the statemets for Theorem 5.4.1.

3. Compute and plot the points 21 points xi (n = 20).

4. The 5th Laguerre polynomial is 120600x+600x2 200x3 +25x4 x5 .

Approximate the roots of L5 . Descrive how they are disctributed.

5. Prove Theorem 5.4.2.

a. Derive the expression for the Laguerre polynomials from
Z
0= ex pn+1 qdx,
0

for q Pn .
b. derive the expression given E
c. Find Hn and then derive the expression for i .

6. Let f (x) = xex 1 and consider the guide points (1, f (1)),
(1.5, f (1.5)), (2, f (2)), (2.5, f (2.5)), (3, f (3)), (3.5, f (3.5)) and (4, f (4)).
a. Use a B-spline to fit this set of guide points. Plot the resulting curve.
b. Beginning with the B-spline, use the trapezoid method to approx-
imate the integral of f . Compare this result with the output of 1 and 3
above.

7. Compute the arc length of the B-spline derived in Problem 6(a)

above. In this case the curve is the join of four B-spline segments, i ,
i = 1, 2, 3, 4. Compute the length of each B-spline segment individually
using a uniform partition with t = 0.1
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 115

Numerical Integration 115

8. Use the quadrature technique of this section to estimate

Z
x4 + 2xy + y 4 dxdy
E

where E is the triangle with vertices (-1, 2), (5, -1), (0, 4).

9. Gicve an example of a funcion for which f m ()/m! does not converge

to zero as m gets large.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 116
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 117

Chapter 6

Numerical Ordinary Differential

Equations

Introduction

In this chapter we consider first order ordinary differential equations, ODE,

boundary value problems. In particular, a first order ODE is an expression
du
= u0 = f (x, u(x)) (6.0.1)
dx
with a boundary value,

u0 = u(x0 ) (6.0.2)

The very basic idea is that we know the initial location, x0 , and the initlal
function value u(x0 ), Hence, we know f (x0 , u(x0 )) = u0 (x0 ). Now with this
information and Taylorss theorem we can begin to approximate u.
There is an existence, uniqueness theorem for first order ODE. This
theorem requires that the Lipshitz condition. The result is due to Picard.

Theorem 6.0.1. Consider the first order ODE, u0 = f (x, u) with given
boundary value u0 = u(x0 ) and where f is a continuous function of its
domain D and (x0 , u0 ) is interior to D. Suppose that f satisfies the Lipshitz
condition on D. In particular, suppose that there is a non-negative real K
so that

|f (x, y2 ) f (x, y1 )| < K|y2 y1 |,

for any pair of points (x, yi ) in D. Then there is an interval I about x0 so

that the ODE has unique solution on I.

Proof. See [Simmons and Robertson (1991)] for a proof of the existence
step.

117
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 118

118 Elements of Numerical Analysis with Mathematica

A word of caution is necessary. The techniques developed in this chapter

apply to any ODE. It is not difficult to write down an equation for which
there is no solution or alternately an equation without unique solution.
In either case, the results of the numerical method may not represent the
setting which has been modeled. it is important to stay within the context
of the theorem.
The shooting method introduced in Section 3 is designed to handle
second order ODE. We will see in that context that second or higher order
equations can be reduced to a system of first order equations. For instance,
consider a second order equation with initial values,
u00 = g(x, u0 , u); u0 = u(x0 ), v0 = u0 (x0 ).
It is easy to see that this equation may be recast as a pair of first order
equations. In particular, we set v = u0 and write the pair of equations
v 0 = g(x, v, u), v0 = v(x0 ); u0 = v, u0 = u(x0 ).
We will see that the techniques we develop for order 1 equations are rou-
tinely extended to sets of order 1 equations and thereby apply to order
two equations. Additionally, we will see that order two equations with two
values given for u may be solved as if we had a value for u and a value for
u0 . Altogether, by focusing on first order equations we are in fact including
a broad range of cases.
We begin the chapter with three initial value ODE techniques, forward
Euler, forward Euler as predictor/corrector and midpoint method. The
predictor/corrector uses the stated technique is a predictor. The corrector
is a related process that modifies the value returned by the predictor. If
the two values are sufficiently close, then we proceed to the next step using
the current predictor value. Otherwise, we have the option to repeat the
corrector and compare the two corrected values. When successive values of
the corrector are sufficient close, then we proceed with the current corrector
value. If we continued with forward Euler and midpoint, we would come to
Runge-Kutta method. This method is certainly important. However, the
mathematical foundation is too detailed for this treatment. See [Loustau
(2016)] for Runge-Kutta or R-K method.
following the initial vlaue methods, we develop the shooting method for
second order ODE. This is an example of a boundary value method. We
end the chapter with a treatment of the method lines. This is a hybrid
technique that is sometimes referred to as ODE time stepping. In this
setting, you develop the spatial variable with FDM or some other discrete
procedure, then handle the time stepping as an ODE problem.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 119

Numerical Ordinary Differential Equations 119

A second matter of note is that the case studies of this chapter are
mostly population studies. We will see a simple birth/death model and a
prey/predator model. In the latter case we look at competing cell types
in an organism as a preditor/prey model. This sort of model arises in
mathematical oncology.

6.1 First order ODE Techniques, Forward Euler and Cor-

rector Method

We begin with a first order ODE, u0 = f (x, u) and a boundary value u(x0 ) =
u0 . Given x, we seek a means to approximate u1 = u(x1 ) where x1 =
x0 + x. Our first technique is called forward Euler. In this case we set
u1 = u0 + u0 (x0 )x = u0 + f (x0 , u0 )x. More gernearlly, we have an
interating process,
un+1 = un + f (xn , un )x, (6.1.1)
where xn = x0 + nx. Note that we have encountered the terminology
(forward Euler) in another context. This intance is related but distinct.
Consider an example. We begin with an ODE that we know describes
an exponental function. In particular, we take u0 = f (x, u(x)) = u(x)
with x0 = 0, and u0 = u(0) = 1. In addition, we choose x = 0.1. Hence,
for = 2,
u1 = u0 + f (x0 , u(x0 ))x = u0 + u0 x = 1 + 2 1 0.1 = 1.2
The actual solution to the equation is u(x) = ex . Therefore, u(x1 ) =
e20.1 1.221 and the error is about 0.021.
In turn,
u2 = u1 + f (x1 , u(x1 ))x = u1 + u1 x = 1.2 + 0.24 = 1.44,
whereas u(x2 ) = e20.2 1.492. Now the error is about 0,052, more than
twice the error after the first step. Continuing in this manner, each suc-
cessive approximation is worse. In fact, it will grow without bound. (See
Exercise 1).
There is an alternative approach. The problem is that we are using
f (x0 , u0 ), the slope at x0 to approximate the slop at x1 . However, we can
take advantage of the approximate value u1 and compute a second estimate
of the slope for u at x1 , 1/2(f (x0 , u0 ) + f (x1 , u1 )). Continuing, we get a
second estimate for u at x1 . This value is called the correction for u1 and
denoted it as u11 .
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 120

120 Elements of Numerical Analysis with Mathematica

For instance, suppose we continue with the current example. In this

case u1 = 0.2 and ((f (x0 , u0 ) + f (x1 , u1 ))/2 = (2 + 2.4)/2 = 2.2. Whereas,
the actual derivative at x = 0.1 is about 2.44. Therefore, the corrected
estimate of u0 (0, 1) is much better. Prpoceeding with this value
u11 = u0 + 2.2 x = 1 + 0.22 = 1.22
This is correct to two decimal places.
Note that if |u1 u11 |  0, then we will want to apply the corrector a
second time to calculate u12 . Again we copmpute |u11 u12 | amd decide if
a third corrector is needed. In the end we have u1n to approcximate u at
x1 . Next, which value should we use to compute x2 ? Since we report u1n
as the estimated value of u at x1 , then we should use it to compute u2 .
Genarally speaking, The forward Euler is not considered to be a good
approximation procedure. The forward Euler with corrector is better but
not efficient. And the more often you execute the corrector, the less efficient
it becomes. For instance, if you execute the corrector logic multiple times
at each step, then the execution time increases by a fixed quantity while the
data improvement decreases with each iteration. Hence, multiple executions
of the corrector in not often advised.
We end the section iwth an example. Consider the following popula-
tion problem. Let u = N/NM denote the population desnity of an isolated
group, as a function of time t. Here we suppose that Nm is the maximal
supported population so that the ratio is the density, a continuous variable.
The net change in N relative to t should be the number births minus the
number of deaths. The births may be estimated at the birth rate times the
population size. Whereas, the death rate will be dependent on environmen-
tal factors (for instance, temperature, available of food, infectious agents
in the environment). Suppose that there is an experiment where migration
is prevented, the birth rate is known to be 0.09 and the death rate as a
function of time and population size, is given by B(t)N 1.7 . In particular
N 0 (t) = 0.09 N (t) B(t)N (t)1.7 = f (t, N (t)) (6.1.2)
. In addition, suppose that B(t) is known for 10 equally spaced increments
of t. These values are given in the following table.

t 0 1 2 3 4
B 0.007 0.0036 0.0011 0.0001 0.0004

5 6 7 8 9 10
0.0013 0.0028 0.0043 0.00056 0.00044 0.0004
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 121

Numerical Ordinary Differential Equations 121

This type of population model is called a birth/death model. It is the

simplest of the population models.
Finally, we are given N (0) = 100 We must estimate the population
density at each of the given times. In addidion, we set the time interval
[0, 1] and t = 0.1. Hence, the values in the first row of the table now refer
to integer multiples of t.
Next, looking at the equation (6.1.2), we observe that population growth
is exponential while decline is polynomial. Even though, the rate of growth
is small, the values for B are also small. Hence, we should expect to see
overall growth during the duration of the experiment. We first estimate
the population over the time period using the basic Euler method. Since
we expect integer values of N , we round the computed values at each step.
We compute
f (0, 100) 8.49, N (0.1) = 99; f (0.1, 99) 0.02, N (0.2) = 99,

f (0.2, 99) 6.19, N (0.3) = 100; f (0.3, 99) 8.75, N (0.4) = 101,
We continue through 11 time values and report out output in Figure 6.1.1
has been recorded at each time step. Next we employ the corrector. The
first few calculations are summarized here. Now we use g for the corrected
derivative estimate.
g(0, 100) 8, 58, N (0.1) = 99; g(0.1, 99) 0.02, N (0.2) = 99,

g(0.2, 99) 6.21, N (0.3) = 100; g(0.3, 99) 8.79, N (0.4) = 101,
The estimated derivatives are slightly changed, but the population figures
showned in Figure 6.12 are rounded to the nearest integer and unchanged.

Exercises:
1. Continue the example developed above, u0 = u(x) with x0 = 0,
u0 = 1 and x = 0.1. Given that u(x) = e2x Use forward Euler to com-
pute the estimates u3 , ..., u1 0. for each case compute the corresponding
error.

2. Apply forward Euler to the equation given by (6.1.2).

3. Repeat 2 for Euler with a single corrector.

4. Repeat 2 for Euler with 3 correctors. How does this data change
with each iteration?
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 122

122 Elements of Numerical Analysis with Mathematica

105

105
104

104
103

103
102

102
101

101
100

100
99
2 4 6 8 10
99
2 4 6 8 10
Figure 6.1.1. Output for forward Euler
Figure 6.1.2. Output for forward Euler with corrector

Fig. 6.1 Figure 6.1.1: Fordward Eulsr Fig. 6.2 Figure 6.1.2. Midpoint
method, method with tangent at the midpoint.

5. Use least equares to fit a cubic polynomial to the given values for N .
(See Figure 6.1.3.) Now repeat Exercise 2 for using the cubic to represent
N . Note that the cubic allows is to infer values of N between the given
ones. This will be usful in the next section.

6.2 Midpoint Method with an Application to Mathematical

Oncology

In Section 6.1 we considered the following equation, u0 = u and a boundary

value u(0) = 1. This equation has actual solution u = ex . In addition,
we took x = 0.1. At that time, we used the forward Euler method and
computed apprxmiate values for u. In addition, we considered a corrector
method and oberved that the corrected values were much better than the
initial approximations. Alternatively, we say another example where the
results were the same whether or not the corrector was used.
In this section, we develop the midpoint method. This is the thrid
ODE technique, s a general rule, midpoint is more efficient than the pre-
dictor/corrector while producing higher quality results. The basic idea
is, given an ODE, u0 (x) = f (x, u(x)) and an iterated solution procedure
un+1 = un + h(x, u(x))x, how should you define h relative to f . For
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 123

Numerical Ordinary Differential Equations 123

forward Euler, f = h. For the corrector, h(xn , un ) = 1/2(f (xn , un ) +

f (xn+1 , un+1 )). For midpoint method there is yer another choice for h.
Recall Equation (6.1.1) were we use f (xn , un ) to approvimate the deriva-
tive of u at xn+1 . For the midpoint method, we compute un+1 from
xn + x/2 and un + f (xn , un )x/2. In particular, the midpoint method is
given by
x x

un+1 = un + f xn + , un + f (xn , un ) x. (6.2.1)
2 2
Note the similarity to Crank Nicolson FDM. In that case, you create a
fictitious time step tn+1/2 . Now we create a fictitious value of the dependent
variable, un+1/2 . But now, we do this using the Taylor expansion for u at
xn+1/2 .
In Section 6.1, we looked at the ODE u0 = 2u with u(0) = 1 and x =
0.1. The solution to the equation u(x) = ex has values u(0.1) = 1.2214
and u(0.2) = 1.49182. Using (2.6.1)

u1 = u0 + f (x0 + x/2, u0 + u0 x/2)x =

1 + (1 + 0.05) 0.1 = 1.22,

and

u2 = u1 + f (x1 + x/2, u1 + u1 x/2)x =

1.22 + 2(1.22 + 2 1.22 0.05) 0.1 = 1.4884

We see that midpoint values are very good.

Runge Kutta method is significantly better. This is the next stage
in ODE simulation technique. Not only is it more correct, Runge Kutta
includes an error estimation procedure. [Loustau (2016)]
Sceintists use birth/death processes to model tumor growth. In this
case, cell vision is governed by the mitosis rate. This is exponential. But
tumors rarely grow exponentially. Rather, only the cells on the tumor
surface have access to sufficient nutrients via the blood supply to divide.
Hence, the growth rate is a function of the tumor surface area or in the
case of a more or less sphereical tumor, the tumor radius. Furthermore,
there is cell death. Hence, we should expect a birth/death model. As
with population of dividing cells, dying cells can be identified geometrically.
Indeed, cell death should be more prevelent with older cells, those lying
closer to the tumor center. Hence, both processes are functions of a portion
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 124

124 Elements of Numerical Analysis with Mathematica

of cell population. Researchers often consider the following model. [Wodarz

and Komarova (2014)]
dU
= aU bU , (6.2.2)
dx
where the parameters a, b and , are to be determined through experi-
mental observation.
These parameters identify subpopulations and rates of change. In par-
ticular, U is the subpopulation of dividing cells and a is the mitosis rate,
U is the aging subpopulation and b is the death rate. There are parameter
values for which U is known. For instance, Ut = aU bU 2 has solution
U = KU0 eat /(K + U0 (eat 1)) where K = a/b is called the carrying ca-
pacity. Using lHopital, limt U = K. Hence, U is increasing taking
values on the interval [U0 , K) for t [0, ). The resulting curve is called
sigmomial. We have plotted U in Figure 6.2.1 for the case K = 2.
For the ODE based on (6.2.2), f is only a function of U and not directly
a function of x. Hence, (6.2.1) becomes
x

Un+1 = Un + f Un + f (Un ) x =
2
h x
x
i
Un + a Un + f (Un ) b Un + f (Un ) x.
2 2
In Exercise 4, the reader is asked to apply the midpoint method to this
ODE.
In passing we remark that this model identifes a particular problem
that arises in cancer therapy. Many drug therapies affect only the currently
dividing subpopulation. Whereas, there is always a subpopulation behind
this one. Once the surface cells have died, this resevoir of cells has acess to
the nutreint rich blood supply and begins the reproduce.

Exercises:
1. Continue the basic example from Section 6.1 by comparing the mid-
point estimates for u3 , ..., u10 against the actual data and the forward Euler
exgtmates.

2. Apply the midpoint method to Ut = 2U U 2 , U0 = 100 and

K = 2. Compare your estimated results to the acutucal values for
U = 200e2t /(2 + 100(e2t 1).
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 125

Numerical Ordinary Differential Equations 125

0.5

0.4

0.3

0.2

0.1

-4 -2 2 4

Fig. 6.3 Figure 6.2.1: The sigmoidal curve, U = 0.1e2t /(0.5 + 0.2(e2t 1))

3. In (6.2.2) take = 0.2 and = 0.5 with K = 2 and U0 = 0. Use

the midpoint method to estimamte values of U . Plot the result. Is this a
sigmoidal curve?

4. If the tumor is assumed sphereical, then the researchers sometimes

use the following verions of 6.2.2.
dU
= aU 2/3 bU,
dx
Estimate values for U if K = 0.5 and U0 = 2. Plot the points and use a
B-spline to resolve the computed values as a curve.

6.3 Shooting Method with an Application to Cooling Fins

Many of us have had the experience of being on the atheletic field and
receiving a thrown, kicked or batted ball. In order to receive the ball we
must estimate the initial lift or angle. From experience, we know that the
trajectory is a parabolic arc, we can see the initial location. When we know
the inital angle we can resolve the trajectory. We also know that it takes
some experice to do this reliably and quickly. We see below that we were
executing the shooting method.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 126

126 Elements of Numerical Analysis with Mathematica

In this section we add two levels of complexity to the cases considered

up to this point. First the equation under consideration is second order.
The standard approach to a second order ODE is to reduce it to two first
order equations that are then solved simultaneously. For instance given
u00 = f , then we set v = u0 and write the two equations, v 0 = f and u0 = v.
To solve such a system, we would expect to be given two initial values, u(0)
and v(0). We call this setting an initial value problem. Another possibility
is to be given values of u at the endpoints of the domain. This is called
a boundary value problem. In this section we present the shooting method
for solving boundary value problems.
Cooling fins are a biological adaptation that is copied in mechanical
engineering designs. We are all used to the use of cooling fins on the cylinder
head of internal combustion engines, for instance the cooling fins used on
lawn mower or motorcycle engines. The idea is that the relatively large
surface area compared to volume accelerates heat transfer from the object
through the fin and then to the surrounding medium. Termite mounds in
North Australia prokvide an example from nature. If you are not familiar
with these structures, them please do a web search. In particualr, the
insect mound uses cooling fins to dissapate the heat from the colony to the
surrounding environment.
We consider the following example based in a one dimensional model.
Let u = u(x) denote the temperature measured in absolute degrees or
Kelvin at location x. Then u satisfies the following second order non-linear
ODE.

u00 (x) = k(u4 T 4 ), (6.3.1)

where T is the temperature of the surrounding medium, k is a heat flux

constant and x [0, L]. The first task toward solving (6.3.1) is to rewrite
it as a system of two first order equations,

u0 (x) = v(x), v 0 (x) = k(u4 T 4 ). (6.3.2)

This now can be expressed as u0 (x) = f (x, v, u), v 0 (x) = g(x, v, u), where

f (x, v, u) = v(x), g(x, v, u) = k(u4 T 4 ).

Before continuing with the solution process, we consider the data. First
if T were in normal range, say 27 C, then T = 300 Kelvin. If for instance,
u(0) = 35C = 308K, then v 0 (0) = 899, 176, 496. Now, think about what
will happen if we were to develop this pair of equations as an initial value
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 127

Numerical Ordinary Differential Equations 127

problem using a simple procedure such as Euler forward. In this case, we

would have a value for v(0) = and write
v(0 + x) = v(0) + x v 0 (0) = + 899, 176, 496(x),

u(0 + x) = u(0) + x u0 (0) = u(0) + v(0)x = 308 + x.

Now a serious problem occurs at the next iteration. Here we have
u(0 + 2x) = 308 + x() + x(v(0 + x)) =

308 + x + x( + 899, 176, 496(x)).

In order to keep the values of u within reasonable limits, either is
a very large negative number, or x is very close to zero. The first case
is consistent with the model as we expect u to be decreasing at along
the interval 0, L. But a huge drop off in temperature is not intruitive.
Alternatively, we must take x very close to zero. In terms of the given
data, it must be a small multiple of 105 . Now if L = 0.5, then we will
need 50,000 iterations. For the purpose of this introductory presentation,
this is far to many iterations. Hence, the data used below is intentionally
not in normal range. When looking at the given data, keep in mind that
even liquid nitrogen is 77 K.
We set T = 3.0, k = 0.23, L = 0.5 and u(0) = 4.5. As this is to be
a boundary value problem, we take u(L) = 4.0. To solve for approximate
values of u, the most common technique is to estimate v(0) with and then
use one of the standard techniques to derive a corresponding estimate u for
u(L), If the estimate is not equal to the given value, then we re-estimate
with and recalculate to get u
. The immediate goal is to get two estimates
which bracket the desired value. With these two estimates for v(0) we are
now able to derive a third estimate using linear interpolation,


+ (4.0 u
= ) . (6.3.3)
u
u
If the result is still not sufficiently close to the desired value we repeat
the process using two of prior estimates that bracket the desired value.
This technique is called the shooting method for solving a boundary value
problem. Historically, it was developed to aim canon in a military context.
The basic idea is that given the location of the cannon and the target, then
the problem of aiming the cannon reduces to determining the cannon angle,
u0 . After two reasoned misses, you are able to determine the correct canon
angle, u0 as the interpolate of the angles from the two prior attempts.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 128

128 Elements of Numerical Analysis with Mathematica

We need to select a technique. For simplicity we choose to employ the

midpoint method for u and forward Euler for v. For this case we have

vn+1 = vn + g(un )x = vn + k(u4n T 4 )x,

x x
un+1 = un + [f (un ) + f (un )]x = un + [vn + vn ]x,
2 2
where vn is the computed approximation of u0 (xn ).
Setting x = 0.01 and testing the result for negative integer values
of we see quickly that estimates of 0.9 and 1.0 bracket the desired
result. The corresponding values for u0 (0) = v(0) are 4.0476 and 3.9975.
Employing the shooting method (6.3.3) we next test at 0.995. This yields
the a result that is correct to 4 decimal places, 4.00001. Figure 6.3.1 and
6.3.2 show values for u and u0 for u0 (0) = 0.995.

Exercises:
1. Execute the example from the text.

2. Repeat the example from the section using forward Euler to extimate
u and v.

3. Execute the problem begun in 2. Set L = 0.5 and u(79).

4. Use the shooting method to estimate the solution to the 1-D

Helmholtz equation u00 + u = 0 on the interval [0, 2] with u(0) = 0 and
u(2) = 0.

5. The actual solution to 4 is u(x) = sin(x). Using your estimated

solution conmpute the mean realtive absolute error.

6. If you were to set up a problem using a cooling fin immersed in liquid

nitrogen with u(0) = 80, what would be the appropriate value for x?

6.4 The Method of Lines, Revisiting the Heat Equation

There are two ways to looik at method of line We begin with the one
that ighlights the role of lines. To provide context, we recall the 1-D heat
equation, ut = uxx .
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 129

Numerical Ordinary Differential Equations 129

The method of lines proceeds as follows. From the set up, the time-
space domain of the problem is a rectangle D = [0, T ] [0, 10.0]. We first
designate t and x and thereby determine a lattice of points (tn , xi ) in
the domain. Exactly as with FDM where we wrote uni for u(tn , xi ). We
now write ui for u(t, xi ). Using a second central difference and a single
forward difference on the right hand side of (9.5.6) we have
ui  
= 2
u(t, xi+1 ) 2u(t, xi ) + u(t, xi1 ) =
t x
 
2
ui+1 (t) 2ui (t) + ui1 (t) (6.4.1)
x
Hence, we have arrived at an ODE where the left hand side is the
derivative of ui and the right hand side is a function of ui , ui1 and ui+1 .
From here we may employ any of our ODE techniques. Should we choose
to use forward Euler we have
t  n
un+1 = uni + ui+1 2uni + uni1 .

i 2
x
This we immediately recognize as FTCS FDM.
Alternatively, consider the extimadtor/corrector. In this case
t  n
un+1 = uni + (ui+1 2uni + uni1 ) + (un+1 n+1
+ un+1

i 2 i+1 2ui i1 ) ,
2x
We recognize this as Crank Nicolson.

Exercises:
1. Develop the 1-D heat equation with the method of lines with modi-
point method of the ODE side.
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 130
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 131

Bibliography

Atkinson, K. E. (1989) An Introduction to Numerical Analysis, 2nd edn. (J.

Wiley).
Golub, G. H. and Van Loan, C. F. (2012). Matrix Computations, 4th. edn. (Johns
Hopkins).
Grcar, J. P. (2011). John von Neumanns Analysis of Gaussian Elimination and
the Origins of Modern Numerical Analysis, SIAM Review 53 pp 607-682.
Hildebrand, F. B. (1974). Introduction to Numerical Analysis, 2nd edn,( Dover).
Liesen, J. and Strakos, Z. (2015). Krylov Methods: Principles and Analysis, Ox-
ford Univ. Press).
Loustau, J. and Dillon M. (1993). Linear Geometry with Computer Graphics,
(Marcel Dekker).
Loustau, J (2016). Numercial Differential Equastions, Theory and Technique,
ODE Methods, Finite Differences, Finite Elements and Collocation, World
Sceinctific Press.
Marsden, J. E. and Tromba, A. J. (2003). Vector Calculus 5th. edn. (W. H.
Freeman).
Rudin, W. (1976). Elements of Real Analysis, 3ed. edn. (McGraw-Hill).
Simmins, G. F. and Robertson, J. S. (1991),Differential Equations with Applica-
tions and Historical Notes, 2nd. edn. (McGraw-Hill).
Su, B., Q. and Liu, D. Y. (1989), Comptational Geometry: Curve and Surface
Modeling, (Academic Press).
Thomas, J. W. (1999), Numerical Partial Differential Equations, Conservation
Laws and Elliptical Equations, (Springer).
Wodarz, D. and Komarova, N. L. (2014) Dynanmics of Cancer, Mathematical
Foundations of Oncology, (World Scientific Press).

131
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 132
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 133

Index

affine mapping, 114 Neumann stability analysis, 85, 91

finite differences
boundary value problem, 126 first backward difference, 77
first central difference, 77
condition number, 2931 first forward difference, 77
second central difference, 77
dimensionaless, 7 first order wave equation, 85, 91, 95
fluid flow
error, 7 potential function, 78
absolute error, 7 Fourier transform
relative absolute error, 7 discrete Fourier coefficients, 87
relative error, 7 discrete Fourier interpolation, 87
estimating roots discrete Fourier transform,
absolute error estimate, 17 properties, 88
FindRoot, 11, 17
Hessian method, 40 Hessian metnod, 40
maximal descent, 38
Newtons Method, 9 ill conditioned, 30
secant method, 14 initial value problem, 126
seed, 11, 14 interpolation error, 48

FDM Lagrange polynomials, 47, 108

backward Euler, 92 Laguerre polynomials, 111
BTCS, 91 linear systems
Courant number, 85 condition number, 2931
Crank Nicolson, 93, 95, 123, 129 elementary matrices, 21
explicit, 81 elementary row operations, 21
forward Euler, 81 Gauss-Jordan elimination, 21
FTCS, 82, 129 Gauss-Seidel method, 33
implicit, 92 ill conditioned, 30
method of lines, 128 Jacobi method, 33

133
August 21, 2016 1:36 ws-book9x6 Elements of Numerical Analysis with Mathematica... ws-book9x6Book2 page 134

134 Elements of Numerical Analysis with Mathematica

LinearSolve, 21 Bezier curve, 54

LUDecomposition, 21, 24 Bezier guide points, 53
residual correction, 32 knot points, 63
linear transformation local control, 68
operator norm, 27 numerical integration, 112
local control, 50 segments, 63
spline curve, 63
maximal descent, 38 spline guide points, 66
metric, 28, 29 permutation matrix, 24
polynomial interpolation, 46
normed linear space, 28 Hermite interpolation, 71
numercial integration Hermite polynomials, 71
functions in parametric form, 112 interpolation error, 48
numerical integration Lagrange Polynomials, 106
Chebyshev quadrature, 111 Lagrange polynomials, 47, 108
Gaussian quadratuer for functions local control, 50
in parpmetric form, 113 mean norm interpolation error, 45
Gaussian quadrature, 108 norm interpolation error, 45
Gaussian quadrature over a Taylor polynomial interpolation, 44
rectangle, 114 Vandermonde matrix, 46
Gaussian quadrature over a population model
triangle, 114 birth/death model, 121
Gaussian quadrature points, 108, power iteration method, 34
109
Hermite quadrature, 107 residual, 9
Hermitian quadrature, 106 residual correction, 32
Hermitian quadrature error, 107 residual correction method, 32
Hermitian quadrature weights, 106
Laguerre quadrature, 111 semi-norm, 86
midpoint method, 102 spectral radius, 30
Simpson;s rule, 100 power iteration method, 34
trapezoid method, 100 spline curve, 63

ODE carrying capacity, 124 Vandermonde matrix, 46

ODE method
forward Euler, 119
forward Euler with corrector, 119
method of lines, 128
midpoint method, 123
one dimensional heat equation, 81
orthongal functions, 106

parametric curve
B-spline basis functions, 67
B-spline fit, 68
B-spline guide points, 68