NUMERICAL MATHEMATICS & COMPUTING

7th Edition

Ward Cheney/David Kincaid

c

UT Austin

Engage Learning: Thomson-Brooks/Cole

www.engage.com

www.ma.utexas.edu/CNA/NMC6

October 16, 2011

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 1 / 48
 Systems of Linear Equations

Figure: Electrical network

A simple electrical network contains a number of resistances and a

single source of electromotive force (a battery) as shown:

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 2 / 48
 Using Kirchhoffs Laws and Ohms Law, we can write a system of
linear equations that govern this circuit.
If x1 , x2 , x3 , and x4 are the loop currents as shown, then the equations
are

15x1 2x2 6x3 = 300
2x1 + 12x2 4x3 x4 = 0

6x1
4x2 + 19x3 9x4 = 0
x2 9x3 + 21x4 = 0

Systems of equations like this, even those that contain hundreds of

unknowns, can be solved by using the methods developed here.
The solution to the preceding system is

x1 = 26.5, x2 = 9.35, x3 = 13.3, x4 = 6.13

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 3 / 48
 2.1 Naive Gaussian Elimination

One of the fundamental problems in many scientific and engineering

applications is to solve an algebraic linear system

Ax = b

for the unknown vector x when the coefficient matrix A and

right-hand side vector b are known.
Such systems arise naturally in various applications, such as
approximating nonlinear equations by linear equations or
differential equations by algebraic equations
The cornerstone of many numerical methods for solving a variety of
practical computational problems is the efficient and accurate solution
of linear systems.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 4 / 48
 The system of linear algebraic equations

Ax = b

may or may not have a solution, and if it has a solution, it may or

may not be unique.
Gaussian elimination is the standard method for solving the linear
system by using a calculator or a computer.
This method is undoubtedly familiar to most readers, since it is the
simplest way to solve a linear system by hand.
When the system has no solution, other approaches are used, such as
linear least squares, which is discussed in Chapter 9.
Here we assume that the coefficient matrix A is n n and invertible
(nonsingular).

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 5 / 48
 In a pure mathematical approach, the solution to the problem Ax = b
is simply x = A1 b, where A1 is the inverse matrix.
But in most applications, it is advisable to solve the system directly
for the unknown vector x rather than explicitly computing the inverse
matrix.
In applied mathematics and in many applications, it can be a daunting
task for even the largest and fastest computers to solve accurately
extremely large systems involving thousands or millions of unknowns.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 6 / 48
 Some of the questions are the following:
How do we store such large systems in the computer?
How do we know that the computed answers are correct?
What is the precision of the computed results?
Can the algorithm fail?
How long will it take to compute answers?
What is the asymptotic operation count of the algorithm?
Will the algorithm be unstable for certain systems?
Can instability be controlled by pivoting?
Permuting the order of the rows of the matrix is called pivoting.
Which strategy of pivoting should be used?
How do we know whether the matrix is ill-conditioned and whether the
answers are accurate?

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 7 / 48
 Gaussian elimination transforms a linear system into an upper
triangular form, which is easier to solve.
This process, in turn, is equivalent to finding the factorization

A = LU

where L is a unit lower triangular matrix and U is an upper triangular

matrix.
This factorization is especially useful when solving many linear
systems involving the same coefficient matrix but different right-hand
sides, which occurs in various applications!

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 8 / 48
 When the coefficient matrix A has a special structure such as being
symmetric, positive definite,
triangular, banded,
block, or sparse,
the general approach of Gaussian elimination with partial pivoting
needs to be modified or rewritten specifically for the system.
When the coefficient matrix has predominantly zero entries, the
system is sparse and iterative methods can involve much less
computer memory than Gaussian elimination.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th EditionOctober 16,www.ma.utexas.edu/
2011 9 / 48
 Our objective is to develop a good program for solving a system of n
linear equations in n unknowns:

a11 x1 + a12 x2 + a13 x3 + + a1n xn =


b1
a21 x1 + a22 x2 + a23 x3 + + a2n xn = b2








a31 x1 + a32 x2 + a33 x3 + + a3n xn = b3
. .. .. .. ..
.. . . . . (1)




ai1 x1 + ai2 x2 + ai3 x3 + + ain xn = bi

.. .. .. .. ..
. . . . .





an1 x1 + an2 x2 + an3 x3 + + ann xn = bn

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 10 / 48
 In compact form, this system can be written simply as
n
X
aij xj = bi (1 i n)
j=1

In these equations, aij and bi are prescribed real numbers (data), and
the unknowns xj are to be determined.
Subscripts on the letter a are separated by a comma only if necessary
for clarityfor example, in a32,75 , but not in aij .

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 11 / 48
 A Larger Numerical Example

Here, the simplest form of Gaussian elimination is explained.

The adjective naive applies because this form is not usually suitable
for automatic computation unless essential modifications are made.
We illustrate naive Gaussian elimination with a specific example that
has four equations and four unknowns:


6x1 2x2 + 2x3 + 4x4 = 16
12x1 8x2 + 6x3 + 10x4 = 26

(2)

3x1 13x2 + 9x3 + 3x4 = 19
6x1 + 4x2 + x3 18x4 = 34

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 12 / 48
 In the first step of the elimination procedure, certain multiples of the
first equation are subtracted from the second, third, and fourth
equations so as to eliminate x1 from these equations.
Thus, we want to create 0s as coefficients for each x1 below the first
(where 12, 3, and 6 now stand).
It is clear that we should subtract 2 times the first equation from the
second.
Likewise, we should subtract 1/2 times the first equation from the
third.
Finally, we should subtract 1 times the first equation from the
fourth.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 13 / 48
 When all of this has been done, the result is

6x1 2x2 +
2x3 + 4x4 = 16
4x2 + 2x3 + 2x4 = 6

(3)

12x2 + 8x3 + x4 = 27
2x2 + 3x3 14x4 = 18

Note that the first equation was not altered in this process, although
it was used to produce the 0 coefficients in the other equations.
In this context, it is called the pivot equation.
Notice also that Systems (2) and (3) are equivalent in the following
technical sense:
Any solution of System (2) is also a solution of System (3), and vice
versa.
This follows at once from the fact that if equal quantities are added
to equal quantities, the resulting quantities are equal.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 14 / 48
 In the second step of the process, we mentally ignore the first
equation and the first column of coefficients.
This leaves a system of three equations with three unknowns.
The same process is now repeated using the top equation in the
smaller system as the current pivot equation.
Thus, we begin by subtracting 3 times the second equation from the
third.
Then we subtract 1/2 times the second equation from the fourth.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 15 / 48
 After doing the arithmetic, we arrive at


6x1 2x2 + 2x3 + 4x4 = 16
4x2 + 2x3 + 2x4 = 6

(4)

2x3 5x4 = 9
4x3 13x4 = 21

The final step consists in subtracting 2 times the third equation from
the fourth.
The result is


6x1 2x2 + 2x3 + 4x4 = 16
4x2 + 2x3 + 2x4 = 6

(5)

2x3 5x4 = 9
3x4 = 3

This system is said to be in upper triangular form.

System (4) equivalent to System (2).
This completes the first phase forward elimination in the Gaussian
algorithm.
Ward Cheney/David Kincaid
c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 16 / 48
 The second phase (back substitution) will solve System (5) for the
unknowns starting at the bottom.
Thus, from the fourth equation, we obtain the last unknown

x4 = 3/(3) = 1

Putting x4 = 1 in the third equation gives us

2x3 5 = 9

and we find the next to last unknown

x3 = 4/2 = 2

and so on.
The solution is

x1 = 3, x2 = 1, x3 = 2, x4 = 1

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 17 / 48
 Algorithm

To simplify the discussion, we write System (1) in matrix-vector form.

The coefficient elements aij form an n n square array, or matrix.
The unknowns xi and the right-hand side elements bi form n 1
arrays, or vectors.
To save space, we occasionally write a vector as [x1 , x2 , . . . , xn ]T ,
where the T stands for the transpose.
It tells us that this is an n 1 array or vector and not 1 n, as would
be indicated without the transpose symbol.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 18 / 48

a11 a12 a13 a1n x1 b1

a21 a22 a23 a2n
x2
b2


a31 a32 a33 a3n
x3
b3

.. .. .. .. .. ..

. . . .
. =
.
(6)

ai1 ai2 ai3 ain

xi
bi

.. .. .. .. .. ..
. . . . . .
an1 an2 an3 ann xn bn

So
Ax = b
Operations between equations correspond to operations between
rows in this notation.
We shall use these two words interchangeably.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 19 / 48
 Now let us organize the naive Gaussian elimination algorithm for
the general system, which contains n equations and n unknowns.
In this algorithm, the original data are overwritten with new
computed values.
In the forward elimination phase of the process, there are n 1
principal steps.
The first of these steps uses the first equation to produce n 1 zeros
as coefficients for each x1 in all, but the first equation.
This is done by subtracting appropriate multiples of the first equation
from the others.
In this process, we refer to the first equation as the first pivot
equation and to a11 as the first pivot element.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 20 / 48
 For each of the remaining equations (2 i n), we compute


aij aij ai1 /a11 a1j (1 j n)


bi bi ai1 /a11 b1

The symbol indicates a replacement.

Thus, the content of the memory location allocated to aij is replaced
by aij (ai1 /a11 )a1j , and so on.
This is accomplished by the following line of pseudocode:

aij aij (ai1 /a11 )a1j

Note that the quantities (ai1 /a11 ) are the multipliers.

The new coefficient of x1 in the i-th equation will be 0 because

ai1 (ai1 /a11 )a11 = 0

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 21 / 48
 After the first step, the system has the form

a11 a12 a13 a1n x1 b1
0 a22
a23 a2n
x2
b2

0 a23
a33 a3n

x3
b3

.. .. .. .. .. ..
. . . .
. = .

0 ai2
ai3 ain
xi
bi

.. .. .. .. .. ..
. . . . . .
0 an2 an3 ann xn bn

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 22 / 48
 From here on, we do not alter the first equation, nor do we alter any
of the coefficients for x1 (since a multiplier times 0 subtracted from 0
is still 0).
Thus, we can mentally ignore the first row and the first column and
repeat the process on the smaller system.
With the second equation as the pivot equation, we compute for
each remaining equation (3 i n)


aij aij ai2 /a22 a2j (2 j n)


bi bi ai2 /a22 b2

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 23 / 48
 Just prior to the k-th step in the forward elimination, the system
appears as follows:

a11 a12 a13 a1n x1 b1
0 a22 a23 a2n x2 b2

0
0 a33 a3n
x3
b3

.. .. .. .. .. .. ..
.
. . . .
.
.

0
0 0 akk akj akn
xk =
bk

.. .
.. .. .. .. .. .. .. ..
. . . . . . . .

0
0 0 a ik aij ain
xi
bi

.. .. .. .. .. .. .. .. ..
. . . . . . . . .
0 0 0 ank anj ann xn bn

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 24 / 48
 Here, a wedge of 0 coefficients has been created, and the first k
equations have been processed and are now fixed.
Using the k-th equation as the pivot equation, we select multipliers to
create 0s as coefficients for each xi below the akk coefficient.
Hence, we compute for each remaining equation (k + 1 i n)


aij aij aik /akk akj (k j n)


bi bi aik /akk bk

Obviously, we must assume that all the divisors in this algorithm are
nonzero.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 25 / 48
 Pseudocode

We now consider the pseudocode for forward elimination.

coefficient array is stored as a double-subscripted array (aij );
right-hand side of the system of equations is stored as a
single-subscripted array (bi );
solution is computed and stored in a single-subscripted array (xi )

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 26 / 48
 It is easy to see that the following lines of pseudocode carry out the
forward elimination phase of naive Gaussian elimination:

integer i, j, k; real array (aij )1:n1:n , (bi )1:n

for k = 1 to n 1
for i = k + 1 to n
for j = k to n
aij aij (aik /akk )akj
end for
bi bi (aik /akk )bk
end for
end for

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 27 / 48
 Since the multiplier aik /akk does not depend on j, it should be moved
outside the j loop.
Notice also that the new values in column k become 0, at least
theoretically, because when j = k, we have

aik aik (aik /akk )akk

Since we expect this to be 0, no purpose is served in computing it.

The location where the 0 is being created is a good place to store the
multiplier.
If these remarks are put into practice, the pseudocode looks like this:

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 28 / 48
 Pseudocode

integer i, j, k; real xmult; real array (aij )1:n1:n , (bi )1:n

for k = 1 to n 1
for i = k + 1 to n
xmult aik /akk
aik xmult
for j = k + 1 to n
aij aij (xmult)akj
end for
bi bi (xmult)bk
end for
end for

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 29 / 48
 Here, the multipliers are stored because they are part of the
LU-factorization that can be useful in some applications.
At the beginning of the back substitution phase, the linear system is
of the form
+ a13 x3 + + a1n xn = b1
+ a23 x3 + + a2n xn = b2
a33 x3 + + a3n xn = b3
.. .. ..
. . .
aii xi + ai,i+1 xi+1 + + ain xn = bi
.. .. ..
. . .
an1,n1 xn1 + an1,n xn = bn
ann xn = bn

Here the aij s and bi s are not the original ones from (6), but instead
are the ones that have been altered by the elimination process.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 30 / 48
 The back substitution starts by solving the n-th equation for xn :

xn = bn /ann

Then, using the (n 1)th equation, we solve for xn1 :

xn1 = bn1 an1,n xn /an1,n1

We continue working upward, recovering each xi by the formula

 n
X 
xi = bi aij xj /aii (i = n 1, n 2, . . . , 1) (7)
j=i+1

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 31 / 48
 Pseudocode

integer i, j, n; real sum; real array (aij )1:n1:n , (xi )1:n

xn bn /ann
for i = n 1 to 1 step 1
sum bi
for j = i + 1 to n
sum sum aij xj
end for
xi sum/aii
end for

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 32 / 48
 Now we put these segments of pseudocode together to form a
procedure, called Naive Gauss, which is intended to solve a system of
n linear equations in n unknowns by the method of naive Gaussian
elimination.
This pseudocode serves a didactic purpose only; a more robust
pseudocode is be developed in the next section.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 33 / 48
 procedure Naive Gauss(n, (aij ), (bi ), (xi ))
integer i, j, k, n; real sum, xmult
real array (aij )1:n1:n , (bi )1:n , (xi )1:n
for k = 1 to n 1
for i = k + 1 to n
xmult aik /akk
aik xmult
for j = k + 1 to n
aij aij (xmult)akj
end for
bi bi (xmult)bk
end for
end for
xn bn /ann

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 34 / 48
 for i = n 1 to 1 step 1 do
sum bi
for j = i + 1 to n
sum sum aij xj
end for
xi sum/aii
end for
end procedure Naive Gauss

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 35 / 48
 Before giving a test example, let us examine the crucial computation
in our pseudocode, namely, a triply nested for-loop containing a
replacement operation:

for k
for i
for j
aij aij (aik /akk )akj
end do
end do
end do

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 36 / 48
 Here, we must expect all quantities to be infected with roundoff
error.
Such a roundoff error in akj is multiplied by the factor (aik /akk ).
This factor is large if the pivot element |akk | is small relative to |aik |.
Hence, we conclude, tentatively, that small pivot elements lead to
large multipliers and to worse roundoff errors.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 37 / 48
 Testing the Pseudocode

One good way to test a procedure is to set up an artificial problem

whose solution is known beforehand.
Sometimes the test problem includes a parameter that can be
changed to vary the difficulty.
Fixing a value of n, define the polynomial
n
X
2 n1
p(t) = 1 + t + t + + t = t j1
j=1

The coefficients in this polynomial are all equal to 1.

We shall try to recover these known coefficients from n values of the
polynomial.
We use the values of p(t) at the integers t = 1 + i for i = 1, 2, . . . , n.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 38 / 48
 If the coefficients in the polynomial are denoted by x1 , x2 , . . . , xn , we
should have
n
X
(1 + i)j1 xj = (1 + i)n 1 i
 
(1 i n) (8)
j=1

Here, we have used the formula for the sum of a geometric series on
the right-hand side; that is,
n
X
(1 + i)j1 = (1 + i)n 1/(1 + i) 1 = (1 + i)n 1 i
 
p(1 + i) =
j=1
(9)
i)j1 i)n

Letting aij = (1 + and bi = [(1 + 1] i, we have a linear
system.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 39 / 48
 Example
We write a pseudocode for a specific test case that solves the system of
System (8) for various values of n.

Since the naive Gaussian elimination procedure Naive Gauss can be

used, all that is needed is a calling program.
We decide to use n = 4, 5, 6, 7, 8, 9, 10 for the test.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 40 / 48
 A Suitable Pseudocode

program Test NGE

integer parameter m 10
integer i, j, n; real array, (aij )1:m1:m , (bi )1:m , (xi )1:m
for n = 4 to 10
for i = 1 to n do
for j = 1 to n
aij (i + 1)j1
end for
bi [(i + 1)n 1]/i
end for
call Naive Gauss(n, (aij ), (bi ), (xi ))
output n, (xi )1:n
end for
end program Test NGE

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 41 / 48
 When this pseudocode is run on a machine that carries approximately
seven decimal digits of accuracy, the solution is obtained with
complete precision until n = 9, and then the computed solution is
worthless because one component exhibited a relative error of
16,120%!

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 42 / 48
 The coefficient matrix for this linear system is an example of a
well-known ill-conditioned matrix called the Vandermonde matrix.
This accounts for the fact that the system cannot be solved
accurately using naive Gaussian elimination.
What is amazing is that the trouble happens so suddenly!
When n 9, the roundoff error that is present in computing xi is
propagated and magnified throughout the back substitution phase so
that most of the computed values for xi are worthless!
Insert some intermediate print statements in the code to see for
yourself what is going on here.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 43 / 48
 Residual and Error Vectors

For a linear system

Ax = b
having the true solution x and a computed solution e
x , we define

xx
e=e (error vector)
xb
r = Ae (residual vector)

An important relationship between the error vector and the residual

vector is
Ae = r

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 44 / 48
 Suppose that two students using different computer systems solve the
same linear system
Ax = b
What algorithm and what precision each student used are not known.
Each vehemently claims to have the correct answer, but the two
computer solutions e
x and b
x are totally different!
How do we determine which, if either, computed solution is correct?
We can check the solutions by substituting them into the original
system, which is the same as computing the residual vectors

xb
er = Ae
xb
br = Ab

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 45 / 48
 Of course, the computed solutions are not exact because each must
contain some roundoff errors.
So we would want to accept the solution with the smaller residual
vector.
However, if we knew the exact solution x, then we would just
compare the computed solutions with the exact solution, which is the
same as computing the error vectors

xx
e=e
e
xx
e=b
b

Now the computed solution that produces the smaller error vector
would most assuredly be the better answer.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 46 / 48
 Since the exact solution is usually not known in applications, one
would tend to accept the computed solution that has the smaller
residual vector.
But this may not be the best computed solution if the original
problem is sensitive to roundoff errorsthat is, is ill-conditioned.
In fact, the question of whether a computed solution to a linear
system is a good solution is extremely difficult and beyond the scope
of this book.

Ward Cheney/David Kincaid

c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 47 / 48
 Summary 2.1
The basic forward elimination procedure using equation k to
operate on equations k + 1, k + 2, . . . , n is
(
aij aij (aik /akk )akj (k j n, k < i n)
bi bi (aik /akk )bk
Here we assume akk 6= 0.
The basic back substitution procedure is
 X n .
xi = bi aij xj aii (i = n 1, n 2, . . . , 1)
j=i+1

When solving the linear system Ax = b, if the true or exact solution is

x and the approximate or computed solution is ex, then important
quantities are
e=e xx (error vectors)
xb
r = Ae (residual vectors)
Ward Cheney/David Kincaid
c (UT Austin[10pt]
NUMERICAL
Engage Learning:
MATHEMATICS
Thomson-Brooks/Cole
& COMPUTING www.engage.com[10pt]
7th Edition
October 16, www.ma.utexas.edu/
2011 48 / 48