CSC475: Numerical Computing

Slides Prepared by: Tanveer Ahmed Siddiqui

Presented by: Shawana Jamil
Department of Computer Science
COMSATS University, Islamabad
Recap
Numerical Methods
for Solving
Nonlinear Equations
Lecture 3-5

Department of Computer Science 2

 Recap

 In Lecture 3-5, the techniques for the numerical

solution of algebraic and transcendental
equations have been presented.
 Numerical methods involving iterative solution of
nonlinear equations are more powerful.
 These methods can be divided into two
categories:
 Direct methods
 Indirect (or iterative) methods.

Department of Computer Science 3

 Recap

 The indirect or iterative methods are further

divided into two categories:
 Bracketing methods
 Open methods.
 The bracketing methods require the limits
between which the root lies, whereas the open
methods require the initial estimation of the
solution.
 Bisection and False Position methods are two
known examples of the bracketing methods.

Department of Computer Science 4

 Recap

 Among the open methods, the Newton-Raphson

and the method of Successive Approximation are
most commonly used.
 The Bisection method, the method of False
Position, and the method of Successive
Approximations converge linearly while the
Newton-Raphson method converges quadratically.
 Newton-Raphson method requires less number of
iterations than the other three methods.
 The Successive Approximation method converges
only when the condition |φ′(x)| < 1 is satisfied.

Department of Computer Science 5

 Recap

Department of Computer Science 6

 Comparison of the methods

Department of Computer Science 7

 Lecture No 6

Solution of
System of Linear equation

Department of Computer Science 8

 Today Covered
,

After completing this lecture you should be able to

know
 What is linear equation?
 What do we mean by system of linear
equations(SLE)?
 Classifications of various Methods for finding
solution of SLE?
 Gauss Elimination Method
 Example
 Gauss Jordon Method
 Example

Department of Computer Science

 Linear Equation

 Linear Equations
 Only simple variables are allowed in linear equations.
 No x2, y3, √x, etc.
 Linear vs non-linear

Department of Computer Science 10

 Dimension

 A Linear Equation can be in 2 dimensions

(such as x and y)
 Examples:
 x+y=6
 −3x + y = 2
 It represents a line
 A Linear Equation can be in 3 dimensions
 (it makes a plane)
 Example
 3x + 7y +z = 12

Department of Computer Science 11

 What is System of Linear Equations

 A System of Linear Equations(SLE) is when we

have two or more linear equations working
together.

 There can be any combination:

 2 equations in 3 variables,
 6 equations in 4 variables,
 9,000 equations in 567 variables,
Department of Computer Science 12
 You VS Horse
 You can run 0.2km every minute
 Horse can run 0.5 km per minute but take 6
min to saddle horse
 How far can you get before horse catches you?
 D=0.2t
 D=0.5(t-6)

Department of Computer Science 13

 How to write SLE in various Form?

 A set of m simultaneous linear equation in n

unknown x1, x2, ,……xn can be written in various
form as:

Compact from
Ax = b
General form

Department of Computer Science Matrix form 14

 System of Linear Equations

Department of Computer Science 15

 System of Linear Equations

If m = n, that is number of equations and number

of unknown are equals, then the set of n equations
and n unknowns is given
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
. .
. .
. .
an1 x1  an 2 x2  an 3 x3  ...  ann xn  bn

Department of Computer Science

 Types of System of Linear Equations

Department of Computer Science

 Properties of SLE

 Independence
 The equations of a linear system are independent if
none of the equations can be derived algebraically
from the others.
 For example, following equations are not independent
— they are the same equation when scaled by a factor
of two, and they would produce identical graphs.


Department of Computer Science 18
 Properties of SLE

 Independence
 For example, the equations

x − 2y = −1,
3x + 5y = 8,
4x + 3y = 7
are not independent, because the third equation is
the sum of the other two.

Department of Computer Science 19

 Solution of a linear system

 A solution of a linear system is an assignment

of values to the variables x1, x2, ..., xn such that
each of the equations is satisfied.
 The set of all possible solutions is called
the solution set.
 The solution set for the equations
x − y = −1 and 3x + y = 9
is the single point (2, 3).

Department of Computer Science 20

Geometric interpretation of solution of a linear system

 For a system involving two variables (x and y),

each linear equation determines a line on
the xy-plane.
 Because a solution to a linear system must
satisfy all of the equations, the solution set is
the intersection of these lines, and is hence
either
 a line,
 a single point,
 or the empty set.

Department of Computer Science 21

Geometric interpretation of solution of a linear system

 Because a solution to a linear system must satisfy all of the

equations, the solution set is the intersection of these lines, and is
hence either
 a line
 The system has infinitely many solutions, namely all of
the points on the blue line.
 a single point
 The system has a single unique solution, namely the
intersection of the two lines.
 or the empty set
 The system has no solutions, since the three lines share
no common point.

Department of Computer Science 22

Geometric interpretation of solution of a linear system

 It must be kept in mind that the pictures in

previous slide show only the most common case
(the general case).
 It is possible for a system of two equations and
two unknowns to have no solution (if the two
lines are parallel), or for a system of three
equations and two unknowns to be solvable (if
the three lines intersect at a single point).

Department of Computer Science 23

Geometric interpretation of solution of a linear system

 The solution set for two equations in three

variables is, in general, a line
 For example,

is a system of three equations in the three

variables x, y, z.
 A solution to the system above is given by:

 the solution set of the equations of three planes

intersecting at a point is single point;
Department of Computer Science 24
 Upshot Solution of SLE

 The solution of the system of equations gives n

unknown values x1, x2, …, xn, which satisfy the
system simultaneously.
 There are three possible cases:
 No solution
 One/unique solution
 Infinitely many solutions

Department of Computer Science 25

What is the condition for existence of solution of SLE?

 Now let us understand the condition for

existence of solution of SLE or not.
 In general, the behavior of a linear system is
determined by the relationship between the
number of equations and the number of
unknowns.
 Here, "in general" means that a different
behavior may occur for specific values of the
coefficients of the equations.

Department of Computer Science 26

 Condition for existence of solution of SLE

 In general, a system with fewer equations than

unknowns(i.e. m < n) has infinitely many
solutions, but it may have no solution.
 Such a system is known as an underdetermined
system
 In general, a system with the same number of
equations and unknowns(i.e. m = n) has a
single unique solution.
 In general, a system with more equations than
unknowns(i.e. m > n) has no solution.
 Such a system is also known as an overdetermined
system
Department of Computer Science 27
 Condition for existence of solution of SLE

 No solution
 If m = n and then there exists no solution
 In this case equations are called inconsistent
 Unique solution
 If m= n and then the system will have a unique
solution.
 In this case equations are called consistent
 Infinitely many solutions
 If m < n, the system usually will have an infinite
number of solutions.
 In this case equations are also called consistent

Department of Computer Science 28

 Upshot

 No solution
 One solution
 Infinitely many solutions

Department of Computer Science 29

 Upshot

 No solution
 One solution
 Infinitely many solutions

Department of Computer Science 30

 Example
 Given linear system of two equations
x + y = 7,
−x + 3y = 1, (1)
By adding both equations, we obtain 4y = 8,
and hence y = 2.
Substituting y = 2 into the first equation (1) yields
x = 5.

Department of Computer Science 31

 Methods of solving SLE

 In our lectures we will find solution of SLE that

satisfying the condition m = n
 The general form of a system of n linear
algebraic equations is:

 The matrix form of the equations is

Department of Computer Science 32

Classification of Methods of solving SLE

 There are several algorithms for solving a

system of linear equations.
 We classify these methods into two groups:
 Direct Methods
 Iterative Methods
 The common characteristics of direct methods
are that they transform the original equation
into equivalent equations (equations that have
the same solution) that can be solved more
easily.
 The transformation is carried out by applying
certain operations.
Department of Computer Science 33
Classification of Methods of solving SLE

 The transformation is carried out by applying

certain operations.
 replacing an equation with its nonzero multiple
 replacing an equation with a sum or difference of
this equation and some multiple of another equation
 exchanging two equations of the system
 The solution does not contain any truncation
errors but the round off errors is introduced due
to floating point operations.

Department of Computer Science 34

 Direct Methods

 1. Matrix Inverse Method

 2. Gauss Elimination Method
 3. Gauss-Jordan Method
 4. Cholesky’s Triangularization Method
 5. Crout’s Method
 6. Thomas Algorithm for Tridiagonal System

Department of Computer Science 35

 Upshot

 We use three operations to simplify the linear

system:
 Equation Ei can be multiplied by any nonzero
constant c with the resulting equation used in place
of Ei. This operation is denoted (c Ei) → (Ei).
 Equation Ej can be multiplied by any constant c and
added to equation Ei with the resulting equation
used in place of Ei. This operation is denoted
(Ei+cEj) → (Ei).
 Equations Ei and Ej can be transposed in order. This
operation is denoted (Ei) ↔ (Ej).

Department of Computer Science 36

Classification of Methods of solving SLE

 Iterative or indirect methods, start with a guess

of the solution x, and then repeatedly refine the
solution until a certain convergence criterion is
reached.
 Iterative methods are generally less efficient
than direct methods due to the large number of
operations or iterations required.

Department of Computer Science 37

Classification of Methods of solving SLE

 Iterative methods are self-correcting, meaning

that round off errors (or even arithmetic
mistakes) in one iteration cycle are corrected in
subsequent cycles.
 The solution contains truncation error.
 A serious drawback of iterative methods is that
they do not always converge to the solution.
 The initial guess affects only the number of
iterations that are required for convergence.
 The indirect solution technique (iterative) is
 more useful to solve a set of ill-conditioned
equations.
Department of Computer Science 38
 Indirect or Iterative Methods

 1.Jacobi’s Iteration Method

 2. Gauss-Seidal Iteration Method

In iterative methods, an initial approximate

solution is assumed and then used in an
iterative process for obtaining successively more
accurate solutions.

Department of Computer Science 39

 Matrix Revision

 Computing a Determinant

 Lower-triangular matrix L
 It has zero coefficients above the diagonal.

Department of Computer Science 40

 Matrix Revision

 Upper-triangular matrix U
 It has all zero coefficients below the diagonal

 Diagonal Matrix
 It has nonzero coefficients along the diagonal and
zeros everywhere else

Department of Computer Science 41

 Your Turn

 Let

 a) Compute det(A)
 b) Find its lower triangular Matrix
 c) Find its upper triangular Matrix
 d) Find its diagonal form

Department of Computer Science 42

What general algorithmic design technique the direct methods based on?

 Direct methods employ transform-and-conquer

algorithmic designing technique.
 We call this designing technique transform-and-
conquer because these methods work as two-
stage procedures.
 First, in the transformation stage, the problem’s
instance is modified into simpler instance.
 Then, in the second or conquering stage, it is
solved.

Department of Computer Science 43

 Direct Methods

 In direct methods how we transform system of

linear equation into simpler instance?
 In direct methods, the system of equations that
is initially given in the general form, is
manipulated to an equivalent system of
equations that can be easily solved.
 Three systems of equations that can be easily
solved are:
 Upper triangular forms
 Lower triangular forms
 Diagonal forms.

Department of Computer Science 44

 Upper triangular form

 The upper triangular form is shown in Eqs. (4.4)

and is written in a matrix form for a system of
four equations in Fig. 4-4.
Transformation

Figure 4-4: A system of four

equations in upper triangular form.

Department of Computer Science 45

 Upper triangular form

 The system in this form has all zero coefficients

below the diagonal and

Figure 4-4: A system of

four equations in upper
triangular form.
 The upper triangular form is used in
Gauss Elimination method which we will
discuss shortly.
Department of Computer Science 46
 Lower triangular form

 The lower triangular form is shown in Eqs.

( 4.6) and is written in matrix form for a system
of four equations in Fig. 4-5.
Transformation

Figure 4-5: A system of four

equations in lower triangular form.

Department of Computer Science 47

 Lower triangular form

 The lower triangular form is shown in Eqs.

( 4.6) and is written in matrix form for a system
of four equations in Fig. 4-5.

Figure 4-5: A system of

four equations in lower
triangular form.
 The lower triangular form is used
together with the upper triangular form in
LU decomposition method for solving a
system of equations.
Department of Computer Science 48
 Diagonal form

 The diagonal form of a system of linear

equations is shown in Eqs. (4.9), and is written
in matrix form for a system of four equations in
Fig. 4-6.

Transformation

Figure 4-6: A system of four

equations in diagonal form.

Department of Computer Science 49

 Diagonal form

 The diagonal form of a system of linear equations is shown in Eqs.

(4.9), and is written in matrix form for a system of four equations in
Fig. 4-6.

 A similar form is used in the Gauss-Jordan method.

Figure 4-6: A system
of four equations in
diagonal form.

Department of Computer Science 50

 Summary

Department of Computer Science 51

 Direct Method of solving SLE

 We will discuss efficient methods for solving SLE that

satisfies the condition m = n
 That is number of equations = The number of
unknowns
 The general form of a system of n linear algebraic
equations is:

 The matrix form of the equations is

Department of Computer Science 52

 Gauss Elimination Method

 A method to solve simultaneous linear

equations of the form [A][X]=[B]
 Two steps
 Forward Stage(Transform)
 This stage is concerned with the manipulation of
equations in order to eliminate some unknowns
from the equations and produces an upper
triangular system.
 Back Substitution(Conquer)
 This stage is concerned with the actual solution of
the equation and uses the back-substitution
process on the reduced upper triangular system.
Department of Computer Science 53
Illustration of Gauss Elimination Method with 4x4 SLE

Let us illustrate GE method with 4x 4 SLE

Department of Computer Science

 Example

 Solve the following system of equations

using Gaussian elimination method

2x  3y  z  5
4 x  4 y  3z  3
2 x  3 y  z  1
Department of Computer Science
 Solution

 Stage I (Reduction to upper-triangular form):

 We eliminate x from the 2nd and 3rd equations.
 Divide first equation by 2 and then subtract the
resulting equation (multiplied by 4 and –2) from the
2nd and 3rd equations respectively. Thus,

2x  3y  z  5 ½*R1 3
x y 
z 5 
2 2 2 

4 x  4 y  3z  3 R2 – 4R1  2 y  z  7 
6 y  2z  6 
2 x  3 y  z  1 R3 -2R1 

Department of Computer Science
 Example

 The resulting new system is given by

3 z 5
x y  
2 2 2

 2 y  z  7 
6 y  2z  6 


Department of Computer Science
 Example

Now, we divide the 2nd equation by –2 and

eliminate y from the last equation and the
modified system is given by

3 z 5 3 z 5 
x  y    -1/2R x y  
2 2 2 2 2 2
2

 z 7 
 2 y  z  7  R -6R y  
2 2 
6 y  2z  6 
3 2

  5 z  15
 


Department of Computer Science

 Example

 Stage II (Backward substitution):

 From the last equation, we get

z 3
 using this value of z, the second equation
gives
7 3
y  2
2 2

Department of Computer Science

 Putting these values of y and z in the first
equation, we get
5 3
x   3 1
2 2
 Thus, the solution of the given system is
given by

x = 1, y = 2, z = 3

Department of Computer Science

 Your Turn

 Solve the following systems of equations

using Gaussian elimination method.

Department of Computer Science

Generalization of GE Method: Forward Stage

A set of n equations and n unknowns

a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
. .
. .
. .
an1 x1  an 2 x2  an 3 x3  ...  ann xn  bn

(n-1) steps of forward elimination

Department of Computer Science

 Forward Stage

Step 1
For Equation 2, divide Equation 1 by a11 and
multiply by a21 .

 a21 
 a ( a11 x1  a12 x2  a13 x3  ...  a1n xn  b1 )
 11 
a21 a21 a21
a21 x1  a12 x2  ...  a1n xn  b1
a11 a11 a11

Department of Computer Science

 Forward Stage

Subtract the result from Equation 2.

a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
a21 a21 a21
− a21 x1  a a12 x2  ...  a a1n xn  a b1
11 11 11
_________________________________________________
 a21   a21  a21
 a22  a12  x2  ...   a2 n  a1n  xn  b2  b1
 a11   a11  a11

or a x  ...  a x  b
'
22 2
'
2n n
'
2

Department of Computer Science

 Forward Stage

Repeat this procedure for the remaining equations

to reduce the set of equations as
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
'
a32 x2  a33
'
x3  ...  a3' n xn  b3'
. . .
. . .
. . .

an' 2 x2  an' 3 x3  ...  ann

'
xn  bn'

End of Step 1
Department of Computer Science
 Forward Stage

Step 2
Repeat the same procedure for the 3rd term of
Equation 3.
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  a3" n xn  b3"
. .
. .
. .

an" 3 x3  ...  ann

"
xn  bn"
End of Step 2
Department of Computer Science
 Forward Stage

At the end of (n-1) Forward Elimination steps, the system of

equations will look like
a11 x1  a12 x 2  a13 x3  ...  a1n x n  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
a x  ...  a x  b
"
33 3
"
3n n
"
3
. .
. .
. .

n 1 n 1 
ann xn  bn

End of Step (n-1)

Department of Computer Science
 Matrix form at the end of Forward Stage

a11 a12 a13  a1n   x1   b1 

0 a '
a ' '
 a 2 n x2     b2' 
 22 23    
0 0 a "
33  a3n   x3    b3 
" "

    
          
 0 0 0 0 ann   xn  bn 
(n 1 )
    (n-1 )

Department of Computer Science

 Back Substitution Starting Equations

a11 x1  a12 x 2  a13 x3  ...  a1n x n  b1

'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  an" xn  b3"
. .
. .
. .

n 1 n 1 
ann xn  bn

Department of Computer Science

 Back Substitution

Start with the last equation because it has only one

unknown ( n 1)
bn
x n  ( n 1)
a nn

Department of Computer Science

 Back Substitution

( n 1)
b
xn  n
( n 1)
a nn

bii 1  ai,ii11 xi 1  ai,ii12 xi  2  ...  ai,in1 xn

xi  i 1 for i  n  1,...,1
aii
n
i 1
bi   aiji 1 x j
j i 1
xi  i 1 for i  n  1,...,1
a ii

Department of Computer Science

 GE With pivoting

 What are the potential difficulties when

applying the Gauss Elimination Method?
 Since the pivot row is divided by the pivot
element, a problem will arise during the
execution of the Gauss elimination procedure if
the value of the pivot element is equal to zero.
 For example, for the following system of three
equations:

Department of Computer Science

 Pitfalls of GE

 GE procedure starts by taking the first

equation as the pivot equation and the
coefficient of x1, which is 0, as the pivot
coefficient. To eliminate the term 4x1 in the
second equation, the pivot equation is
supposed to be multiplied by 4/0 and then
subtracted from the second equation.
Obviously, this is not possible when the
pivot element is equal to zero.

Department of Computer Science

 GE With pivoting

 The division by zero can be avoided if the

order in which the equations are written is
changed such that in the first equation the
first coefficient is not zero. For example, in
the system above, this can be done by
exchanging the first two equations.

Department of Computer Science

 GE With pivoting

 In the general Gauss elimination

procedure, an equation (or a row) can be
used as the pivot equation (pivot row) only
if the pivot coefficient (pivot element) is not
zero.
 If the pivot element is zero, the equation
(i.e., the row) is exchanged with one of the
equations (rows) that are below, which has
a nonzero pivot coefficient. This exchange
of rows is called pivoting.
Department of Computer Science
 GE With pivoting

Department of Computer Science

 Pitfalls of GE

 The pivot element is small relative to the

other terms in the pivot row

 Significant errors due to rounding can

occur when the pivot element is small
relative to other elements in the pivot row.

Department of Computer Science

 Additional comments about pivoting

 If during the Gauss elimination procedure a

pivot equation has a pivot element that is
equal to zero, then if the system of
equations that is being solved has a
solution, an equation with a nonzero
element in the pivot position can always be
found.

Department of Computer Science

 Additional comments about pivoting

 The numerical calculations are less prone

to error and will have fewer round-off
errors if the pivot element has a larger
numerical absolute value compared to the
other elements in the same row.
 Consequently, among all the equations that can be
exchanged to be the pivot equation, it is better to select
the equation whose pivot element has the largest
absolute numerical value.


Department of Computer Science

 Example

 Stage I (Reduction to upper-triangular form)

R1/9
-4R1+R2
-4R1+R3
.
.
.
.

 Stage II Back substitution

Department of Computer Science 80

GAUSS JORDON
ELIMINATION METHOD

Department of Computer Science 81

 Gauss Jordon Elimination Method

 The Gauss-Jordan elimination method is a

procedure for solving a system of linear
equations, [A][X] = [B].
 In this procedure, a system of equations
that is given in a general form is
manipulated into an equivalent system of
equations in diagonal form as shown
below with normalized elements along the
diagonal. Transformation

Department of Computer Science

 Gauss Jordon Elimination Method

 This means that when the diagonal form of

the matrix of the coefficients, [A] , is
reduced to the identity matrix, the new
vector [B'] is the solution.

Department of Computer Science

 Example

Solve the system of equations using Gauss-Jordan elimination method:



x  2y  z  8 

2 x  3 y  4 z  20 

4 x  3 y  2 z  16 
Department of Computer Science
 Example

 In matrix notation, the given system can

be written as

1 2 1   x   8 
 2 3 4   y    20 
    
   
 4 3 2   z   16 
Department of Computer Science
 Example

 (-2) R1 +R2 and (-4) R1+R3

1 2 1   x   8 
0 1 2   y    4 
    
   
0 5 2   z   16 
Department of Computer Science
 Example

 Now, we eliminate y from the first and third rows using

the second row. Thus, we get

1 0 5   x   16 
0 1 2   y    4 
    
0 0 12   z   36 

Department of Computer Science

 Example

 Before, eliminating z from the first and

second row, normalize the third row with
respect to the pivot element, we get

1 0 5   x  16 
0 1 2   y    4 
    
  
0 0 1   z   3 

Department of Computer Science

 Example

 Using the third row of above Equation , and

eliminating z from the first and second rows,
we obtain

1 0 0   x   1 
0 1 0   y    2 
    
  
0 0 1   z   3  

Department of Computer Science

 Example

 The solution is
x =1,
 y =2,
 z =3.

Department of Computer Science