SOLVING A SYSTEM OF LINEAR EQUATIONS

4.4.1. Gauss Elimination .Used to solve a system of linear equations of the form B is an Mx1 matrix. Ax=B , where A is an MxN Matrix and

The Algorithm .At the k th iteration, the Gauss forward elimination results in: k k1 k1 . amn=amn amk /ak1 ak1 for m ,n=k 1, k1,, M (1) kk kn k k1 k1 k1 . bm =bm a mk /a kk bkk1 for m=k 1, k2,, M ...(2) . After having the triangular matrixvector equation, we find the solutions using backward substitution. . x m=bmm1
n=m1

m1 m1 amn x n/amm for m=M , M 1,, 1 ...(3)

k The core formula used for Gauss elimination requires division by akk 1 at the k th stage, where k k akk1 is the diagonal element in the k th row. What if akk1=0 ? In such a case, it is customary to switch the k th row and another row below it having the element of the largest absolute value in the k th column. This procedure, called partial pivoting .

Read more about this! 4.4.2. LU Decomposition Gaussian elimination with pivoting is the most efficient and accurate way to solve a system of linear equations of the form Ax=b . Most of the work in this method is spent on the matrix A itself. If we need to solve several different systems with the same A , and A is big, then we would like to avoid repeating the steps of Gaussian elimination on A for every different b . This can be accomplished by the LU decomposition, which in effect records the steps of Gaussian elimination. LU decomposition (factorization) of a nonsingular (square) matrix A means expressing the matrix as the multiplication of a lower triangular matrix L and an upper triangular matrix U , where a lower/ upper triangular matrix is a matrix having no nonzero elements above/below the diagonal. For the case where some row switching operation is needed like in the Gauss elimination, we include a permutation matrix P representing the necessary row switching operation(s) to write the LU decomposition as P A=LU The usage of the permutation matrix P is exemplified by: 0 0 1 a11 a12 a 13 a31 a 32 a33 PA= 1 0 0 a21 a22 a 23 = a11 a 12 a13 0 1 0 a31 a32 a 33 a21 a 22 a23

Method1: To take a close look at the LU decomposition, we consider a 3 3 nonsingular matrix: a 11 a12 a13 1 0 0 u11 u12 u 13 a 21 a22 a23 = l 21 1 0 0 u22 u 23 a 31 a32 a33 l 31 l 32 1 0 0 u 33

a 11 a12 a13 u11 u12 u13 = l 21 u11 l 21 u12u 22 a 21 a22 a23 l 21 u 13u23 a 31 a32 a33 l 31 u11 l 31 u12l 32 u 22 l 31 u13l 32 u23u 33

First, equating the first rows of both sides yields: u1n=a1n , n=1, 2,3 Then, equating the second rows of both sides yields: a 21=l 21 u11 , a22 =l 21 u12 u22 , a 23=l 21 u13u 23 from which we can get: l 21=a21 /u11 ,u22=a21 l 21 u12 , u23=a23 l 21 u13 Now, equating the third rows of both sides yields: a31=l 31 u 11 , a32 =l 31 u 12l 32 u22 , a33 =l 31 u 13 u32 u23u33 from which we can get l 31=a31 /u11 , l 32= a32 l 31 u 12/u22 , u33=a33 l 31 u13 l 32 u 23 From these, we determine the two matrices L and U . Method 2 The main idea of the LU decomposition is to record the steps used in Gaussian elimination on A in the places where the zero is produced. Consider the matrix: 1 2 3 A= 2 5 12 0 2 10 The first step of Gaussian elimination is to subtract 2 times the first row from the second row. In order to record what we have done, we will put the multiplier, 2, into the place it was used to make a zero, i.e. the second row, first column. In order to make it clear that it is a record of the step and not an element of A, we will put it in parentheses. This leads to: 1 2 3 A= 2 1 6 0 2 10 There is already a zero in the lower left corner, so we dont need to eliminate anything there. We record this fact with a (0). To eliminate the third row, second column, we need to subtract 2 times the second row from the third row. Recording the 2 in the spot it was used we have: 1 2 3 2 1 6 0 2 2 Let U be the upper triangular matrix produced, and let L be the lower triangular matrix with the records and ones on the diagonal, i.e.:

1 0 0 L= 2 1 0 , 0 2 1 then we have the following mysterious coincidence:

1 2 3 U = 0 1 6 0 0 2 LU = A

Using LU to solve equations If we also include pivoting, then an LU decomposition for A consists of three matrices P , L and U

such that:

P A=LU To use the information P , L , U to solve a linear system multiplying by the pivot matrix:
P Ax=P b d
Substituting LU for P A we get:

Ax=b we first pivot both sides by

LU x=d Ly=d

Then we need only to solve two back substitution problems: and

U x= y. Example: Find the solution of the system of linear equations whose A and b are given as: 2 3 1 5 A= 4 4 3 , b= 3 2 3 1 1

Solution: After the first Gauss elimination step, (b)=(b)-2(a) and (c)=(c)-(-a) 2 3 1 A= 2 2 1 1 6 2

After the second Gauss elimination step, (c)=(c)-(-3)(a) 2 3 1 A= 2 2 1 1 3 5

From this, we can easily find the matrices L and U 1 0 0 2 3 1 1 0 0 L= 2 U= 0 2 1 and since no row exchanges were done, P= 0 1 0 1 0 and 1 3 1 0 0 5 0 1 1 PA=LU

To solve the system of linear equations: forward substitution Ly=Pb=b y1=5 2y1 y2=3 y2=7 y13y2 y3=1 y3=1521=15 backward substitution Ux= y 5x3=15 x3=3 2x2x3=7 x2=2 2x13x2x3=5 x1=1

4.5 Iterative Methods

4.5.1.

Jacob Method
An algorithm for determining the solutions of a system of linear equations iteratively Given a square system of n linear equations Ax=b where

Then A can be decomposed into a diagonal component D, and the remainder R: A = D + R where

The system of linear equations may be written as DR x=b D xR x=b , and finally, D x=b R x 1 Eqn (1) results in the following equivalent system of linear equations a11 x 1=b 1{a12 x 2a13 x 3a1n x n } a22 x 2=b 2{a21 x 1a23 x 3a2n x n } 2 a nn x n=b n{an1 x 1an2 x 2a 1 {n1 } x n1 } The Jacobi method solves the left hand side of eqn (1) or eqn (2) for x using previous values of x on the right hand side. k 1 1 1 k x =D bD R x Or the element-based formula is given by xik1= 1 b a x k a ii i ij j ji

Algorithm Choose an initial guess x0 to the solution while convergence not reached do for i=1 step until n do =0 for j=1 step until n do if j !=i then =aij x k1 j end if end (j-loop) bi k xi = aii end (i-loop) check if convergence is reached end (while convergence condition not reached loop)

The iteration process results in the vector sequence x 0 , x 1 ,x k . If the matrix A is strictly diagonally dominant, the sequence converges to the exact solution x*. => a ii a ij for each row i
ji

The Jacobi method sometimes converges even if these conditions are not satisfied The stop criterion in the algorithm can be given by: if x k x k1 then x* = x k with accuracy Example 1: Perform 5 iterations using the Jacobi method for the system given below. Work with an intermediate accuracy of five digits after the decimal comma, and for initial guess choose the zero vector x 0=0,0,0 . 4 x1 x2 =2 x1 + 4 x2 x3 = 6 x2 + 4 x3 = 2 Solution 4 1 0 A= 1 4 1 0 1 4

2 b= 6 2 0 x 0 = 0 0

Formula: x k 1 =D1 bD1 R x k 1 1 D b=d and D R=C 0.5 0 0.25 0 d = 1.5 and C= 0.25 0 0.25 0.5 0 0.25 0 k 1 k x =d C x Convergence check:

3 <4 and 2 < 4 and 3 < 4 => The Jacobi method will converge Iteration 1 0 x =dC x 0.5 0 0.25 0 0 0.5 x 1= 1.5 0.25 0 0.25 0 = 1.5 0.5 0 0.25 0 0 0.5

x 2 =d C x 1 0.5 0 0.25 0 0.5 0.875 2 x = 1.5 0.25 1.5 = 1.75 0 0.25 0.5 0 0.25 0 0.5 0.875

x 3 =d C x 2 0.5 0 0.25 0 0.875 0.9375 3 x = 1.5 0.25 0 0.25 1.75 = 1.9375 0.5 0 0.25 0 0.875 0.9375

x 4 =dC x 3 0.5 0 0.25 0 0.9375 0.98438 4 x = 1.5 0.25 0 0.25 1.9375 = 1.96875 0.5 0 0.25 0 0.9375 0.98438
5

x =dC x 0.5 0 0.25 0 0.98438 0.99219 x 5= 1.5 0.25 0 0.25 1.96875 = 1.99219 0.5 0 0.25 0 0.98438 0.99219 At the 13th iteration, 1.00000 x 13=2.00000 1.00000

A x=b

Home Work: 1. Use Jacobi method to find the solution of a linear system of the form with initial x 0 given by A= 2 1 , b= 11 and x 0= 1 5 7 13 1 The solution after 25 iterations is: x= 7.111 3.222

2. Use the Jacobi method to solve the following system of linear equations 7x 13x 2x 3=18 2.85997 6 2 x 1 9x 24 x 3=12 Answer: At x = 0.68261 x 1 4 x 212 x 3 =6 0.03365

4.5.2.

Gauss-Seidel Method
An iterative method used to solve systems of linear equations Similar to the Jacobi method Given a square system of n linear equations A x=b where A , x and b are given as in the Jacobi method A can be decomposed into a lower triangular component L and a strictly upper triangular component U A= LU where

The system of linear equations may be rewritten as: L x=bU x 3 Eqn (3) results in the following equivalent system of linear equations a11 x1 =b1{a12 x 2a13 x 3a 1n x n } a 21 x 1a22 x 2=b2{a23 x3 a24 x 4a 2n x n } a n1 x 1an2 x 2a n3 x 3ann x n=bn The GaussSeidel method, like the Jacobi method, solves the left hand side of this expression for x, using previous value for x on the right hand side. k 1 1 k x =L bU x However, by taking advantage of the triangular form of L , the elements x k 1 can be computed sequentially using forward substitution. The element-based formula is given by x k1= i 1 b i a ij xjk aij x jk1 ,i=1, 2,n aii ji ji

Algorithm Inputs: A , b Output: x Choose an initial guess x 0 repeat until convergence to the solution

for i from 1 until n do =0 for j from 1 until i 1 do =aij x k1 j end (j-loop)

for j from i + 1 until n do =aij x k j end (j-loop) 1 k1 x i = bi a ii end (i-loop) check if convergence is reached end (repeat)

The iteration process results in the vector sequence x 0 , x 1 ,x k . If the matrix A is strictly diagonally dominant, the sequence converges to the exact solution x*. => a ii a ij for each row i
ji

The GaussSeidel method sometimes converges even if these conditions are not satisfied. The stop criterion in the algorithm can be given by: if x k x k1 then x* = x k with accuracy Example 1 A linear system shown as A x=b is given by: 3 11 A= 16 and b= 7 11 13 0 1 Use an initial estimate x = to solve for x using the Gauss-Seidel method. 1 Solution: k 1 k x =dC x where C=L1 U and d =L1 b 16 0 0 0 L= , U= 7 11 0 3

T = 0 0.1875 , C=0.6875 0.7443 0 0.1193 Convergence check 16 > 3 and 11 > 7 => The iterative process will converge to the solution Iterations

The iteration converges to

x=

0.8122 0.6650

Homework 1. A system of linear equations shown as A x=b is given by 2 3 11 A= and b= 5 7 13 0 1.1 use x = to find the solution of the system using the Gauss-Seidel method. 2.3 [check for convergence] 2. Solve the following system of linear equations using the Gauss-Seidel method 7x 13x 2x 3=18 2 x 1 9x 24 x 3=12 x 1 4 x 212 x 3 =6 Answer: 2.85957 4 At x = 0.68273 0.03412