Ax B A Linear Equation: MN MN MN MN

Systems of Linear Equations
n
1. Solve a x
i 1
i i b a linear equation
How do we solve a set of them?
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1

a21 x1  a22 x2  a23 x3  ...  a2n xn  b2
... ............................................ = ...
am1 x1  am2 x2  am3 x3  ...  amn xn  bm
This is an m  n system. If m  n , we will not have

unique solutions. If m  n , we may get into a
situation inconsistency. If m  n , we may get unique
set of solutions.
Some observation:
Suppose, the equation is L0 : 2x  3 y  5

Nothing is changed if we multiply or divide this
equation by a constant.
Therefore, k  L0 can be used to replace L0 . We
denote this as kL0  L0
Given two linear equations Li and L j , we can form

any combination of these two to replace, say, L j .
Thus,
kLi  L j  L j
Interchanging the positions of two linear equations

does not change anything.
Li  L j
These operations that leave the system of equations

invariant are called elementary operations.
2. Consider solution of linear equations.
a. One unknown, one solution. If we have ax  b,

b
we have a solution x  , if a  0
a
b. Two unknown, two solutions. Assume,
2x  3 y  5
we have here x  1, y  1
3x  y  2
If one equation is the multiple of another equation,
we have infinite solutions. e.g.
2x  3 y  5
4x  6 y  10
Geometrically, the two lines are parallel.
a11 x1  a12 x2  b1
The equations have unique
a21 x1  a22 x2  b2
solutions if the determinant of this equation
a11 a12
| A|  a11a22  a12 a21  0
a21 a22
The determinant of a 3  3 matrix could also be

computed in a similar manner. For instance,
a11 a12 a13

a a23 a a23
| A| a21 a22 a23  a11  22  a12  21
a32 a33 a31 a33
a31 a32 a33
a21 a22
 a13 
a31 a32
The determinant | A| could be expressed as an
n n
expansion | A|  aij Aij   ( 1 ) aij M ij

i j
j 1 j 1
Where M ij  min or Aij  cofactor
A minor M ij is obtained from the original matrix the

resultant submatrix obtained by leaving ith row and
jth column behind.
3. Systems in triangular and echelon form:
An equation is in a triangular form if it can be

expressed, via appropriate elementary operations, like
below:
2x1  x2  3x3  2x4  4

5x2  6 x3  x4  0
2x3  3x4  1
2x4  2
This is easy to solve.
A set of equations is in echelon form if (a) no

equation is degenerate, (b) the leading term in an
equation begins at the right of the leading term of the
previous equation. The leading unknowns are called
pivot element.
Theorem. Suppose a set of m equations is in echelon
form with n unknowns. If m  n , we have a unique
set of solutions. If m  n , we can assign arbitrary
values (parameters) for ( n  m ) variables, and solve
the remaining m variables in terms of these
parameters.
See parametric form/free-variable form. Boils down

to the same thing.
Gaussian elimination method:
Part 1: (Forward elimination) Using elementary

operations, reduce the system of equation to either a
triangular form or an echelon form.
Part2: (Backward elimination) Step-by-step back

substitution yields now the solution of the system.
Show an example.
An echelon matrix must have all zero rows at the

bottom and it must obey the basic format of an
echelon as far as the position of the pivots is
concerned.
This is an echelon matrix.

0 2 2 1
0 0 3 8
  The pivots are underlined.
0 0 0 4
0 0 0 0 

The rank of a matrix is the number of pivots in its

echelon form.
A matrix is A is in row-canonical form if (a) it's an

echelon matrix, (b) each pivot (leading to non-zero
entry) is 1, (c) each pivot is the only nonzero entry in
that column.
0 1 4 0 0 4 
 
e.g. 0 0 0 1 0 2  is in row-canonical form.
0 0 0 0 1 3 
Row equivalence of a matrix. A matrix A  B (row

equivalent) if B is obtained from A by elementary
operations.
To set up the Gaussian elimination, work on the

augmented matrix  A b  and reduce A to a triangular
matrix (or to an echelon form) by elementary
operations. Then solve by back substitution.
Use Gaussian Elimination to solve
L1 : 2x1  x2  x3  4
L2 :  x1  2x2  3x3  6
L3 : x1  x2  x3  1
The augmented matrix is
 2 1 1 4   2 1 1 4 
 1 2 3 6  0 1 4 5 
   
 1 1 1 1  0 0 7 7 
back substitution now leads to
x3  1,x2  1,x1  1
Note that the O( GE )  n 3 This is prohibitively large

for large n.
Secondly, we have to be careful about error. Consider

the following.
0.003x1  59.14x2  59.17

5.291x1  6.130x2  46.78
with the exact solution x1  10, x2  1
Numerical solution using a 4 digit arithmetic gives us
0.003x1  59.14x2  59.17

-104300x2  104400
x2  1.001 (tolerable),
this gives us
x1  10.00 (outrageous)_
We have to come up with a pivoting strategy. Choose

the element in column k such that | a pk | max
k i  n
|aik |
This is called "partial pivoting".
For instance, in our case, | a21 | max

1i  2
|ai1 |
We now swap the rows so that the row with this pivot
becomes the topmost row.
5.291x1  6.130x2  46.78

Thus,
0.003x1  59.14x2  59.17
With this arrangement, the GE leads to
5.291x1  6.130x2  46.78

59.14x2  59.14
This now gives the correct solution
x1  1.000 x2  10.00
Failure of partial pivoting. If we scale equation 1 by
10 4 , then we get
30.00x1  591400x2  591700

5.29x1  6.130x2  46.78
partial pivoting will choose row1, which then again

leads to
x2  1.001 and x1  10.00
Remedy: Scaled partial pivoting
step1: design for each row a scale-factor

si  max |aij |
j 1..n
| ak 1 | |a j1 |
step 2: Choose k such that  max
sk j 1..n sj
s1  max  | 30.00 |,| 591400 |  591400

In our case,
s2  max  | 5.291|,| 6.1301|  6.1301
| a11 | 30.00
  0.507  10 4
s1 591400
| a21 | 5.291
  0.8631
s2 6.130
Therefore, we choose the a21 as the pivot, swap row1
and row2 and continue with GE.
Note. (a) This can get expensive if you recompute si

at each step, so usually they are not recomputed.
(b) Full pivoting is expensive – search all n  n

max | aij |
elements to find
si
An example. Naïve Gaussian Elimination may fail.
Scaled partial pivoting. (Cheney and Kincaid, “Numerical mathematics

and Computing”, pp 234, Brooks/Cole Publishing)
Augmented equation.
 3  13 9 3  19 
 6 4 1  18  34
 
 6 2 2 4 16 
 12 8 6 10 26 

The last column of the matrix is augmented to the

original matrix.
The index vector and the scale vector at the first
pivoting:
l  1 2 3 4
s  13 18 6 12 si  max j {aij }
| ali ,1 |   3 6 6 12 
Compute now 
 sli
i  4  , , , 
 13 18 6 12 
Choose the index for which the first maximum is

found. In our case, it is 3. Therefore, the first pivot is
going to be on the first element of row 3. Exchange
l1  l3 in the vector l . Now the augmented matrix
R
appears as (after R1  R1  23 , R2  R2  R3 ,
R4  R4  2R3 )
0  12 8 1  27 
0 2 3  14  18 
A 
6 2 2 4 16 
0 4 2 2  6 

Now we are to select the pivot number 2 given that

the vectors are now
l  3 2 1 4 and s  13 18 6 12
obtain the pivot row index from

| ali , 2   2 12 4 
 i  2, 3, 4   , , 
 sli  18 13 12 
The largest value corresponds to index 3 (which

12
yields 13 ) , so we exchange 2nd and the 3rd elements of
the vector l realizing l   3 1 2 4 . The pivot in 2nd
step is in Row 1 now. In the augmented matrix,
Row 2 and Row 4 under column 2 now become zero.
R R
This is done by R2  61  R2 and R4  R4  31 .
0  12 8 1  27 
0 0 13 / 3  83 / 6  45 / 2
A 
6 2 2 4 16 
0 0 2/3 5/3 3 

We select the final pivot now, pivot 3. The vectors

are now l   3 1 2 4 and s  13 18 6 12 .
We now obtain the pivot row index from
| ali ,3  13 / 3 2 / 3

 i  3, 4  , 
 sli   18 12 
Since the first number is bigger and that corresponds

to 3, therefore we have now l3  l3 leaving the index
vector as before. The pivot equation is l3  2 . So, our
pivot is on the 2nd row. Now the augmented matrix
takes the following shape:
0  12 8 1  27 
0 0 13 / 3  83 / 6  45 / 2
A 
6 2 2 4 16 
0 0 0  6 / 13  6 / 13 

The rest is easy. This gives us through back

substitution the solution vector x  (3 1  2 1) .
t
More on numerical computation of solution

matrix.
Error computation on an iterative framework. Let

x( k ) be the solution vector of the equation Ax  b on
kth iteration. The difference between this and the
actual is the error vector e( k )  x  x( k ) and the
residual on the kth iteration is
r ( k )  b  Ax( k )  Ax  Ax( k )  Ae( k )
Therefore, if we solve this error equation Ae( k )  r ( k )

in double precision, the better solution at the next
( k  1 ) iteration level comes out to be
x( k 1 )  x( k )  e( k ) . We continue with the process till
a convergence is reached.
Convergence criterion: stop when

|| x( l 1 )  x( l ) || some 
4. General Iterative techniques.

Gauss-Jacobi's method. Ideal for large sparse
matrices. To solve for Ax  b convert this into an
iterative form x  Tx  c , and solve it iteratively.
For instance, suppose the system is
2x  3 y  z  8
x  y  2z  1
x  y  3z  6
its exact solution is the vector x  ( 2 1 -1)t
can be expressed in an iterative format as
x( k )  4  1.5 y( k 1 )  0.5z ( k 1 )
y ( k )  1  x( k 1 )  2z ( k 1 )
z ( k )  2  0.3333x( k 1 )  0.3333 y( k 1 )
Start with a trial solution x( 0 )  ( 1.0 0.5 0.5 )t and

obtain the next level solution.
This refers to the generalization of the iterative

solution scheme  x  starting with x( 0 ) getting
k
 
1
xi( k 1 )  bi   aij x(j k ) 
aii  j 1

 j i 
We can write a matrix A as
A  D  L U
D  Diagonal matrix
L  Lower Triangular
U=Upper Triangular
e.g.
 2 5 3   2 0 0   0 0 0  0 5 3 
 1 6 4   0 6 0    1 0 0   0 0 4 
       
 3 2 1  0 0 1  3 2 0  0 0 0 
Given this Ax  b becomes
( D  L  U )x  b or Dx=(L+U)x+b
or x=D -1 ( L  U )x  D 1b
This is Gauss-Jacobi. Given the diagonal matrix D,

it’s easy to compute D 1 . Multiplying matrices is an
easier than taking an inverse.
Comparing this with x  Tx  c we see Gauss-Jacobi's

approach amounts to using
T  D 1 ( L  U ) and c=D -1b
e.g. The equation matrix is
 1 1 1 0 
   
A  A   1 3 0  b=  2  with this
 1 0 2   -3 
A  D  L  U , and here
1 0 0   0 -1 1 
D  0 3 0  L+U=  1 0 0 
   
0 0 2  -1 0 0 
1 0 0 
1  0 
Now, D  0 0.3333 
0 0 0.5 
Therefore,
1 0 0   0 1 1
D 1 ( L  U )  0 0.3333 0   1 0 0
  
0 0 0.5   1 0 0 
 0 1 1 
 
= 0.3333 0 0 
 0.5 0 0 
1 0 0  0   0 
1  0   2   0.6666 
and D b  0 0.3333    
0 0 0.5   3   1.5 
Therefore, our iterative problem appears to be
 0 1 1   0 
x ( k 1 )  0.3333 0 0  x( k )  0.6666 
   
 0.5 0 0   1.5 
We start with x( 0 )  ( 0 0 0)t . The iteration

progressed through the following:
0 0.666667 1.5
0.833333 0.944444 1.91667
0.972222 0.990741 1.98611
0.99537 0.998457 1.99769
0.999228 0.999743 1.99961
0.999871 0.999957 1.99994
0.999979 0.999993 1.99999
0.999996 0.999999 2
0.999999 1 2
1 1 2
Gauss-Seidel's Iteration Method:
From above, we can write
( D  L )x  Ux  b or x=(D-L)-1Ux  ( D  L )1 b .
Thus, the iterative scheme attempts to get
x( k 1 )  ( D  L )1Ux( k )  ( D  L )1 b
In computational format, the improved solution x( k 1 )

is obtained in the following way:
1  i 1 n
k
bi   aij x j   aij x j 
( k 1 ) ( k 1 )
xi 
aii  j 1 j i 1 
In other words, those next level of computed

variables that are already available could be used
immediately when situation arises. Apparently, this
converges twice as fast as Gauss-Jacobi.
Convergence analysis. Like successive substitution

Define error
e( k )  x  x( k )  Tx  c  (Tx( k 1 )  c )  T( x  x( k 1 ) )
 Te( k 1 )
Thus, the magnitude of the error

|| e( k ) ||||Te( k 1 ) ||||T || || e( k 1 ) ||
This will be reduced only if ||T || 1.
We’ll come return to more computational aspects

after we develop the topic further.

Ax B A Linear Equation: MN MN MN MN

Uploaded by

Copyright:

Available Formats

Ax B A Linear Equation: MN MN MN MN

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ax B A Linear Equation: MN MN MN MN

Uploaded by

Copyright:

Available Formats

Systems of Linear Equations

How do we solve a set of them?

a11 x1  a12 x2  a13 x3  ...  a1n xn  b1

This is an m  n system. If m  n , we will not have

Suppose, the equation is L0 : 2x  3 y  5

Given two linear equations Li and L j , we can form

Interchanging the positions of two linear equations

These operations that leave the system of equations

2. Consider solution of linear equations.

a. One unknown, one solution. If we have ax  b,

Geometrically, the two lines are parallel.

solutions if the determinant of this equation

The determinant of a 3  3 matrix could also be

a11 a12 a13

expansion | A|  aij Aij   ( 1 ) aij M ij

Where M ij  min or Aij  cofactor

A minor M ij is obtained from the original matrix the

3. Systems in triangular and echelon form:

An equation is in a triangular form if it can be

2x1  x2  3x3  2x4  4

A set of equations is in echelon form if (a) no

See parametric form/free-variable form. Boils down

Gaussian elimination method:

Part 1: (Forward elimination) Using elementary

Part2: (Backward elimination) Step-by-step back

An echelon matrix must have all zero rows at the

This is an echelon matrix.

The rank of a matrix is the number of pivots in its

A matrix is A is in row-canonical form if (a) it's an

Row equivalence of a matrix. A matrix A  B (row

To set up the Gaussian elimination, work on the

back substitution now leads to

Note that the O( GE )  n 3 This is prohibitively large

Secondly, we have to be careful about error. Consider

0.003x1  59.14x2  59.17

Numerical solution using a 4 digit arithmetic gives us

0.003x1  59.14x2  59.17

We have to come up with a pivoting strategy. Choose

For instance, in our case, | a21 | max

5.291x1  6.130x2  46.78

With this arrangement, the GE leads to

5.291x1  6.130x2  46.78

30.00x1  591400x2  591700

partial pivoting will choose row1, which then again

Remedy: Scaled partial pivoting

step1: design for each row a scale-factor

s1  max  | 30.00 |,| 591400 |  591400

Note. (a) This can get expensive if you recompute si

(b) Full pivoting is expensive – search all n  n

Scaled partial pivoting. (Cheney and Kincaid, “Numerical mathematics

The last column of the matrix is augmented to the

s  13 18 6 12 si  max j {aij }

Choose the index for which the first maximum is

Now we are to select the pivot number 2 given that

l  3 2 1 4 and s  13 18 6 12

obtain the pivot row index from

The largest value corresponds to index 3 (which

We select the final pivot now, pivot 3. The vectors

We now obtain the pivot row index from