www.numerical-methods.

com

LU Factorisation and the solution of linear systems of equations

LU factorization or decomposition involves a method for writing a matrix as the
product of a lower-triangular matrix (denoted L) and an upper-triangular matrix
(U)
1
. LU factorization is useful in solving problems such as linear systems of
equations or matrix-vector systems
2
, generally of the form



where is a matrix and and are vectors. Through the determination of the
LU factors of , we can write , and hence that



The solutions of the latter equation can now be realised in two stages, both of
which are a straightforward sequence of substitutions. Firstly a solution is
found to the system and then the sought solution is found from solving
.

The method of solving systems of the form through LU factorisation has
parallels with the alternative method of Gaussian elimination
3
. From the point of
view of a direct computational comparison, there is little to choose between the
two methods. However LU factorisation has a significant strategic advantage in
that the bulk of the computation occurs in the determination of the factors L and
U and hence, once they are determined, a range of vectors can be applied,
whereas in Gaussian eliminations the (or set of s) is specified before the
processing takes place.

In most cases LU factorisation is applied to square matrices and this will be the
focus in this document. An Excel spreadsheet
4
demonstrating LU factorisation
and back and forward substitution has been developed along with a guide to
using the spreadsheet. The spreadsheet illustrates the methods on 33, 55 and
1010 systems. The spreadsheet includes two subroutines in the visual basic
(for applications) language: LUfac.bas
5
for finding the LU factors of a matrix and
LUbfsb.bas
6
for carrying out the back and forward substitution.

LU factorisation of a 2x2 Matrix

In order to understand, illustrate the method, let is consider applying it to an
example of a 22 system:

(


) (

) (

)

1
Matrix Definitions
2
Linear systems and 2x2 matrices
3
Gaussian Elimination
4
LU.xlsm
5
LUfac.bas
6
LUfbsub.bas
www.numerical-methods.com

The matrix in this system is (


) or (

) (


).

The LU factorisation requires that the matrix is written as the product of a
lower and an upper triangular matrix;

(

) (

) (

)

Matrix multiplication
7
is then applied in order to derive equations relating the
values of

For a 22 matrix, there are four equations and six unknowns. In general there
are n more unknowns than there are equations in the general LU factorisation,
where n is the dimension of the matrix. In order to secure a unique solution,
therefore, n values are pre-determined and it is the convention that the diagonals
or the lower-triangular matrix L are all set to a value of 1. Hence for a 22
system
(

) (

) (

)

and

For the example

(

) (


)

the equations above give

and

directly and then

and

by substitution. This gives the following LU factorisation

7
Matrix Arithmetic
www.numerical-methods.com

(


) (


) (


)

Returning to the equation that we set out to solve:

(


) (

) (

)

this may be written in the following form

(


) (


) (

) (

)

by replacing A by its LU factors.

This latter equation can be solved in two steps, using the method of forward
substitution, followed by back substitution.

Forward Substitution

Forward substitution is applied in order to solve

(


) (

) (

)

In forward substitution the elements of the unknown vector are found in
sequence by following the matrix equation row by row.

The top row of the matrix multiplied by the vector (

) is as follows:

Similarly the second row gives the following:

Backsubstitution

Backsubstitution is applied in order to solve the remaining problem

(


) (

) (

)

In backward substitution the elements of the unknown vector are found in
reverse order:

www.numerical-methods.com

LU factorisation of a 3x3 Matrix

In order to understand, illustrate and refine the method, let is consider applying
it to an example of a 33 system:

(



)(

) (

)

The matrix in this system is (



) or (

) (



).

Let us return to the LU factorisation of a general 3x3 matrix:

(

) (

)(

)

At the beginning of the process the unknown elements of L and U are greyed-out
in the equation above. The LU factorisation method must determine these values.

There are nine unknowns and nine equations:

(

)

We first note that the determination of the top row of U is straightforward.

U row 1

(

) (

)(

)

In the equation above the blue elements are the ones that are evaluated at this
stage and the yellow elements are the elements that are used. This leads to the
following assignments.

www.numerical-methods.com

L - column 1

Once

is evaluated,

can be evaluated straightforwardly

(

) (

)(

)

Giving the equations:

That is

U row 2

(

) (

)(

L - column 2

(

) (

)(

U - row 3
(

) (

)(

www.numerical-methods.com

Example
For the matrix in the example, (



), the application of the method
outlined above gives the following results.

U row 1

L - column 1

U row 2

L - column 2

U - row 3

Hence the original matrix can be factorised as follows:

(



)(



) (



)

Substituting the LU factorisation into the original matrix-vector problem gives

(



)(



)(

) (

)

As with the 2x2 example, forward substitution is used to solve

(



)(

) (

)
www.numerical-methods.com

giving

Backsubstitution is then used to solve

(



)(

) (

)

yielding the solution

,

The LU factorisation algorithm

The LU factorisation method described above evaluates the elements of the L and
U matrices by first calculating the first row of the U matrix, then the first sub-
diagonal row of the L matrix, then the second rows diagonal and super-diagonal
element of the U matrix and then the second row of sub-diagonal elements of the
U matrix.

Each of the expressions of

have only the corresponding element of the

matrix A (

, the remaining terms are evaluated elements of L and U. This is a

useful practical property of LU factorisation as a computational method; the L
and U matrix elements can successively overwrite the original matrix A, L
forming its lower triangle (excluding the diagonal, since the diagonal values of L
are predefined) and U forming its upper triangle (including the diagonal). The
over-writing of the matrix A by the elements of L and U saves computer memory
and is standard within most implementations of the LU factorisation algorithm.

For example in the LU factorisation of (



), the L and U matrices
(



) (



) can be stored separately and separate to the
matrix A. However, the upper triangular elements of L and the lower triangular
elements of U are pre-defined and hence it wastes computer memory to store
them as two separate matrices and they will be normally be stored as
(



), and again to save computer memory, that this matrix would
over-write the original matrix A.

The LU factorisation method discussed so far can be written in the form of the
following pseudocode, which is also known as Crouts Algorithm. The following
algorithm is implemented in visual basic with the subrouting LUfac.bas
8
within
the module LUfac_module in the LU.xlsm
9
Excel spreadsheet.





8
LUFac.bas
9
LU.xlsm
www.numerical-methods.com
















The LU factorisation algorithm involves

flops and hence is

10
.

However, in the terms of a robust numerical algorithm, the method is not
complete. We have not dealt with the possibility of a singular matrix A. Also the
last line of the algorithm contains a division by a(i,i), which could take the value
of zero (or close to zero) even when the original matrix is non-singular (and
well-conditioned), causing the algorithm to fail. The algorithm requires
improvement, if it is to be useful as a general purpose method for solving
systems of equations, and, as with Gaussian elimination a pivoting method is
normally included, and this is considered in the next section.

Pivoting

Pivoting is included in the LU factorisation method so that division by zero or a
small number is avoided. The LU factorisation algorithm can be viewed as
passing through the diagonal elements and setting the row of U followed by the
column on L in turn. The worry with the LU factorisation method above is in the
line a(i,j)=(a(i,j)-sum)/a(i,i); if a(i,i) is zero or small then the method will fail or will
not sustain numerical confidence.

For example the following matrix is non-singular, but it would fail very early into
the application of the method outlined above because of the zero in the first row
and first column:

(



)

The inclusion of (partial) pivoting involves exchanging rows. The exchange of rows
has to be recorded and this is equivalent to maintaining a permutation matrix
11
.
Whereas the LU factorisation (without pivoting) of a matrix is unique (if it exists), the

10
Big O notation in computing
11
Permutation Matrix
The LU factorisation method; Crouts Algorithm
' The main loop is a counter effectively down the diagonal
for i=1 to n
' The first inner loop counts j=i..n, so that a(i,j) addresses the diagonal and super-
' diagonal elements of a on row i. That is ith row of the U matrix is formed.
for j=i to n
sum=0
for k=1 to i-1
sum=sum+a(i,k)*a(k,j)
a(i,j)=a(i,j)-sum
' The second inner loop counts j=i..n, so that a(j,i) addresses the sub-
' diagonal elements of A on column i. That is the ith column of the L matrix is formed.
for j=i+1 to n
sum=0
for k=1 to i-1
sum=sum+a(j,k)*a(k,i)
a(i,j)=(a(i,j)-sum)/a(i,i)

www.numerical-methods.com

inclusion of pivoting has the result of the outcome of the factorisation method
depending on the pivoting method.

In (partial) pivoting method, the divisor from each row is calculated. Exchanging the
row with the largest pivot for the superdiagonal row that is to eliminated is the most
straightforward form of pivoting. However, in the case when one row is made up of a
set of element that are all smaller in magnitude then this may cause an unnecessary
and actually counter-productive exchange of rows. For example with the matrix

(


)

the initial temptation would be to exchange rows since 1>>0.2, but this would be
unnecessary and on balance unhelpful. Hence it is proposed that the row-norm is also
taken into account. The following pseudo-code relates such a pivoting method.



























Through applying a (partial) pivoting method within the LU factorisation
algorithm, the result is generalised so that the matrix A is factorised as follows;



with a permutation matrix P included.

The Pivoting Method
' Pass down from the diagonal to check which row would have the
' largest divisor.
uiicolmax = 0#
for j = i To n
sum = 0
for k = 1 To i - 1
sum = sum + a(j, k) * a(k, i)
reldivisor = (Abs(a(j, i) - sum)) / rownorms(perm(i))
if (reldivisor > uiicolmax) Then
uiicolmax = reldivisor
imax = j

' If the largest divisor is zero or very small then the matrix is
' singular or ill-conditioned and the factorisation is abandoned
If (uiicolmax < Tiny) Then
MsgBox ("Matrix is Singular or very Ill-conditioned. LU factorisation is abandoned")
Exit Sub

' The rows are swapped and the exchange is recorded in perm
if (i <> imax) Then
for k = 1 to n
dum = a(imax, k)
a(imax, k) = a(i, k)
a(i, k) = dum
permi = perm(i)
perm(i) = perm(imax)
perm(imax) = permi

www.numerical-methods.com

The pivoting method above is implemented in visual basic with the subroutine
LUfac.bas
12
within the module LUfac_module in the LU.xltm
13
Excel spreadsheet.

Forward and Back Substitution Algorithm

The forward and back substitution algorithms were introduced earlier in this
document. Returning to the original matrix-vector equation



where is a matrix and and are vectors. Through the determination of the
LU factors of , with pivoting, we can write , and hence that
Using the elementary property of permutation matrices
14
(P
-1
=P
T
) and hence we
may write

and hence there is effectively a rearrangement of the

elements of b prior to the forward and back substitution, which was outlined
earlier. The pseudo-code for the overall method is given below.




















For a typical implementation of the forward and back substitution method, see
LUfbsub.bas
15
, and its demonstration on the Excel Spreadsheet LU.xlsm
13

12
LUfac.bas
13
LU.xlsm
14
Permutation Matrix
15
LUfbsub_bas
The forward and back substitution methods
Initialise pb as b with rows exchanged as indicated by perm
for i = 1 to n
pb(i) = b(perm(i))

Solve Ly=b by forward substitution (pb stores y on completion)
for i = 1 to n
sum = pb(i)
for j = 1 to i - 1
sum = sum - a(i, j) * pb(j)
pb(i) = sum

Solve Ux=y by back substitution (pb stores the solution 'x' on completion)
for i = n to 1 step -1
sum = pb(i)
for j = i + 1 to n
sum = sum - a(i, j) * pb(j)
pb(i) = sum / a(i, i)

b is over-written by pb, the solution 'x'.
for i = 1 to n
b(i) = pb(i)