MatriA Decomposition and its

Application in Statistics
Part I
Chaiyaporn Khemapatapan, Ph.D
DPU

Overview

Introduction
LU decomposition
QR decomposition
Cholesky decomposition
Jordan Decomposition
Spectral decomposition
Singular value decomposition
Applications
2

Introduction
This Lecture covers relevant matriA decompositions, basic
numerical methods, its computation and some of its applications.
Decompositions provide a numerically stable way to solve
a system of linear equations, as shown already in [Wampler,
1970], and to invert a matriA. Additionally, they provide an
important tool for analyzing the numerical stability of a system.

Some of most frequently used decompositions are the LU, QR,

Cholesky, Jordan, Spectral decomposition and Singular value
decompositions.
3

Easy to solve system (Cont.)

Some linear system that can be easily solved

The solution:

b1 / a11
b /a
2 22

bn / a nn

Easy to solve system (Cont.)

Lower triangular matriA:

Solution: This system is solved using forward substitution

Easy to solve system (Cont.)

Upper Triangular MatriA:

Solution: This system is solved using Backward substitution

LU Decomposition
LU decomposition was originally derived as a decomposition of quadratic
and bilinear forms. Lagrange, in the very first paper in his collected
works( 1759) derives the algorithm we call Gaussian elimination. Later
Turing introduced the LU decomposition of a matriA in 1948 that is used to
solve the system of linear equation.
Let A be a m m with nonsingular square matriA. Then there eAists two
matrices L and U such that, where L is a lower triangular matriA and U is an
upper triangular matriA.
u11 u12
0 u
22
U

0
0

Then,

u1m
u 2 m

and

u mm

A LU

l11
l
L 21

l m1

0
0

l 22

lm2

l mm

J-L Lagrange

A. M. Turing

(1736 1813)

(1912-1954)
7

How to decompose A=LU?

Finding echelon matriA of A
e(A ) = U (upper triangular)
U = Ek E1 A
A = (E1)1 (Ek)1 U
A = LU

6 2 2
A 12 8 6 Now,
3 13 2
0 0 6 2 2
6 2 2 1
0 4 2 2 1 0 12 8 6

0 12 1 1 / 2 0 1 3 13 2
0 0
6 2 2 1 0 0 1

0 4 2 0 1 0 2 1 0
0 0 5 0 3 1 1 / 2 0 1

E2

E1

6 2 2
12 8 6

3 13 2

If each such elementary matriA Ei is a lower triangular matrices,

it can be proved that (E1)1, , (Ek)1 are lower triangular, and
(E1)1 (Ek)1 is a lower triangular matriA.
Let L=(E1)1 (Ek)1 then A=LU.
8

Calculation of L and U (cont.)

6 2 2
A 12 8 6
3 13 2

1
= 0
0

0
1
0

0
0
1

6
12

2
8
13

2
6
2

Now reducing the first column we have

6
0

2 2
0 0
1
4 2 2 1 0

12 1
1 / 2 0 1

6
12

2 2
8 6

13 2

Then reducing the matriA to upper triangle matriA

6
0

2
4
0

2
1
2 0

5
0

0 0
1 0

3 1

1
0 0
2 1 0

1 / 2 0 1

6
12

2 2
8 6

13 2
9

Calculation of L and U (cont.)

Now

1
0 0
2 1 0
1 / 2 0 1

1 0 0
0 1 0

0 3 1

1 0 0 1 0 0 1 0 0
2 1 0 0 1 0 2 1 0
1 / 2 0 1 0 3 1 1 / 2 3 1

Therefore,
6 2 2
A 12 8 6
3 13 2

1 0 0
2 1 0

1 / 2 3 1

6 2 2
0 4 2

0 0 5

= LU

If A is a Non singular matriA then for each L (lower triangular matriA) the upper
triangular matriA is unique but an LU decomposition is not unique. There can be
more than one such LU decomposition for a matriA. Such as
6 2 2
A 12 8 6
3 13 2

6 0 0
12 1 0

3 3 1

1 2 / 6 2 / 6
0 4
2

0
0
5

= LU

10

Calculation
U (cont.)
(cont.)
Calculationof
of LL and
and U
Thus LU decomposition is not unique. Since we compute LU
decomposition by elementary transformation so if we change
L then U will be changed such that A=LU
To find out the unique LU decomposition, it is necessary to
put some restriction on L and U matrices. For eAample, we
can require the lower triangular matriA L to be a unit one (i.e.
set all the entries of its main diagonal to ones).
LU Decomposition in MATLAB:
Fuunction lu in MATLAB is LU facotrization not decomposition. So, find
out from internet
11

Function LU Decomposition in MATLAB

function [ L, U, P ] = LUdecomposition(A)
% Ensures A is n by n
sz = size(A);
if sz(1)~=sz(2)
fprintf('A is not n by n\n');
clear A;
return;
end
n = sz(1);

L = eye(n);

P = eye(n);

U = A;

for i=1:sz(1)
% Row reducing
if U(i,i)==0
maAimum = maA(abs(U(i:end,1)));
for k=1:n
if maAimum == abs(U(k,i))
temp = U(1,:);
U(1,:) = U(k,:);
U(k,:) = temp;
temp = P(:,1);
P(1,:) = P(k,:);
P(k,:) = temp;
end
end
end
if U(i,i)~=1
temp = eye(n);
temp(i,i)=U(i,i);
L = L * temp;
U(i,:) = U(i,:)/U(i,i); %Ensures the pivots are 1.
end
if i~=sz(1)
for j=i+1:length(U)
temp = eye(n);
temp(j,i) = U(j,i);
L = L * temp;
U(j,:) = U(j,:)-U(j,i)*U(i,:);
end
end

end

end
P = P';

12

Calculation of L and U (cont.)

Note: there are also generalizations of LU to non-square and singular

matrices, such as rank revealing LU factorization.
[Pan, C.T. (2000). On the eAistence and computation of rank revealing LU
factorizations. Linear Algebra and its Applications, 316: 199-222.
Miranian, L. and Gu, M. (2003). Strong rank revealing LU factorizations.
Linear Algebra and its Applications, 367: 1-16.]

Uses: The LU decomposition is most commonly used in the solution of

systems of simultaneous linear equations. We can also find determinant
easily by using LU decomposition (Product of the diagonal element of
upper and lower triangular matriA).

13

Solving system of linear equation

using LU decomposition
Suppose we would like to solve a mm system AA = b. Then we can find
a LU-decomposition for A, then to solve AA =b, it is enough to solve the
systems

Thus the system LY = b can be solved by the method of forward

substitution and the system UA = Y can be solved by the method of
backward substitution. To illustrate, we give some eAamples
Consider the given system AA = b, where

6 2 2
A 12 8 6
3 13 2

and

8
b 14

17

6 2 2 x1 8
AX b 12 8 6 x2 14

3 13 2 x3 17
14

Solving system of linear equation

using LU decomposition
We have seen A = LU, where
1 0 0
L 2 1 0
1 / 2 3 1

6 2 2
U 0 4 2
0 0 5

Thus, to solve AA = b, we first solve LY = b by forward substitution

1
2

0 0 y1
8
1 0 y 2 14
1 / 2 3 1 y 3
17

Then

y1
8
Y y 2 2
y 3
15

15

Solving system of linear equation

using LU decomposition
Now, we solve UA =Y by backward substitution
6 2
0 4

2
2

x1 8
x 2
2

5 x3 15

then

x1
1
x 2
2

x3
3

16

QR Decomposition
Orthogonal-triangular decomposition

Jrgen Pedersen Gram

(1850 1916)

Firstly QR decomposition
originated with Gram(1883).
Later Erhard Schmidt (1907)
proved the QR Decomposition
Theorem

Erhard Schmidt
(1876-1959)

If A is a mn matriA with linearly independent columns, then A can be

decomposed as ,
A QR
where Q is a mn matriA whose columns form an orthonormal basis for the
column space of A. Thus, Q is an orthogonal matriA (its columns are
orthogonal unit vectors meaning QTQ = I)
R is an nonsingular upper triangular matriA (right triangular matriA).
17

QR-Decomposition (Cont.)
Proof: Suppose A=[u1 | u2| . . . | un] and rank (A) = n.
Apply the Gram-Schmidt process to {u1, u2 , . . . ,un} and the
orthogonal vectors v1, v2 , . . . ,vn are

vi ui

vi
qi
vi

Let
orthonormal

ui , v1
v1

v1

ui , v2
v2

v2

ui , vi 1
vi 1

vi 1

for i=1,2,. . ., n. Thus q1, q2 , . . . ,qn form a

basis for the column space of A.

Where inner product u, v u v

T

or in case of hermitian

u, v u v
H

18

QR-Decomposition (Cont.)
u i vi

Now,

u i , v1
v1

v1

ui , v2
v2

v2

u i , vi 1
vi 1

vi 1

u i vi qi u i , q1 q1 u i , q 2 q 2 u i , qi 1 qi 1
i.e.,

u i span{v1 , v 2 , , vi } span{qi , q 2 , qi }

Thus ui is orthogonal to qj for j>i;

u1 v1 q1
u 2 v 2 q 2 u 2 , q1 q1
u 3 v3 q3 u 3 , q1 q1 u 3 , q 2 q 2

u n v n q n u n , q1 q1 u n , q 2 q 2 u n , q n 1 q n 1

19

QR-Decomposition (Cont.)
Let Q= [q1 q2 . . . qn] , so Q is a mn matriA whose columns form an
orthonormal basis for the column space of A .
Now,

A u1 u 2 u n q1 q 2

i.e., A=QR.
Where,

v1

0
R 0

v1

0
qn 0

u 2 , q1
v2
0

u 2 , q1

u 3 , q1

u n , q1

v2

u3 , q2

v3

u n , q2
u n , q3

vn

u 3 , q1
u3 , q2
v3

u n , q1
u n , q2
u n , q3

0
vn

Thus A can be decomposed as A=QR , where R is an upper triangular and

nonsingular matriA.

20

QR Decomposition
EAample: Find the QR decomposition of

1 1 1
1 0

A
1 1 0

0 0 1
21

Calculation of QR Decomposition
Applying Gram-Schmidt process of computing QR decomposition
1st Step: r11 a1 3
q1

2nd Step:
3rd Step:

1
a1

1
3

1
3
a1

3
1

r12 q1T a 2 2

1 3
1
1 / 3

1 3
0
2/3
T
q 2 a 2 q1 q1 a 2 a 2 q1 r12
(2 / 3 )
1 / 3
1
1
3

0
0
0

r22 q 2 2 3
1/ 6

1
23
q2
q 2

q 2
1/ 6

22

Calculation of QR Decomposition
4th Step:
5th Step:

r13 q1T a3 1 3

r23 q 2T a3 1

6th Step:
1/ 2

q 3 a3 q1 q1T a3 q 2 q 2T a3 a3 r13 q1 r23 q 2

1/ 2

r33 q 3 6 / 2

q3

1/ 6

1
q
q3 3 1 / 6

2/ 6

23

Calculation of QR Decomposition
Therefore, A=QR
1 1 1 1 / 3 1 / 6

1 0
0 1/ 3 2 / 6

1 1 0 1/ 3 1/ 6

0
0

1
0

1/ 6

3 2 / 3 1/ 3

1/ 6
0
2 / 6
0

2/ 6
0

1/ 6
6 / 2

QR Decomposition in MATLAB:
[Q,R] = qr(A)

Uses: QR decomposition is widely used in computer codes to find the

eigenvalues of a matrix A, to solve linear systems, and to find least squares
approAimations.
24

Least square solution using QR

Decomposition
The least square solution of b is

X X b X Y
t

Let A=QR. Then

X X b QR QR b R Q QRb R Rb
t

X Y RQY
t

Therefore,

R Rb R Q Y R
t

t 1

R Rb R
t

t 1

R t Q t Y Rb Q t Y Z
25

Cholesky Decomposition
Cholesky died from wounds received on the battle field on 31 August 1918
at 5 o'clock in the morning in the North of France. After his death one of
his fellow officers, Commandant Benoit, published Cholesky's method of
computing solutions to the normal equations for some least squares data
fitting problems published in the Bulletin godesique in 1924. Which is
known as Cholesky Decomposition
Andre-Louis Cholesky
1875-1918

Cholesky Decomposition: If A is a real nn matriA, symmetric (AT = A )

and positive definite matriA then there eAists a unique lower triangular
matriA L with positive diagonal element such that
.

A LL or LL
T

or LL * .

Note L is not such as equl to L in LU

26

Cholesky Decomposition
Proof: Since A is a nn real and positive definite so it has a LU
decomposition, A = LU. Also let the lower triangular matriA U to be a unit
one (i.e. set all the entries of its main diagonal to ones). So in that case LU
decomposition is unique. Let us suppose D diag (l11 , l22 ,, lnn ). Since A
is positive definite so all diagonal elements of D are positive.
Let L1 = L/D
L1 = UT
observe that L1 is a unit lower triangular matriA, i.e. det(L1) = det(U) = 1.
Thus, A = L1DU = L1D1/2(D1/2)TL1T = L1D1/2(D1/2L1)T .
Let L2 = L1D1/2
Then we can write

A = L2L2T

27

LDL Decomposition
A closely related variant of the classical Cholesky decomposition is the LDL
decomposition,

where L is a unit lower triangular (unitriangular) matriA and D is a diagonal matriA

which diagonal element is from lower triangular matriA L from LU decomposition.
This decomposition is related to the classical Cholesky decomposition, of the form LL*,
as follows:

The LDL variant, if efficiently implemented, requires the same space and computational
compleAity to construct and use but avoids eAtracting square roots. [5] Some indefinite
matrices for which no Cholesky decomposition eAists have an LDL decomposition with
negative entries in D. For these reasons, the LDL decomposition may be preferred. For
real matrices, the factorization has the form A = LDLT and is often referred to as LDLT
decomposition

Note: L in LDL is related to L in LU and in Cholesky decompositions but not same

28

Cholesky Decomposition (Cont.)

Procedure To find out the cholesky decomposition
a11 a12 a1n
Suppose

a 21

a n1

We need to solve
the equation

a 22 a 2 n

a n 2 a nn
a11 a12
a a
A 21 22

a n1 a n 2

a1n l11 0 0
a 2 n l 21 l 22 0

a nn l n1 l n 2 l nn

L

l11
0

l 21
l 22

l n1
l n 2

l nn

LT

29

EAample of Cholesky
Decomposition
Suppose

For k from 1 to n
k 1

l kk a kk l ks2
s 1

2 2
4

A 2 10 2
2 2 5

Then Cholesky Decomposition

For j from k+1 to n

1/ 2

k 1

l jk a jk l js l ks
s 1

l kk

Now,
2 0
L 1 3

1 1

0
0

30

Cholesky and LDL Decompositions

Cholesky decomposition , in MATLAB

2 2
4
2 0
A 2 10 2 LLT 1 3

5
2 2
1 1

L = chol(A,'lower')
0
0

2 1 1
0 3 1

3
0 0

LDLT decomposition , in MATLAB [L,D] = ldl(A)

2 2
0 0 4 0 0 1 1 / 2 1 / 2
4
1
A 2 10 2 LDLT 1 / 2
1 0 0 9 0 0 1
1/ 3

5
1
2 2
1 / 2 1 / 3 1 0 0 3 0 0

31

Application of Cholesky
Decomposition
Cholesky Decomposition is used to solve the system
of linear equation AA=b, where A is real symmetric
and positive definite.
In regression analysis it could be used to estimate the
parameter if ATA is positive definite.
In Kernel principal component analysis, Cholesky
decomposition is also used (Weiya Shi; Yue-Fei
Guo; 2010)
32