Université Paris Sciences et Lettres

&
Paris Dauphine

Linear Algebra and Markov Chains

Supervisor:
Constantin Dalyac Cyril Labbé

June 28, 2018

Contents

I Matrix Representation of Markov Chains 3

1 Stationary Distributions: existence and uniqueness 4

II Markov Chain Mixing 5

2 The Convergence theorem 5

3 Reversibility and Time Reversals 5

4 Eigenvalues and relaxation time 6

4.1 The relaxation time . . . . . . . . . . . . . . . . . . . . . . . . . 7

III Two examples of Markov chains 8

5 Random walk on the n-cycle 8

6 Random walk on the path 10

1
Introduction
Markov chains were introduced by Andrei Markov in the early 20th century in an
argument with Pavel Nekrasov, who claimed independence was necessary for the
weak law of large numbers to hold. Markov managed to show in 1906 that under
some conditions, the average of Markov chains will converge to a stationary
distribution, thus proving a weak law of large numbers without the assumption
of independence. Today, Markov chains are used in many domains, ranging from
Biology and Physics to speech recognition. Google decided to model websites
and links as Markov chains: using its mathematical properties was key in making
it the most-used research engine in the world. We will see in the mathematical
introduction that Markov chains can be described with matrices; a central aim of
this paper is to use the tools of linear algebra in order to understand the different
properties of Markov Chains, illustrating them with examples simulated with
Matlab. We will first explore the different characteristics of Markov chains
and the way they evolve in time.

Figure 1: Example of a visualization of a Markov chain space. Vertices repre-

sents states that can be taken by a sequence of random variables. At each step,
Xt moves to a different state. For example, if Xt is in state 7, Xt+1 has 70% of
chance of being state 7 and 30% of being state 6.

2
Part I
Matrix Representation of Markov
Chains
Let Ω be a finite set of the form Ω = {x1 , x2 , . . . , xN }. A finite Markov chain is
a process which moves along the elements of Ω in the following manner: when at
xi ∈ Ω, the next position is chosen according to a fixed probability distribution
P (xi , ·). More precisely, a sequence of random variables (X0 , X1 , . . .) is a Markov
chain with state space Ω and transition matrix P if for all i, j ∈ J1; N K, all t ≥ 1,
and all events Ht−1 = ∩t−1 s=0 {Xs = ys }, ys ∈ Ω, satisfying P(Ht−1 ∩ {Xt = x}) >
0, we have:

P{Xt+1 = xj | Ht−1 ∩ {Xt = xi }} = P{Xt+1 = xj ∩ Xt = xi } = P(xi , xj ) (1)

This equation is called the Markov property, meaning that the con-
ditional probability of proceeding from state xi to state xj does not depend on
the states preceding xi . Hence the total information on the Markov chain is
contained in a matrix P ∈ MN ([0; 1]). P is a stochastic matrix, i.e. its entries
are all non-negative and
X
P (x, y) = 1, for all ∈ Ω.
y∈Ω

Matrix representation enables us to use tools of linear algebra. Suppose

we start at t = 0 in position x2 . We introduce a distribution vector of the form:

µ0 := (0, 1, 0, . . . , 0),

where the j th coordinate corresponds to the probability of presence at the state

xj . The following distribution µ1 will then be given by the multiplication µ1 =
µ0 P . By recurrence, we see that multiplying by P on the right updates the
distribution by another step:
µt+1 = µt P
and for any initial distribution µ0 ,

µt = µ0 P t

If we multiply a column vector f by P on the left, thinking of f as a

function on the state space Ω, then the x-th entry of the resulting vector is:
|Ω| |Ω|
X X
P f (xi ) = P (xi , xj )f (xj ) = f (y)Pxi {X1 = xj } = Exi (f (X1 )
j=1 j=1

Hence multiplyng a column vector by P takes us from a function on the state

space to the expected value of that function at the following time. A natural

3
question then rises: can we expect µt to converge to a certain distribution when
t goes to infinity? And if it is the case, does the long-term distribution depend
on the initial distribution µ0 ?

1 Stationary Distributions: existence and unique-

ness
We call a distribution π on Ω a stationary distribution if it satisfies the
following equation:
π = πP.
It is stationary because updating the distribution by a step is done by multi-
plying π by P on the right, but πP = π by definition, hence the distribution is
unchanged. This does not imply that we "loose" the randomness in the process
but it describes the fact that the probability of being in a certain state of Ω is
fixed. Let us now show that under some assumptions, stationary distributions
exist and are unique.
Definition 1.1. A chain P is called irreducible if for any i, j, ∈ J1; N K, there
exists an integer t (possibly depending on i and j) such that P t (xi , xj ) > 0. This
means it is possible to go from one state to any other using only transitions of
positive probability.
Definition 1.2. Let T (xi ) := {P t (xi , xi ) > 0} be the set of times when it is
possible for the chain to return to a starting position xi . The period of the state
xi is defined to be the greatest common divisor of T (xi ).
Lemma 1.1. If P is irreducible, then gcd(T (xi ))=gcd(T (xj ), for all i, j ∈
J1; N K.
Proof. Let’s choose xi and xj in Ω. P is irreducible therefore there exists r and
l such that P r (xi , xj ) > 0 and P l (xj , xi ) > 0. Let m := r + l. Then m ∈
T (xi ) ∩ T (xj ) and T (xi ) ⊂ T (xj ) − m. Hence gcdT (xj ) divides all elements
of T (xi ), i.e. gcdT (xj ) ≤ gcdT (xi ). By a symmetric reasoning, we obtain
gcdT (xi ) ≤ gcdT (xj ).
For an irreducible chain, the period of the chain is defined to be the
period which is common to all states. The chain will be called aperiodic if all
states have period 1. If a chain is not aperiodic, we call it periodic.
Proposition 1.1. Let P be the transition matrix of an irreducible Markov chain.
There exists a unique probability distribution π satisfying π = πP .
This proposition is proved by the Convergence Theorem, stated in the
next part. The Convergence Theorem shows that if a Markov chain is irreducible
and aperiodic, it converges in distribution to its unique stationary distribution.
Moreover the theorem quantifies the speed of convergence to the stationary
distribution.

4
Part II
Markov Chain Mixing
Since we are interested in quantifying the speed of convergence of Markov chains,
we need to choose an appropriate metric for measuring the distance between
distributions.
The total variation distance between two probability distributions
µ and ν on Ω is defined by

kµ − νkT V = max |µ(A) − ν(A)| (2)

A⊂Ω

2 The Convergence theorem

Theorem 1. Suppose that P is irreducible and aperiodic, with stationary dis-
tribution π. Then there exists constants α ∈ (0,1) and C > 0 such that

max kP t (x, ·) − πkT V ≤ Cαt (3)

x∈Ω

Proof. See p.52 of [1].

In order to bound the maximal distance between P t (x0 , ·) and π, we
define
d(t) := max kP t (x, ·) − πkT V
x∈Ω

. We also introduce a parameter which measures the time required by a Markov

chain for the distance to stationarity to be small. The mixing time is defined
by
tmix () := min{t : d(t) ≤ } and tmix := tmix (1/4).

3 Reversibility and Time Reversals

Tools of linear algebra can only be applied to Markov chains that are reversible.
We will therefore give the definition of reversibility then show how useful linear
algebra can be in that case.
Suppose a probability π on Ω satisfies for all i, j ∈ J1; N K

π(xi )P (xi , xj ) = π(xj )P (xj , xi ) (4)

.
Theses equations are called the detailed balanced equations.

Proposition 3.1. Let P be the transition matrix of a Markov chain with state
space Ω. Any distribution π satisfying the detailed balanced equations is station-
ary for P.

5
Proof. π is a stationary distribution
PN i.i.f. π = πP .PLet π̃ = πP . Then
N
for all j ∈ J1; N K , π̃j = i=1 P (xi , xj )π(xi ) = i=1 P (xj , xi )π(xj ) = π(xj )
since P is stochastic. Hence π̃ = π
Furthermore, when (2) holds,

π(x0 )P (x0 , x1 ) · · · P (xN −1 , xN ) = π(xN )P (xN , xN −1 ) · · · P (x0 , x0 )

which we can rewrite in the following suggestive form:

Pπ {X0 = x0 , · · · , XN = xN } = Pπ {X0 = xN , · · · , Xn = x0 }

In other words, if a chain Xt satisfies (2) and has stationary initial distribution
, then the distribution of (X0 , X1 , · · · , XN ) is the same as the distribution of
(XN , XN −1 , · · · , X0 ). For this reason, a chain satisfying (2) is called reversible.

4 Eigenvalues and relaxation time

We start by giving some facts about the eigenvalues of transition matrices:
Lemma 4.1. Let P be the transition matrix of a finite Markov chain.
1. If λ is an eigenvalue of P, then |λ| ≤ 1.
2. If P is irreducible, the vector space of eigenfunctions corresponding to the
eigenvalue 1 is the one-dimensional space generated by the column vector
1 := (1, 1, . . . , 1)T .
3. If P is irreducible and aperiodic, then −1 is not an eigenvalue of P.
Proof. (A écrire)
We denote by h·, ·i the usual inner product on RΩ , given by
X
hf, gi = f (x)g(x)
x∈Ω

. We also define the inner product h·, ·iπ as:

X
hf, giπ = f (x)g(x)π(x) (5)
x∈Ω

Because we regard elements of RΩ as functions from Ω to R, we will

call eigenvectors of the matrix P eigenfunctions.
Lemma 4.2. Let P be reversible with respect to π. The inner product space
|Ω|
(RΩ , h·, ·iπ ) has an orthonormal basis of real-valued eigenfunctions {fj }j=1 cor-
responding to real eigenvalues {λj }.
Proof. (A écrire)

6
4.1 The relaxation time
For a reversible transition matrix P, we label the eigenvalues of P in decreasing
order:
1 = λ1 > λ2 ≥ · · · ≥ λ|Ω| ≥ −1.
We define λ? := max{|λ| : λ is an eigenvalue of P, λ 6= 1}
The difference γ? := 1−λ? is called the absolute spectral gap.Lemma
4.1 implies that if P is periodic and irreducible, γ? > 0. The spectral gap of a
reversible chain is defined by γ := 1 − λ2 .
The relaxation time trel of a reversible Markov chain with absolute
spectral gap γ? is defined to be
1
trel :=
γ?
Theorem 2. Let P be the transition matrix of a reversible, irreducible Markov
chain with state space Ω, and let πmin := minx∈Ω π(x). Then
1 1
(trel − 1)log( ) ≤ tmix () ≤ log( )trel
2 πmin
We will now illustrate the previous definitions with two family of ex-
amples. The first one will be a Markov chain on a cyclic group, then we will see
how it is linked to a random walk on a path.

7
Part III
Two examples of Markov chains
We decided to study the random walk on a cycle and on a segment.

5 Random walk on the n-cycle

Let ω = e2πi/n . The set Wn := {ω, ω 2 , · · · , ω n−1 , 1} represents the n roots of
unity inscribed in the unit circle of the complex plane. We can therefore view
simple random walk on the n-cycle as the random walk on the group (Wn , ·)
with increment distribution uniform on {ω, ω −1 }. This chain is clearly aperiodic
and irreducible; according to the convergence theorem, there exists a unique
stationary distribution, which is π(xi ) = n1 as one would expect. Consider
P ∈ MN ([0; 1]) the transition matrix of the random walk.

0 12 0 ··· 0 12
 
1 1 .. 
.
2 0 0

2
.

 0 . . . . . . . . . . . . .. 

P = . .

 .. . . . . . . . . . . . 0

 
0 . . . . . . . . . 0 1 
 
2
1 1
2 0 ··· 0 2 0
 
f (ω)
f (ω 2 )
Let f =  .  be an eigenfunction of P with eigenvalue λ. It
 
 .  .
f (1)
satisfies:
f (ω k−1 ) + f (ω k+1 )
∀k ∈ J0; n − 1K, λf (ω k ) = P f (ω k ) =
2
For 0 ≤ j ≤ n − 1, define φj (ω k ) := ω kj . Then

φj (ω k−1 ) + φj (ω k+1 ) ω jk+j + ω jk−j ω j + ω −j 2πj

P φj (ω k ) = = = ω kj ( ) = cos( )φj (ω k ),
2 2 2 n
hence φj is an eigenfunction of P associated to the eigenvalue cos( 2πj
n ).
4π 2 −4
We have λ2 = cos(2π/n) = 1 − 2n2 + O(n ),, therefore the spectral
gap is of order n−2 and the relaxation time is of order n2 .
When n = 2p is even, cos(2πp/n) = −1 and −1 is an eigenvalue so the
absolute spectral gap is 0. That is because random walk on an even number of
states is periodic; states can be classified as even or odd and each step always
goes from one class to another.

8
 (a) (b)

(c) (d)

Figure 2: Matlab simulation of the random walk on the group (W5 , ·) ((a) and
(b)) and (W9 , ·) ((c) and (d)). The eigenvalues are represented in blue in the
complex plane, while the width of the red band represents the spectral gap. As
n → ∞, the spectral gap tends to 0 and relaxation time tends to ∞, as expected
from our calculations.

9
(a) projection of the simple walk on the 12-
cycle onto the real axis. We can see that(b) Projecting a random walk on the odd
for most of the projected points, except atstates of a 16-cycle gives a random walk on
the ends, there is the same probability ofthe 4-path, with holding probability of 1/2
going to the left or to the right. Notice theat the endpoints.
reflecting boundary conditions.

6 Random walk on the path

The random walk on the path is closely linked to the random walk on the cycle.
We first need to introduce the concept of projected chains.
Lemma 6.1. Let Ω be the state space of a Markov chain (Xt ) with transition
matrix P. Let ∼ be an equivalence relation on Ω with equivalence classes Ω] =
{x̄ : x ∈ Ω}, and assume that P satisfies
P (x, ȳ) = P (x0 , ȳ)
whenever x ∼ x0 . Then X̄t is a Markov chain with state space Ω] and transition
matrix P ] defined by P ] (x̄, ȳ) := P (x, ȳ).
We can therefore construct a new chain by taking equivalence classes
for an equivalence relation compatible with the transition matrix in the sense
of the Lemma. If we project the previous simple walk on the n-cycle onto the
real axis, (see figure) we obtain a process which appears to be a random walk
on the path.
The link between the eigenvectors and eigenvalues of the two chains is
given by the following lemma:
Lemma 6.2. With the same notations and conditions as in the previous lemma:
Let f : Ω → R be an eigenfunction of P with eigenvalue λ which is
constant on each equivalence class. Then the natural projection f ] : Ω] → R of
f , defined by f ] (x̄) = f (x), is an eigenfunction of P ] with eigenvalue λ.
Proof. By computation: (P f ] )(x̄) =
P ]
P (x̄, ȳ)f ] (ȳ) =
P
P (x, ȳ)f (y) =
P P P ȳ∈Ω] ȳ∈Ω]
P (x, z)f (z) = P (x, z)f (z) = (P f )(x) = λf (x) = λf (x̄).
ȳ∈Ω] z∈ȳ z∈Ω

10
Path with reflection at endpoints
Let P be the transition matrix of simple random walk on the 2(n-1)-cycle identi-
fied with random walk on the multiplicative group W2(n−1) = {ω, ω 2 , · · · , ω 2n−1 =
1} , where ω = eπi/(n−1) . Now we choose the relation of equivalence as con-
jugation, i.e. ω k ∼ ω −k . The equivalence respects the first lemma, and now
if we identify each equivalence class with the projection of its elements on the
real axis vk = cos(πk/(n − 1)), the projected chain is a simple random walk on
the math with n vertices W ] = {v0 , v1 , · · · , vn−1 }. Note the reflecting boudary
conditions; when at v0 , it moves with probability one to v1 .
According to the previous lemma and the calculation done in the pre-
vious part, the functions fj] : W ] → R defined for all j ∈ J0; n − 1K by

πjk
fj] (vk ) = cos( )
n−1
πj
are eigenfunctions of the projected walk, associated to the eigenvalue cos( (n−1) ).
2
π −4
We have λ2 = cos(π/(n − 1)) = 1 − 2(n−1) 2 + O(n ), therefore the
−2
spectral gap is of order n and the relaxation time is of order n2 , as in the
simple random walk on the cycle.

Path with holding probability 1/2 at endpoints

How could change the initial chain in order to obtain, by projection, a random
walk on the path such that on the endpoints there is a probability of 1/2 of
staying on the same spot?
Consider ω = eπi/(2n) , and the simple random walk on the (2n)-
element set identified with random walk on the multiplicative group Wodd =
{ω, ω 3 , · · · , ω 2n−1 }, where at each step the current state is multiplied by an
uniformly chosen element of {ω 2 , ω −2 }.
Now the calculations we made for the n-cycle are the same here, and
we find that the function fj : Wodd → R defined by

πj(2k + 1)
fj (ω 2k+1 ) = cos( )
2n

is an eigenfunction with eigenvalue cos( πj

n ).
Again, we identify the class of equivalence with the relation of con-
jugate, ω 2k+1 ∼ ω −(2k+1) , and we identify each class of equivalence to the
projections of its elements on the real axis. The projected chain is therefore a
simple random walk on the path with n vertices W ] = {u0 , u1 , · · · , un−1 } and
loops at the endpoints (see figure).
According to the previous lemma and the calculation done in the pre-
vious part, the functions fj] : Wodd
]
→ R defined for all j ∈ J0; n − 1K by

πj(2k + 1)
fj] (uk ) = cos( )
2n

11
are eigenfunctions of the projected walk, associated to the eigenvalue cos( πj
n ).
π2 −4
We have λ2 = cos(π/n) = 1 − n2 + O(n ), therefore the spectral gap
is of order n−2 and the relaxation time is of order n2 .

(a) (b)

(c) (d)

Figure 4: Matlab simulation of the random walk on the 7-path, i.e. the projected
chain of the random walk on the 12-cycle ((a) and (b)) with reflection at the
endpoints. ((c) and (d)) represent the calculations for the random walk on the
4-path as a projection of a random walk on the "odd" states of a 16-cycle. The
eigenvalues are represented in blue in the complex plane, while the width of the
red band represents the spectral gap; the simulations confirm our calculations.

12
Remerciement
Un grand merci à Cyril Labbé qui m’a suivi de près durant cette période, tout
en me laissant le plaisir d’explorer les nombreuses facettes de cet objet math-
ématique si intéressant. J’ai le sentiment d’avoir joué les apprentis-chercheurs
pendant ces quelques mois qui ont été baignés dans la bonne humeur; merci!

References
[1] David A. Levin, Yuval Peres, Elizabeth L. Wilmer. Markov Chains and
Mixing Times. Vol 107,American Mathematical Soc., 2017.
[2] John Tsitsiklis. Lecture 17: Markov Chains II.
https://www.youtube.com/watch?v=ZulMqrvP-Pkfeature=youtu.be
MIT, 2011.

13