Stochastic

APPM 7400

Simulation

!!!

Random Numbers
What is a random number?
A random number is a number
uniformly distributed

that is
between zero and one!

(ie: it is a realization of a uniform(0,1) random variable)

Random Numbers
A random number stream is a sequence of
independent

random

numbers.

Random Numbers
Random numbers are the basic
blocks for all simulation

algorithms!

So, it is important to us, as

a good

building

simulators, to have
random number generator

Random Numbers
As important as this is, this course
on RNGs.

The assumption will be that we are

will not focus

starting with
a decent RNG and we will go from there! (including
evaluation of the RNG)

Now, a brief discussion on RNGs

Random Numbers
A random number generator is an

algorithm
used by a computer to generate a stream of independent
realizations of a uniform(0,1) random variable.

Random Number Generators

The CPU will do this
and then a little of that

and this other thing

with some iteration
but ultimately it can only follow an algorithm with a sequence of
deterministic
steps
a recipe to take
us from one number to another
with absolutely no randomness!!!!!

Random Number Generators

With a good enough algorithm, the

CPU can
produce a stream of seemingly independent
uniform(0,1) numbers
but they are actually pseudo-random

numbers.

(I WILL NOT SAY THIS AGAIN!!!!)

Linear Congruential RNGs

first proposed by Lehmer in 1949.
relies on modular arithmetic:
For an integer m (usually >0), the numbers a and b are congruent modulo m if the
difference, a-b, is exactly divisible by m.
We write
or

a-b|m
a=b(mod m)

Linear Congruential RNGs

a can also be thought of as the remainder in the division of b by m:

b(mod m)=a
For example,

7(mod 2) = 1
7.2(mod 2) = 1.2
7(mod 9) = 7

Linear Congruential RNGs

For a linear congruential RNG, we proceed as follows:
1.

Choose

a modulus m>0
a multiplier a such that 0<a<m

an increment b such that 0<b<m

a seed (starting value) x0 such that 0< x0<m

Linear Congruential RNGs

For a linear congruential RNG, we proceed as follows:
2.

Produce a sequence

xn = (axn-1+b)(mod m)
3.

Scale the xn numbers to between 0 and 1 by setting

un = xn/m

Linear Congruential RNGs

Youre kidding right?
This formulaic sequence of steps is supposed to be giving us a stream of independent
uniformly distributed numbers???
Well, they might at least appear that way with good choices for a,b, and m

and they might not for bad choices

Linear Congruential RNGs

Example:

x0=0,

a=1,

b=2,

m=3

x1 = (ax0+b)(mod
m)

x3 = (ax2+b)(mod
m)

= 2(mod 3) = 2

= 3(mod 3) = 0

x2 = (ax1+b)(mod
m)

x4 = (ax3+b)(mod
m)

= 4(mod 3) = 1

= 2(mod 3) = 2

Linear Congruential RNGs

We get the sequence:

0,2,1,0,2,1,0,2,1,0,2,1,0,2,1
which we divide by m=3 to get the uniform sequence of independent numbers:

0, 2 , 1, 0, 2 , 1, 0, 2 , 1, 0, 2 , 1, 0, 2 , 1, ...
3 3 3 3 3 3 3 3 3 3
(bad!)

Linear Congruential RNGs

One can show what we just observed:
the maximum period (or length of cycle before a
for a LCRNG of this form

is given by m

number is repeated)

but we must be careful about the entire combination of a, b, and m

Example: RANDU (IBM)

a 2 3, b 0, m 2
16

31

RANDU

100,000 values

L
O
O
K
S
U
N
I
F
O
R
M

RANDU

100,000 values

Scatterplot of orderes triples: (x1,x2,x3), (x4,x5,x6),

15 parallel
planes!

(bad!)

Tests of RNGs
Tests for independence
Tests for uniformity

Independence
runs up and runs down
runs above and below the mean
length of runs test

autocorrelation test

Runs Up and Runs Down Test

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

The first 21 numbers in a list of 100,000 generated by my RNG. (read across rows,
order is important!)

Independence

Runs Up and Runs Down Test

0.5124193

+ 0.7327477

0.6935230

0.2356527

+ 0.4611925

0.1976950

0.0562540

+ 0.8739298

0.4151190

0.4061648

0.2835899

0.2453624

0.8346362

0.6285227

+ 0.7430346

0.3687941

0.1235015

+ 0.1849467

0.0082797

+ 0.8593466

0.4350651

This slide: n=21,

runs=14
Overall: n=100,000,
runs= 66,734

Independence

Runs Up and Runs Down Test

Let X1, X2, , Xn be a random sample of size n.
Let Rn be the number of up/down runs in the sample.
Then if X1, X2, , Xn are independent,

2n
-1
E[Rn]
3
(Why?

and

16n
29
Var[Rn]
90
Independenc

Runs Up and Runs Down Test

Let

n E[Rn] and n2 Var[Rn]

Then for n large enough, the CLT gives us:

R
Zn : n n N(0,1)
n
So, if the observed uniform draws are really independent, roughly 95% of the time we
should see this test statistic between 1.96 and 1.96.
Independence

Runs Up and Runs Down Test

For our sample, n=100,000, Rn = 66,734

2(100000)
-1

100000
3

and

16(100000)-29
100000
90

z 0.50755

We passed!
Independence

Runs Up and Runs Down Test

For our sample, n=100,000, Rn = 66,734

2(100000)-1 and
100000
3

16(100000)-29
100000
90

-1.96 Z 1.96
R
100000

should be between

66,405.00236 and 66,927.66430

We passed!
Independence

Runs Above and Below the Mean

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

The mean of the uniform(0,1) distribution is 0.5.

0.43506513399

etc

Independence

Runs Above and Below the Mean

0.51241930545
+

0.73274767607
+

0.69352298968
+

0.23565273400
-

0.46119246644
-

0.19769494628
-

0.05625394357
-

0.87392980821
+

0.41511904218
-

0.40616476953
-

0.28358993554
-

0.24536241676
-

0.83463619173
+
This slide: n=21, runs = 8
0.36879405984
Overall: n=100,000, runs = 49,903
0.00827973058
-

0.62852266468
+

0.74303463833
+

0.12350150713
-

0.18494672983
-

0.85934662808
+

0.43506513399
-

etc

Independence

Runs Above and Below the Mean

Let X1, X2, , Xn be a random sample of size n from the uniform(0,1) distribution.
Let Rn be the number of above/below mean runs in the sample.
Then if X1, X2, , Xn are independent,

1
E[Rn]
2
(Why?

and

n
-1
Var[Rn]
4
Independenc

Runs Above and Below the Mean

Let

n E[Rn] and n2 Var[Rn]

Then for n large enough, the CLT gives us:

R
Zn : n n N(0,1)
n
So, if the observed uniform draws are really independent, roughly 95% of the time we
should see this test statistic between 1.96 and 1.96.
Independence

Runs Above and Below the Mean

For our sample, n=100,000, Rn = 49,903

100000

100000
2
and

100000-1
100000
4

z -0.616647
(49690.598, 50310.402)

We passed!
Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (down) of length 2.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (down) of length 2.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (down) of length 4.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (down) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (down) of length 2.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (down) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

0.00827973058

0.85934662808

0.43506513399

A run (up) of length 1.

Independence

Length of Runs Test

(Up/Down)

0.51241930545

0.73274767607

0.69352298968

0.23565273400

0.46119246644

0.19769494628

0.05625394357

0.87392980821

0.41511904218

0.40616476953

0.28358993554

0.24536241676

0.83463619173

0.62852266468

0.74303463833

0.36879405984

0.12350150713

0.18494672983

Frequency
0.00827973058
0.85934662808

0.43506513399

This slide:

Run Length
1

10

A run (down) of length 1.

Independence

Length of Runs Test

(Up/Down)

Let X1, X2, , Xn be a random sample of size n.

For i=1,2,,n-2, the expected number of runs (under the assumption of independence) of
length i is:

2
2
3
2

E :
n(i 3i 1) - (i 3i i 4)
i

(i 3)!

and

E
2
n 1 n!

Independence

Length of Runs Test

Now, we have a good old-fashioned

(Up/Down)

test.

Ho : The numbers in the sample are independent.

Ha : The numbers in the sample are not independent.
Test statistic :

Under Ho :

(O i E i )
W
Ei
i 1
k

W~ (k 1)

Independence

Length of Runs Test

(O i E i )
W
Ei
i 1
k

(Up/Down)

Under Ho :

the computation of Ei is based on the assumption that Ho

W~

(k 1)
2

is true

if Ho is true, then Oi should be close to Ei

otherwise, they will not be close and W will be large

How large is large?

Answer : We will compare W to a

2
critical
(k 1value
) (upper tail).

Independence

Length of Runs Test

(Up/Down)

In our full sample (n=100,000):

Run Length

Observed

Expected

41,721

41,666.75

18,386

18,333.1

5,178

5,277.647

1,145

1,150.752

220

203.3635

45

30.31132

3.912875

0.445829

Independence

Length of Runs Test

(Up/Down)

So,

2
2
(
41721
41666
.
75
)
(
18386
18333
.
1
)
W

41666.75
18333.1

No!
Independence

Length of Runs Test

(Up/Down)

Not only would this be a difficult infinite

even valid for a test!

sum, but its not

2

We need the expected value in each category to be at least 5.

Independence

Length of Runs Test

(Up/Down)

In our full sample (n=100,000):

Run Length

Observed

Expected

41,721

41,666.75

18,386

18,333.1

5,178

5,277.647

1,145

1,150.752

220

203.3635

6 or 6more

48
45

30.31132
34.72019

3.912875

W 7.837548
8

2 (5)0.445829
11.0705
0.95
Independence

Autocorrelation Test
Let be the lag j autocorrelation :

(X ,X ) : Corr(X ,X )
j

i i j

i i j

Cov(X , X )
Cov(X ,X )
i i j
i i j

Var(X) Var(X )
Var(X )
1
i
i j
Independence

Autocorrelation Test

EX X
-E X E X
i i j
i
i j

j
Var(X )
1
2

EX X - EX

0 j j

(under Ho)
Var(X )
1

12 E[X X ] - 3
0 j
(uniform)

Independence

Autocorrelation Test
12 E[X X ] - 3
j

0 j

Estimate it from the data:

j,n

12

h

i 1

XX
i i j - 3
h

(h is the largest integer so that you dont go past the end of the sequence)

Independence

Autocorrelation Test
( 0 j 0)
j

Ho : the values are independent

Ha : the values are not independent
Idea: Reject Ho if

( 0 for some j)
j

is too
large or too small
j,n

We can once again appeal to the CLT!!!

n E

j,n

13n

n Var
j,n
(n1)2

Independence

Autocorrelation Test
Test statistic:

-
n
Z j,n
j

Reject

if 0zj is outside of (-1.96,1.96).

j

People generally perform this test for some small lags (ex: j=1,2,,10)
Independence