' 8 Covariance Stationary Time Series

Statistics 626
$
So far in the course we have looked at what we have been calling time
series data sets. We need to make a series of assumptions about our
data set in order to accomplish the aims of our analysis. We will do this
in analogy with making inferences about a population (confidence
intervals and tests of hypothesis for means and variances, and so on)
from a sample in introductory statistics courses.

For example, in introductory statistics we get one sample X1 , . . . , Xn

of size n from a population. We assume the population has mean and
variance 2 . If we assume the population has a normal distribution, we
get 100(1 )% confidence intervals for and 2 as
!
s (n 1)s (n 1)s 2 2
X t/2,n1 , , ,
n 2/2,n1 21/2,n1

where X and s2 are the mean and variance of the sample and t,v and
2,v are the values of t and 2 random variables with v degrees of
freedom having area to their right, that is, the 100(1 ) percentiles
of the t and 2 distributions.

& %
These are called 100(1 )% confidence intervals because if we took
random samples many, many times and got the intervals each time, then
100(1 )% of the samples would give intervals containing the true
value of the parameter being estimated.

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 1
' Statistics 626

In time series analysis, a data set is not a random sample from a

population, so how do we end up getting confidence intervals and tests
$
of hypotheses for parameters?

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 2
' 8.1 The Ensemble of Realizations
Statistics 626
$
The first step in our logic that will allow us to do inferential statistics with
time series is as follows. We imagine that our data set is just part of a
realization that lasts from far into the past until far into the future.
Further, we imagine that the realization we are observing is just one of
many, many realizations that we could have observed. Here are two
examples of how this thought process makes sense:

1. If we are observing an EEG record, it would make sense to imagine

that the realization we are looking at today is similar to, but not
identical to, a record we could observe at some future day. We can
imagine that there is some random mechanism generating
realizations. Another way to think of this is that a persons EEG is
very long and we are looking at independent pieces of it.

2. If we must measure our time series variable with error, then

immediately it makes sense to think of our realization as just one of
many we could see.

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 3
' 8.2 Definitions
Statistics 626
$
1. The set of all possible realizations that could be observed is called
the ensemble of realizations. The set of possible values at a
particular time point t is denoted by X(t), and a time series is
denoted by {X(t), t T }. A time series data set is one part of
one realization and is denoted by x(1), . . . , x(n).

2. We denote by

(t) = E(X(t)), K(s, t) = Cov(X(s), X(t)),

the mean and the covariance functions of X .

3. If (t) is the same for each t and K(s, t) only depends on how far
apart s and t are (that is K(s, t)= R(|t s|) for a function R
called the autocovariance function of X ), then we say that X is
covariance stationary with mean and autocovariance function R
and we have:

(a) The autocorrelation function of X is given by

(v) = Corr(X(t), X(t + v)),

& %
and the spectral density function f (), [0, 1] is given by

X
f () = R(v)e2iv , [0, 1].
v=

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 4
' 4. A white noise time series {(t), t
Statistics 626

Z} is a time series satisfying

E((t)) = 0, R(v) = v 2 , where the Kroneker delta function v
$
is 1 if v = 0 and zero otherwise. We denote a white noise time
series by X W N ( 2 ).

5. A time series model is a mathematical formula expressing how the

realizations of the series are formed. For example, a moving
average model of order one with coefficient and noise variance 2
(which we denote by X M A(1, , 2 )) means

X(t) = (t) + (t 1), t Z,

where  W N ( 2 ).

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 5
' Statistics 626

8.3 Ensemble Mean Interpretation of and f

$
We spent much of the first part of this course studying the sample
correlogram and sample spectral density function f (which at the
frequencies 0, 1/n, 2/n, . . . is the periodogram). The true or
population correlogram and spectral density function f can under
most circumstances be thought of as the average value of and f where
the average is taken over all realizations in the ensemble of realizations.

Thus these are the quantities we would really like to know about. For
EEG data, for example, and f are calculated only for the realization we
have observed, but what we really would like to know about would be the
average of these quantities over many realizations.

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 6
' Statistics 626

8.4 Rules for Expectation, Variance, and Covariance

$
It is important to distinguish the mean = E(X), 2 = Var(X), and
= Corr(X, Y ) for random variables from X , s2 , and r for data. One
way to think of this is that , 2 , and are defined for populations (such
as the X(t)s) while X , R(0), and (v) are defined for a data set.

Mathematically, the quantities , R(v), and (v) are calculated via

integration from assumed joint probability distributions for the
populations {X(t), t Z}, for example, if we assume normal
distributions for the X(t)s, we have
Z
= xf(x)dx,

where f is the pdf of the normal distribution. In this course, we will not
need to do such integration but rather use a set of simple rules for
calculating parameters from models. You can think of X(t) either as a
population, or more mathematically, as a random variable.

Note that quantities such as , R(v), and (v) are called moments of
the random variables.

& %
1. The variance of a random variable is defined to be



Var(X) = E (X E(X))2 = E(X 2 ) E(X)2 ,
which means
Var(X) = E(X 2 )

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 7
' if E(X) = 0.
Statistics 626
$
2. The covariance of two random variables is defined to be

Cov(X, Y ) = E ((X E(X))(Y E(Y ))) ,

which means
Cov(X, Y ) = E(XY )
if the means of X and Y are zero.

3. The correlation of random variables X and Y is given by

Cov(X, Y )
Corr(X, Y ) =p .
Var(X)Var(Y )

4. The mean and variance of a constant times a random variable are

given by

E(aX) = aE(X), Var(aX) = a2 Var(X).

5. The mean of the sum of random variables is the sum of the means:

E(X + Y ) = E(X) + E(Y ).

6. The variance of the sum of random variables is not generally the

sum of the variances:

& %
Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y ).

If Cov(X, Y ) = 0, the variance of a sum is the sum of the

variances.

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 8
' Statistics 626

8.5 Application of Moment Rules to Time Series

Models
$
We illustrate the moment rules on three simple time series models. In
the next topic, we study some more general rules that will greatly simplify
determining moments for a set of complex models.

White Noise

The white noise model says that our time series is just a set of zero
mean (that is, E(X(t))= 0 for all t), uncorrelated (that is,
Corr(X(t), X(t + v)) = 0 unless v = 0) random variables with
constant variance R(0) = 2 .

Thus we have for all [0, 1],

X
f () = R(v)e2iv = R(0)e0 = R(0) = 2 ,
v=

which shows why we call such a time series model white noise; it is often
used to model noise, and its spectrum is constant for all frequencies
in analogy with white light.

& %
Random Walk

A time series is said to follow a random walk model if

X(t) = X(t 1) + (t), t 1,

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 9
' Statistics 626

2
where the starting value X(0) has mean zero, variance X ,
 W N ( 2 ), and X(0) is uncorrelated with all of the s (we will see
$
why we cant assume the process started in the inifinite past, that is, why
t must start at some time such as time 0).
By successive substitution we can write

X
t
X(t) = X(0) + (j),
j=1

and so
2
Var(X(t)) = X + t 2 ,

which shows that a random walk is not covariance stationary since the
variance at time t does depend on t. In fact this variance is growing
linearly with t (see the figure below which shows 10 realizations of
length 100 from a random walk where X(0)
= 0 and the s are N(0,1);
p
the values of 2.5 Var(X(t) = 2.5 t are also plotted), which
means that if the process started in the infinite past, the variance would
have become infinite by time 0.

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 10
' Statistics 626
$

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 11
' Moving Average Process
Statistics 626
$
If
X(t) = (t) + (t 1), t Z,
where  W N ( 2 ), then

E(X(t)) = E((t)) + E((t 1)) = 0 + 0 = 0,

so the first requirement for covariance stationarity is met; namely the

mean (which is zero for all t) is constant over time. Now we have to see
if Cov(X(t), X(t + v)) only depends on v no matter which t we use.
We have

Cov(X(t), X(t + v)) = E(X(t)X(t + v)),

since E(X(t) = 0 for all t. Now we have

E(X(t)X(t + v)) = E([(t) + (t 1)][(t + v) + (t + v 1)]),

which becomes the sum of four expectations (using the First, Outside,
Inside, and Last or FOIL rule):

E((t)(t + v)) + E((t)(t + v 1)) + E((t 1)(t + v))

+ 2 E((t 1)(t + v 1)).

& %
The expected value of the product of two s can only take on two values;
if the arguments are the same, then the expectation is 2 , while if they
are different then, the expectation is zero.

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 12
' Thus we have
2
Statistics 626

2
$
(1 + ), if v = 0
Cov(X(t), X(t + v)) = 2 , if v = 1

0, if |v| > 1

which shows an MA(1) process is covariance stationary with this

covariance as R(v) which gives

1, if v = 0
R(v)
(v) = = /(1 + 2 ), if v = 1
R(0)
0, if |v| > 1.

Thus to get the spectral density function f , we have since all of the
values of R(v) are zero except for v = 0 and v = 1:

f () = R(0) + R(1)e2i + R(1)e2i

= R(0) + 2R(1) cos(2)
= 2 ((1 + 2 ) + 2 cos(2)),

since R(1) = R(1) and ei + ei = 2 cos().

& %
Thus the spectral density of an MA(1) is a constant plus another
constant (which has the same sign as that of ) times a cosine that goes
through half a cycle for [0, 0.5], which means f can only look two
ways; if > 0, it has an excess of low frequency, while if < 0, it has

Topic 8: Covariance Stationary Time Series Copyright

c 1999 by H.J. Newton Slide 13
' an excess of high frequency. Finally, if
noise, that is,
Statistics 626

= 0, we have that X is white

$
M A(1, = 0, 2 ) = W N.

&
Topic 8: Covariance Stationary Time Series Copyright
c 1999 by H.J. Newton
%
Slide 14