MIDHILA K MANOJ, SMS CUSAT

Autocorrelation
Autocorrelation refers to the degree of correlation between the values of the same
variables across different observations in the data.  The concept of autocorrelation
is most often discussed in the context of time series data in which observations
occur at different points in time (e.g., air temperature measured on different days
of the month).  For example, one might expect the air temperature on the 1st day of
the month to be more similar to the temperature on the 2nd day compared to the
31st day.  If the temperature values that occurred closer together in time are, in
fact, more similar than the temperature values that occurred farther apart in time,
the data would be autocorrelated.

However, autocorrelation can also occur in cross-sectional data when the

observations are related in some other way.  In a survey, for instance, one might
expect people from nearby geographic locations to provide more similar answers to
each other than people who are more geographically distant.  Similarly, students
from the same class might perform more similarly to each other than students from
different classes.  Thus, autocorrelation can occur if observations are dependent in
aspects other than time.  Autocorrelation can cause problems in conventional
analyses (such as ordinary least squares regression) that assume independence of
observations.

In a regression analysis, autocorrelation of the regression residuals can also occur

if the model is incorrectly specified.  For example, if you are attempting to model a
simple linear relationship but the observed relationship is non-linear (i.e., it follows
a curved or U-shaped function), then the residuals will be autocorrelated.

How to Detect Autocorrelation

A common method of testing for autocorrelation is the Durbin-Watson test. 

Statistical software such as SPSS may include the option of running the Durbin-
Watson test when conducting a regression analysis.  The Durbin-Watson tests
produces a test statistic that ranges from 0 to 4.  Values close to 2 (the middle of
the range) suggest less autocorrelation, and values closer to 0 or 4 indicate greater
positive or negative autocorrelation respectively.

Multicollinearity
Multicollinearity is a state of very high intercorrelations or inter-associations
among the independent variables. It is therefore a type of disturbance in the data,
and if present in the data the statistical inferences made about the data may not be
reliable.

There are certain reasons why multicollinearity occurs:

 It is caused by an inaccurate use of dummy variables.
 It is caused by the inclusion of a variable which is computed from other
variables in the data set.
 Multicollinearity can also result from the repetition of the same kind of
variable.
 Generally occurs when the variables are highly correlated to each other.
Multicollinearity can result in several problems. These problems are as
follows:
 The partial regression coefficient due to multicollinearity may not be
estimated precisely. The standard errors are likely to be high.
 Multicollinearity results in a change in the signs as well as in the magnitudes
of the partial regression coefficients from one sample to another sample.
 Multicollinearity makes it tedious to assess the relative importance of the
independent variables in explaining the variation caused by the dependent
variable.
In the presence of high multicollinearity, the confidence intervals of the
coefficients tend to become very wide and the statistics tend to be very small. It
becomes difficult to reject the null hypothesis of any study when multicollinearity
is present in the data under study.

There are certain signals which help the researcher to detect the degree of
multicollinearity.
One such signal is if the individual outcome of a statistic is not significant but the
overall outcome of the statistic is significant. In this instance, the researcher might
get a mix of significant and insignificant results that show the presence of
multicollinearity.Suppose the researcher, after dividing the sample into two parts,
finds that the coefficients of the sample differ drastically. This indicates the
presence of multicollinearity. This means that the coefficients are unstable due to
the presence of multicollinearity. Suppose the researcher observes drastic change
in the model by simply adding or dropping some variable.   This also indicates that
multicollinearity is present in the data. Multicollinearity can also be detected
with the help of tolerance and its reciprocal, called variance inflation factor
(VIF). If the value of tolerance is less than 0.2 or 0.1 and, simultaneously, the
value of VIF 10 and above, then the multicollinearity is problematic.
PREDICTED AND RESIDUAL SCORES

The regression line expresses the best prediction of the dependent variable (Y),
given the independent variables (X). However, nature is rarely (if ever) perfectly
predictable, and usually there is substantial variation of the observed points around
the fitted regression line (as in the scatterplot shown earlier). The deviation of a
particular point from the regression line (its predicted value) is called
the residual value.

RESIDUAL VARIANCE AND R-SQUARE

R-Square, also known as the Coefficient of determination is a commonly used

statistic to evaluate model fit. R-square is 1 minus the ratio of residual variability.
When the variability of the residual values around the regression line relative to the
overall variability is small, the predictions from the regression equation are good.
For example, if there is no relationship between the X and Y variables, then
the ratio of the residual variability of the Y variable to the original variance is
equal to 1.0. Then R-square would be 0. If X and Y are perfectly related then there
is no residual variance and the ratio of variance would be 0.0, making R-square =
1. In most cases, the ratio and R-square will fall somewhere between these
extremes, that is, between 0.0 and 1.0. This ratio value is immediately interpretable
in the following manner. If we have an R-square of 0.4 then we know that the
variability of the Y values around the regression line is 1-0.4 times the original
variance; in other words we have explained 40% of the original variability, and are
left with 60% residual variability. Ideally, we would like to explain most if not all
of the original variability. The R-square value is an indicator of how well the
model fits the data (e.g., an R-square close to 1.0 indicates that we have accounted
for almost all of the variability with the variables specified in the model).

INTERPRETING THE CORRELATION COEFFICIENT R

Customarily, the degree to which two or more predictors (independent

or X variables) are related to the dependent (Y) variable is expressed in the
correlation coefficient R, which is the square root of R-square. In multiple
regression, R can assume values between 0 and 1. To interpret the direction of the
relationship between variables, look at the signs (plus or minus) of the regression
or B coefficients. If a B coefficient is positive, then the relationship of this variable
with the dependent variable is positive (e.g., the greater the IQ the better the grade
point average); if the B coefficient is negative then the relationship is negative
(e.g., the lower the class size the better the average test scores). Of course, if
the B coefficient is equal to 0 then there is no relationship between the variables.
Descriptive Statistics
"True" Mean and Confidence Interval. Probably the most often used descriptive
statistic is the mean. The mean is a particularly informative measure of the "central
tendency" of the variable if it is reported along with its confidence intervals. As
mentioned earlier, usually we are interested in statistics (such as the mean) from
our sample only to the extent to which they can infer information about the
population. The confidence intervals for the mean give us a range of values around
the mean where we expect the "true" (population) mean is located (with a given
level of certainty, see also Elementary Concepts). For example, if the mean in your
sample is 23, and the lower and upper limits of the p=.05 confidence interval are
19 and 27 respectively, then you can conclude that there is a 95% probability that
the population mean is greater than 19 and lower than 27. If you set the p-level to a
smaller value, then the interval would become wider thereby increasing the
"certainty" of the estimate, and vice versa; as we all know from the weather
forecast, the more "vague" the prediction (i.e., wider the confidence interval), the
more likely it will materialize. Note that the width of the confidence interval
depends on the sample size and on the variation of data values. The larger the
sample size, the more reliable its mean. The larger the variation, the less reliable
the mean (see also Elementary Concepts). The calculation of confidence intervals
is based on the assumption that the variable is normally distributed in the
population. The estimate may not be valid if this assumption is not met, unless the
sample size is large, say n=100 or more.

Shape of the Distribution, Normality. An important aspect of the "description" of

a variable is the shape of its distribution, which tells you the frequency of values
from different ranges of the variable. Typically, a researcher is interested in how
well the distribution can be approximated by the normal distribution (see the
animation below for an example of this distribution) (see also Elementary
Concepts). Simple descriptive statistics can provide some information relevant to
this issue. For example, if the skewness (which measures the deviation of the
distribution from symmetry) is clearly different from 0, then that distribution
is asymmetrical, while normal distributions are perfectly symmetrical. If
the kurtosis (which measures "peakedness" of the distribution) is clearly different
from 0, then the distribution is either flatter or more peaked than normal; the
kurtosis of the normal distribution is 0.

More precise information can be obtained by performing one of the tests of

normality to determine the probability that the sample came from a normally
distributed population of observations (e.g., the so-called Kolmogorov-Smirnov
test, or the Shapiro-Wilks' W test. However, none of these tests can entirely
substitute for a visual examination of the data using a histogram (i.e., a graph that
shows the frequency distribution of a variable).

The graph allows you to evaluate the normality of the empirical distribution
because it also shows the normal curve superimposed over the histogram. It also
allows you to examine various aspects of the distribution qualitatively. For
example, the distribution could be bimodal (have 2 peaks). This might suggest that
the sample is not homogeneous but possibly its elements came from two different
populations, each more or less normally distributed. In such cases, in order to
understand the nature of the variable in question, you should look for a way to
quantitatively identify the two sub-samples.

Correlations
Purpose (What is Correlation?) Correlation is a measure of the relation between
two or more variables. The measurement scales used should be at least interval
scales, but other correlation coefficients are available to handle other types of data.
Correlation coefficients can range from -1.00 to +1.00. The value of -1.00
represents a perfect negative correlation while a value of +1.00 represents a
perfect positive  correlation. A value of 0.00 represents a lack of correlation.

The most widely-used type of correlation coefficient is Pearson r, also

called linear or product- moment correlation.
Simple Linear Correlation (Pearson r). Pearson correlation (hereafter
called correlation), assumes that the two variables are measured on at least interval
scales (see Elementary Concepts), and it determines the extent to which values of
the two variables are "proportional" to each other. The value of correlation (i.e.,
correlation coefficient) does not depend on the specific measurement units used;
for example, the correlation between height and weight will be identical regardless
of whether inches and pounds, or centimeters and kilograms are used as
measurement units. Proportional means linearly related; that is, the correlation is
high if it can be "summarized" by a straight line (sloped upwards or downwards).

This line is called the regression line or least squares line, because it is determined

such that the sum of the squared distances of all the data points from the line is the
lowest possible. Note that the concept of squared distances will have important
functional consequences on how the value of the correlation coefficient reacts to
various specific arrangements of data (as we will later see).
How to Interpret the Values of Correlations. As mentioned before, the
correlation coefficient (r) represents the linear relationship between two variables.
If the correlation coefficient is squared, then the resulting value (r2, the coefficient
of determination) will represent the proportion of common variation in the two
variables (i.e., the "strength" or "magnitude" of the relationship). In order to
evaluate the correlation between variables, it is important to know this "magnitude"
or "strength" as well as the significance of the correlation.
Significance of Correlations. The significance level calculated for each
correlation is a primary source of information about the reliability of the
correlation. As explained before (see Elementary Concepts), the significance of a
correlation coefficient of a particular magnitude will change depending on the size
of the sample from which it was computed. The test of significance is based on the
assumption that the distribution of the residual values (i.e., the deviations from the
regression line) for the dependent variable y follows the normal distribution, and
that the variability of the residual values is the same for all values of the
independent variable x. However, Monte Carlo studies suggest that meeting those
assumptions closely is not absolutely crucial if your sample size is not very small
and when the departure from normality is not very large. It is impossible to
formulate precise recommendations based on those Monte- Carlo results, but many
researchers follow a rule of thumb that if your sample size is 50 or more then
serious biases are unlikely, and if your sample size is over 100 then you should not
be concerned at all with the normality assumptions. There are, however, much
more common and serious threats to the validity of information that a correlation
coefficient can provide; they are briefly discussed in the following paragraphs.
Outliers. Outliers are atypical (by definition), infrequent observations. Because of
the way in which the regression line is determined (especially the fact that it is
based on minimizing not the sum of simple distances but the sum of squares of
distances of data points from the line), outliers have a profound influence on the
slope of the regression line and consequently on the value of the correlation
coefficient. A single outlier is capable of considerably changing the slope of the
regression line and, consequently, the value of the correlation, as demonstrated in
the following example. Note, that as shown on that illustration, just one outlier can
be entirely responsible for a high value of the correlation that otherwise (without
the outlier) would be close to zero. Needless to say, one should never base
important conclusions on the value of the correlation coefficient alone (i.e.,
examining the respective scatterplot is always recommended).
Note that if the sample size is relatively small, then including or excluding specific
data points that are not as clearly "outliers" as the one shown in the previous
example may have a profound influence on the regression line (and the correlation
coefficient). This is illustrated in the following example where we call the points
being excluded "outliers;" one may argue, however, that they are not outliers but
rather extreme values.

Typically, we believe that outliers represent a random error that we would like to
be able to control. Unfortunately, there is no widely accepted method to remove
outliers automatically (however, see the next paragraph), thus what we are left with
is to identify any outliers by examining a scatterplot of each important correlation.
Needless to say, outliers may not only artificially increase the value of a
correlation coefficient, but they can also decrease the value of a "legitimate"
correlation.
See also Confidence Ellipse.
Quantitative Approach to Outliers. Some researchers use quantitative methods
to exclude outliers. For example, they exclude observations that are outside the
range of ±2 standard deviations (or even ±1.5 sd's) around the group or design cell
mean. In some areas of research, such "cleaning" of the data is absolutely
necessary. For example, in cognitive psychology research on reaction times, even
if almost all scores in an experiment are in the range of 300-700 milliseconds, just
a few "distracted reactions" of 10-15 seconds will completely change the overall
picture. Unfortunately, defining an outlier is subjective (as it should be), and the
decisions concerning how to identify them must be made on an individual basis
(taking into account specific experimental paradigms and/or "accepted practice"
and general research experience in the respective area). It should also be noted that
in some rare cases, the relative frequency of outliers across a number of groups or
cells of a design can be subjected to analysis and provide interpretable results. For
example, outliers could be indicative of the occurrence of a phenomenon that is
qualitatively different than the typical pattern observed or expected in the sample,
thus the relative frequency of outliers could provide evidence of a relative
frequency of departure from the process or phenomenon that is typical for the
majority of cases in a group. See also Confidence Ellipse.
Correlations in Non-homogeneous Groups. A lack of homogeneity in the sample
from which a correlation was calculated can be another factor that biases the value
of the correlation. Imagine a case where a correlation coefficient is calculated from
data points which came from two different experimental groups but this fact is
ignored when the correlation is calculated. Let us assume that the experimental
manipulation in one of the groups increased the values of both correlated variables
and thus the data from each group form a distinctive "cloud" in the scatterplot (as
shown in the graph below).

In such cases, a high correlation may result that is entirely due to the arrangement
of the two groups, but which does not represent the "true" relation between the two
variables, which may practically be equal to 0 (as could be seen if we looked at
each group separately, see the following graph).
If you suspect the influence of such a phenomenon on your correlations and know
how to identify such "subsets" of data, try to run the correlations separately in each
subset of observations. If you do not know how to identify the hypothetical
subsets, try to examine the data with some exploratory multivariate techniques
(e.g., Cluster Analysis).
Nonlinear Relations between Variables. Another potential source of problems
with the linear (Pearson r) correlation is the shape of the relation. As mentioned
before, Pearson r measures a relation between two variables only to the extent to
which it is linear; deviations from linearity will increase the total sum of squared
distances from the regression line even if they represent a "true" and very close
relationship between two variables. The possibility of such non-linear relationships
is another reason why examining scatterplots is a necessary step in evaluating
every correlation. For example, the following graph demonstrates an extremely
strong correlation between the two variables which is not well described by the
linear function.
 Measuring Nonlinear Relations. What do you do if a correlation is strong but
clearly nonlinear (as concluded from examining scatterplots)? Unfortunately, there
is no simple answer to this question, because there is no easy-to-use equivalent
of Pearson r that is capable of handling nonlinear relations. If the curve is
monotonous (continuously decreasing or increasing) you could try to transform
one or both of the variables to remove the curvilinearity and then recalculate the
correlation. For example, a typical transformation used in such cases is the
logarithmic function which will "squeeze" together the values at one end of the
range. Another option available if the relation is monotonous is to try a
nonparametric correlation (e.g., Spearman R, see Nonparametrics and Distribution
Fitting) which is sensitive only to the ordinal arrangement of values, thus, by
definition, it ignores monotonous curvilinearity. However, nonparametric
correlations are generally less sensitive and sometimes this method will not
produce any gains. Unfortunately, the two most precise methods are not easy to use
and require a good deal of "experimentation" with the data. Therefore you could:
1. Try to identify the specific function that best describes the curve. After a
function has been found, you can test its "goodness-of-fit" to your data.
2. Alternatively, you could experiment with dividing one of the variables into a
number of segments (e.g., 4 or 5) of an equal width, treat this new variable as
a grouping variable and run an analysis of variance on the data.
Exploratory Examination of Correlation Matrices. A common first step of
many data analyses that involve more than a very few variables is to run a
correlation matrix of all variables and then examine it for expected (and
unexpected) significant relations. When this is done, you need to be aware of the
general nature of statistical significance (see Elementary Concepts); specifically, if
you run many tests (in this case, many correlations), then significant results will be
found "surprisingly often" due to pure chance. For example, by definition, a
coefficient significant at the .05 level will occur by chance once in every 20
coefficients. There is no "automatic" way to weed out the "true" correlations. Thus,
you should treat all results that were not predicted or planned with particular
caution and look for their consistency with other results; ultimately, though, the
most conclusive (although costly) control for such a randomness factor is to
replicate the study. This issue is general and it pertains to all analyses that involve
"multiple comparisons and statistical significance." This problem is also briefly
discussed in the context of post-hoc comparisons of means and
the Breakdowns option.
Casewise vs. Pairwise Deletion of Missing Data. The default way of deleting
missing data while calculating a correlation matrix is to exclude all cases that have
missing data in at least one of the selected variables; that is, by casewise
deletion of missing data. Only this way will you get a "true" correlation matrix,
where all correlations are obtained from the same set of observations. However, if
missing data are randomly distributed across cases, you could easily end up with
no "valid" cases in the data set, because each of them will have at least one missing
data in some variable. The most common solution used in such instances is to use
so-called pairwise deletion of missing data in correlation matrices, where a
correlation between each pair of variables is calculated from all cases that have
valid data on those two variables. In many instances there is nothing wrong with
that method, especially when the total percentage of missing data is low, say 10%,
and they are relatively randomly distributed between cases and variables.
However, it may sometimes lead to serious problems.
For example, a systematic bias may result from a "hidden" systematic distribution
of missing data, causing different correlation coefficients in the same correlation
matrix to be based on different subsets of subjects. In addition to the possibly
biased conclusions that you could derive from such "pairwise calculated"
correlation matrices, real problems may occur when you subject such matrices to
another analysis (e.g., multiple regression, factor analysis, or cluster analysis) that
expects a "true correlation matrix," with a certain level of consistency and
"transitivity" between different coefficients. Thus, if you are using the pairwise
method of deleting the missing data, be sure to examine the distribution of missing
data across the cells of the matrix for possible systematic "patterns."
How to Identify Biases Caused by the Bias due to Pairwise Deletion of Missing
Data. If the pairwise deletion of missing data does not introduce any systematic
bias to the correlation matrix, then all those pairwise descriptive statistics for one
variable should be very similar. However, if they differ, then there are good
reasons to suspect a bias. For example, if the mean (or standard deviation) of the
values of variable A that were taken into account in calculating its correlation with
variable B is much lower than the mean (or standard deviation) of those values of
 variable A that were used in calculating its correlation with variable C, then we
would have good reason to suspect that those two correlations (A-B and A-C) are
based on different subsets of data, and thus, that there is a bias in the correlation
matrix caused by a non-random distribution of missing data.
Pairwise Deletion of Missing Data vs. Mean Substitution. Another common
method to avoid loosing data due to casewise deletion is the so-called mean
substitution of missing data (replacing all missing data in a variable by the mean of
that variable). Mean substitution offers some advantages and some disadvantages
as compared to pairwise deletion. Its main advantage is that it produces "internally
consistent" sets of results ("true" correlation matrices). The main disadvantages
are:
1. Mean substitution artificially decreases the variation of scores, and this
decrease in individual variables is proportional to the number of missing data
(i.e., the more missing data, the more "perfectly average scores" will be
artificially added to the data set).
2. Because it substitutes missing data with artificially created "average" data
points, mean substitution may considerably change the values of correlations.
Spurious Correlations. Although you cannot prove causal relations based on
correlation coefficients (see Elementary Concepts), you can still identify so-
called spurious correlations; that is, correlations that are due mostly to the
influences of "other" variables. For example, there is a correlation between the
total amount of losses in a fire and the number of firemen that were putting out the
fire; however, what this correlation does not indicate is that if you call fewer
firemen then you would lower the losses. There is a third variable (the
initial size of the fire) that influences both the amount of losses and the number of
firemen. If you "control" for this variable (e.g., consider only fires of a fixed size),
then the correlation will either disappear or perhaps even change its sign. The main
problem with spurious correlations is that we typically do not know what the
"hidden" agent is. However, in cases when we know where to look, we can
use partial correlations that control for (partial out) the influence of specified
variables.
Are correlation coefficients "additive?" No, they are not. For example, an
average of correlation coefficients in a number of samples does not represent an
"average correlation" in all those samples. Because the value of the correlation
coefficient is not a linear function of the magnitude of the relation between the
variables, correlation coefficients cannot simply be averaged. In cases when you
need to average correlations, they first have to be converted into additive measures.
For example, before averaging, you can square them to obtain coefficients of
determination, which are additive (as explained before in this section), or convert
them into so-called Fisher z values, which are also additive.
How to Determine Whether Two Correlation Coefficients are Significant. A
test is available that will evaluate the significance of differences between two
correlation coefficients in two samples. The outcome of this test depends not only
on the size of the raw difference between the two coefficients but also on the size
of the samples and on the size of the coefficients themselves. Consistent with the
previously discussed principle, the larger the sample size, the smaller the effect
that can be proven significant in that sample. In general, due to the fact that the
reliability of the correlation coefficient increases with its absolute value, relatively
small differences between large correlation coefficients can be significant. For
example, a difference of .10 between two correlations may not be significant if the
two coefficients are .15 and .25, although in the same sample, the same difference
of .10 can be highly significant if the two coefficients are .80 and .90.

To index
 
t-Test for Independent Samples
Purpose, Assumptions. The t-test is the most commonly used method to evaluate
the differences in means between two groups. For example, the t-test can be used
to test for a difference in test scores between a group of patients who were given a
drug and a control group who received a placebo. Theoretically, the t-test can be
used even if the sample sizes are very small (e.g., as small as 10; some researchers
claim that even smaller n's are possible), as long as the variables are normally
distributed within each group and the variation of scores in the two groups is not
reliably different (see also Elementary Concepts). As mentioned before, the
normality assumption can be evaluated by looking at the distribution of the data
(via histograms) or by performing a normality test. The equality of variances
assumption can be verified with the F test, or you can use the more robust Levene's
test. If these conditions are not met, then you can evaluate the differences in means
between two groups using one of the nonparametric alternatives to the t- test
(see Nonparametrics and Distribution Fitting).
The p-level reported with a t-test represents the probability of error involved in
accepting our research hypothesis about the existence of a difference. Technically
speaking, this is the probability of error associated with rejecting the hypothesis of
no difference between the two categories of observations (corresponding to the
groups) in the population when, in fact, the hypothesis is true. Some researchers
suggest that if the difference is in the predicted direction, you can consider only
one half (one "tail") of the probability distribution and thus divide the standard p-
level reported with a t-test (a "two-tailed" probability) by two. Others, however,
suggest that you should always report the standard, two-tailed t-test probability.
See also, Student's t Distribution.
Arrangement of Data. In order to perform the t-test for independent samples, one
independent (grouping) variable (e.g., Gender: male/female) and at least one
dependent variable (e.g., a test score) are required. The means of the dependent
variable will be compared between selected groups based on the specified values
(e.g., male and female) of the independent variable. The following data set can be
analyzed with a t-test comparing the average WCC score in males and females.
 
  GENDER WCC
case 1 male 111
case 2 male 110
case 3 male 109
case 4 female 102
case 5 female 104
mean WCC in males = 110
 
mean WCC in females = 103

t-test graphs. In the t-test analysis, comparisons of means and measures of

variation in the two groups can be visualized in box and whisker plots (for an
example, see the graph below).
These graphs help you to quickly evaluate and "intuitively visualize" the strength
of the relation between the grouping and the dependent variable.
More Complex Group Comparisons. It often happens in research practice that
you need to compare more than two groups (e.g., drug 1, drug 2, and placebo), or
compare groups created by more than one independent variable while controlling
for the separate influence of each of them (e.g., Gender, type of Drug, and size of
Dose). In these cases, you need to analyze the data using Analysis of Variance,
which can be considered to be a generalization of the t-test. In fact, for two group
comparisons, ANOVA will give results identical to a t-test (t**2 [df] = F[1,df]).
However, when the design is more complex, ANOVA offers numerous advantages
that t-tests cannot provide (even if you run a series of t- tests comparing various
cells of the design).

To index
 
t-Test for Dependent Samples
Within-group Variation. As explained in Elementary Concepts, the size of a
relation between two variables, such as the one measured by a difference in means
between two groups, depends to a large extent on the differentiation of
values within the group. Depending on how differentiated the values are in each
group, a given "raw difference" in group means will indicate either a stronger or
weaker relationship between the independent (grouping) and dependent variable.
For example, if the mean WCC (White Cell Count) was 102 in males and 104 in
females, then this difference of "only" 2 points would be extremely important if all
values for males fell within a range of 101 to 103, and all scores for females fell
within a range of 103 to 105; for example, we would be able to predict WCC pretty
well based on gender. However, if the same difference of 2 was obtained from very
differentiated scores (e.g., if their range was 0-200), then we would consider the
difference entirely negligible. That is to say, reduction of the within-group
variation increases the sensitivity of our test.
Purpose. The t-test for dependent samples helps us to take advantage of one
specific type of design in which an important source of within-group variation (or
so-called, error) can be easily identified and excluded from the analysis.
Specifically, if two groups of observations (that are to be compared) are based on
the same sample of subjects who were tested twice (e.g., before and after a
treatment), then a considerable part of the within-group variation in both groups of
scores can be attributed to the initial individual differences between subjects. Note
that, in a sense, this fact is not much different than in cases when the two groups
are entirely independent (see t-test for independent samples), where individual
differences also contribute to the error variance; but in the case of independent
samples, we cannot do anything about it because we cannot identify (or "subtract")
the variation due to individual differences in subjects. However, if the same sample
was tested twice, then we can easily identify (or "subtract") this variation.
Specifically, instead of treating each group separately, and analyzing raw scores,
we can look only at the differences between the two measures (e.g., "pre-test" and
"post test") in each subject. By subtracting the first score from the second for each
subject and then analyzing only those "pure (paired) differences," we will exclude
the entire part of the variation in our data set that results from unequal base levels
of individual subjects. This is precisely what is being done in the t-test for
dependent samples, and, as compared to the t-test for independent samples, it
always produces "better" results (i.e., it is always more sensitive).
Assumptions. The theoretical assumptions of the t-test for independent
samples also apply to the dependent samples test; that is, the paired differences
should be normally distributed. If these assumptions are clearly not met, then one
of the nonparametric alternative tests should be used.
See also, Student's t Distribution.
Arrangement of Data. Technically, we can apply the t-test for dependent samples
to any two variables in our data set. However, applying this test will make very
little sense if the values of the two variables in the data set are not logically and
methodologically comparable. For example, if you compare the average WCC in a
sample of patients before and after a treatment, but using a different counting
method or different units in the second measurement, then a highly significant t-
test value could be obtained due to an artifact; that is, to the change of units of
measurement. Following, is an example of a data set that can be analyzed using
the t-test for dependent samples.
WCC WCC
 
before after
case 1 111.9 113
case 2 109 110
case 3 143 144
case 4 101 102
case 5 80 80.9
... ... ...
average change between WCC
 
"before" and "after" = 1

The average difference between the two conditions is relatively small (d=1) as
compared to the differentiation (range) of the raw scores (from 80 to 143, in the
first sample). However, the t-test for dependent samples analysis is performed only
on the paired differences , "ignoring" the raw scores and their potential
differentiation. Thus, the size of this particular difference of 1 will be compared
not to the differentiation of raw scores but to the differentiation of the individual
difference scores, which is relatively small: 0.2 (from 0.9 to 1.1). Compared to that
variability, the difference of 1 is extremely large and can yield a highly
significant t value.
Matrices of t-tests. t-tests for dependent samples can be calculated for long lists of
variables, and reviewed in the form of matrices produced
with casewise or pairwise deletion of missing data, much like the correlation
matrices. Thus, the precautions discussed in the context of correlations also apply
to t-test matrices; see:
1. the issue of artifacts caused by the pairwise deletion of missing data in t-
tests and
2. the issue of "randomly" significant test values.
More Complex Group Comparisons. If there are more than two "correlated
samples" (e.g., before treatment, after treatment 1, and after treatment 2), then
analysis of variance with repeated measures should be used. The repeated
measures ANOVA can be considered a generalization of the t-test for dependent
samples and it offers various features that increase the overall sensitivity of the
analysis. For example, it can simultaneously control not only for the base level of
the dependent variable, but it can control for other factors and/or include in the
design more than one interrelated dependent variable (MANOVA; for additional
details refer to ANOVA/MANOVA
To

ANOVA
Contents:
1. The ANOVA Test
2. One Way ANOVA
3. Two Way ANOVA
4. What is MANOVA?
5. What is Factorial ANOVA?
6. How to run an ANOVA
7. ANOVA vs. T Test
8. Repeated Measures ANOVA
9. Sphericity
10. Related Articles
The ANOVA Test
An ANOVA test is a way to find out if survey or experiment results are significant.
In other words, they help you to figure out if you need to reject the null
hypothesis or accept the alternate hypothesis. Basically, you’re testing groups to
see if there’s a difference between them. Examples of when you might want to test
different groups:
 A group of psychiatric patients are trying three different therapies:
counseling, medication and biofeedback. You want to see if one therapy is
better than the others.
 A manufacturer has two different processes to make light bulbs. They want
to know if one process is better than the other.
 Students from different colleges take the same exam. You want to see if one
college outperforms the other.
What Does “One-Way” or “Two-Way Mean?
One-way or two-way refers to the number of independent variables (IVs) in your
Analysis of Variance test. One-way has one independent variable (with 2 levels)
and two-way has two independent variables (can have multiple levels). For
example, a one-way Analysis of Variance could have one IV (brand of cereal) and
a two-way Analysis of Variance has two IVs (brand of cereal, calories).
What are “Groups” or “Levels”?
Groups or levels are different groups in the same independent variable. In the
above example, your levels for “brand of cereal” might be Lucky Charms, Raisin
Bran, Cornflakes — a total of three levels. Your levels for “Calories” might be:
sweetened, unsweetened — a total of two levels.
Let’s say you are studying if Alcoholics Anonymous and individual counseling
combined is the most effective treatment for lowering alcohol consumption. You
might split the study participants into three groups or levels: medication only,
medication and counseling, and counseling only. Your dependent variable would
be the number of alcoholic beverages consumed per day.
If your groups or levels have a hierarchical structure (each level has unique
subgroups), then use a nested ANOVA for the analysis.
What Does “Replication” Mean?
It’s whether you are replicating your test(s) with multiple groups. With a two way
ANOVA with replication , you have two groups and individuals within that group
are doing more than one thing (i.e. two groups of students from two colleges taking
two tests). If you only have one group taking two tests, you would use without
replication.
Types of Tests.
There are two main types: one-way and two-way. Two-way tests can be with or
without replication.

 One-way ANOVA between groups: used when you want to test two

groups to see if there’s a difference between them.
 Two way ANOVA without replication: used when you have one group and
you’re double-testing that same group. For example, you’re testing one set of
individuals before and after they take a medication to see if it works or not.
 Two way ANOVA with replication: Two groups, and the members of those
groups are doing more than one thing. For example, two groups of patients
from different hospitals trying two different therapies.
Back to Top
One Way ANOVA
A one way ANOVA is used to compare two means from two independent
(unrelated) groups using the F-distribution. The null hypothesis for the test is that
the two means are equal. Therefore, a significant result means that the two means
are unequal.
When to use a one way ANOVA
Situation 1: You have a group of individuals randomly split into smaller groups
and completing different tasks. For example, you might be studying the effects of
tea on weight loss and form three groups: green tea, black tea, and no tea.
Situation 2: Similar to situation 1, but in this case the individuals are split into
groups based on an attribute they possess. For example, you might be studying leg
strength of people according to weight. You could split participants into weight
categories (obese, overweight and normal) and measure their leg strength on a
weight machine.
Limitations of the One Way ANOVA
A one way ANOVA will tell you that at least two groups were different from each
other. But it won’t tell you what groups were different. If your test returns a
significant f-statistic, you may need to run an ad hoc test (like the Least Significant
Difference test) to tell you exactly which groups had a difference in means.
Back to Top
Two Way ANOVA
A Two Way ANOVA is an extension of the One Way ANOVA. With a One Way,
you have one independent variable affecting a dependent variable. With a Two
Way ANOVA, there are two independents. Use a two way ANOVA when you
have one measurement variable (i.e. a quantitative variable) and two nominal
variables. In other words, if your experiment has a quantitative outcome and you
have two categorical explanatory variables, a two way ANOVA is appropriate.
For example, you might want to find out if there is an interaction between income
and gender for anxiety level at job interviews. The anxiety level is the outcome, or
the variable that can be measured. Gender and Income are the two categorical
variables. These categorical variables are also the independent variables, which are
called factors in a Two Way ANOVA.
The factors can be split into levels. In the above example, income level could be
split into three levels: low, middle and high income. Gender could be split into
three levels: male, female, and transgender. Treatment groups are all possible
combinations of the factors. In this example there would be 3 x 3 = 9 treatment
groups.
Main Effect and Interaction Effect
The results from a Two Way ANOVA will calculate a main effect and
an interaction effect. The main effect is similar to a One Way ANOVA: each
factor’s effect is considered separately. With the interaction effect, all factors are
considered at the same time. Interaction effects between factors are easier to test if
there is more than one observation in each cell. For the above example, multiple
stress scores could be entered into cells. If you do enter multiple observations into
cells, the number in each cell must be equal.
Two null hypotheses are tested if you are placing one observation in each cell. For
this example, those hypotheses would be:
H01: All the income groups have equal mean stress.
H02: All the gender groups have equal mean stress.
For multiple observations in cells, you would also be testing a third hypothesis:
H03: The factors are independent or the interaction effect does not exist.
An F-statistic is computed for each hypothesis you are testing.
Assumptions for Two Way ANOVA
 The population must be close to a normal distribution.
 Samples must be independent.
 Population variances must be equal.
 Groups must have equal sample sizes.

MULTIPLE REGRESSION

Multiple Regression Analysis

Multiple regression analysis is used to see if there is a statistically
significant relationship between sets of variables. It’s used to find trends in those
sets of data.
Multiple regression analysis is almost the same as simple linear regression. The
only difference between simple linear regression and multiple regression is in the
number of predictors (“x” variables) used in the regression.
 Simple regression analysis uses a single x variable for each dependent “y”
variable. For example: (x1, Y1).
 Multiple regression uses multiple “x” variables for each independent
variable: (x1)1, (x2)1, (x3)1, Y1).
In one-variable linear regression, you would input one dependent variable (i.e.
“sales”) against an independent variable (i.e. “profit”). But you might be interested
in how different types of sales effect the regression. You could set your X1 as one
type of sales, your X2 as another type of sales and so on.
When to Use Multiple Regression Analysis.
Ordinary linear regression usually isn’t enough to take into account all of the real-
life factors that have an effect on an outcome. For example, the following graph
plots a single variable (number of doctors) against another variable (life-
expectancy of women).

Image: Columbia University

From this graph it might appear there is a relationship between life-expectancy of

women and the number of doctors in the population. In fact, that’s probably true
and you could say it’s a simple fix: put more doctors into the population to increase
life expectancy. But the reality is you would have to look at other factors like the
possibility that doctors in rural areas might have less education or experience. Or
perhaps they have a lack of access to medical facilities like trauma centers.
The addition of those extra factors would cause you to add additional dependent
variables to your regression analysis and create a multiple regression analysis
model.
Multiple Regression Analysis Output.
Regression analysis is always performed in software, like Excel or SPSS. The
output differs according to how many variables you have but it’s essentially the
same type of output you would find in a simple linear regression. There’s just more
of it:

 Simple regression: Y = b0 + b1 x.

 Multiple regression: Y = b0 + b1 x1 + b0 + b1 x2…b0…b1 xn.
The output would include a summary, similar to a summary for simple linear
regression, that includes:

 R (the multiple correlation coefficient),

 R squared (the coefficient of determination),
 adjusted R-squared,
 The standard error of the estimate.
These statistics help you figure out how well a regression model fits the data.
The ANOVA table in the output would give you the p-value and f-statistic.

Read the results. As well as creating a regression graph, Minitab will give you
values for S, R-sq and R-sq(adj) in the top right corner of the fitted line plot
window.
s = standard error.
R-Sq = Coefficient of Determination
R-Sq(adj) = Adjusted Coefficient of Determination (Adjusted R Squared).

AUTOCORRELATION TESTS
What Is the Durbin Watson Statistic?
The Durbin Watson (DW) statistic is a test for autocorrelation in the residuals from
a statistical regression analysis. The Durbin-Watson statistic will always have a
value between 0 and 4. A value of 2.0 means that there is no autocorrelation
detected in the sample. Values from 0 to less than 2 indicate positive
autocorrelation and values from from 2 to 4 indicate negative autocorrelation.

A stock price displaying positive autocorrelation would indicate that the price
yesterday has a positive correlation on the price today—so if the stock fell
yesterday, it is also likely that it falls today. A security that has a negative
autocorrelation, on the other hand, has a negative influence on itself over time—so
that if it fell yesterday, there is a greater likelihood it will rise today

KEY TAKEAWAYS

 The Durbin Watson statistic is a test for autocorrelation in a data set.

 The DW statistic always has a value between zero and 4.0.
 A value of 2.0 means there is no autocorrelation detected in the sample.
Values from zero to 2.0 indicate positive autocorrelation and values
from 2.0 to 4.0 indicate negative autocorrelation.
 Autocorrelation can be useful in technical analysis, which is most
concerned with the trends of security prices using charting techniques
in lieu of a company's financial health or management.

The Durbin Watson test reports a test statistic, with a value from 0 to 4,
where:

 2 is no autocorrelation.
 0 to <2 is positive autocorrelation (common in time series data).
 >2 to 4 is negative autocorrelation (less common in time series data).
A rule of thumb is that test statistic values in the range of 1.5 to 2.5 are
relatively normal. Values outside of this range could be cause for concern.
Field(2009) suggests that values under 1 or more than 3 are a definite cause
for concern.

The Breusch-Pagan-Godfrey Test

(sometimes shorted to the Breusch-Pagan test) is a test for heteroscedasticity of
errors in regression. Heteroscedasticity means “differently scattered”; this is
opposite to homoscedastic, which means “same scatter.” Homoscedasticity in
regression is an important assumption; if the assumption is violated, you won’t be
able to use regression analysis.
Running the Test
The test statistic for the Breusch-Pagan-Godfrey test is:
N * R2 (with k degrees of freedom)
Where:
n = sample size
R2 = R2(Coefficient of Determination) of the regression of squared residuals from
the original regression.
k = number of independent variables.
The test statistic approximately follows a chi-square distribution.
The null hypothesis for this test is that the error variances are all equal.
The alternate hypothesis is that the error variances are not equal. More specifically,
as Y increases, the variances increase (or decrease).
A small chi-square value (along with an associated small p-value) indicates the
null hypothesis is true (i.e. that the variances are all equal).
Note that the Breush-Pagan test measures how errors increase across
the explanatory variable, Y. The test assumes the error variances are due to
a linear function of one or more explanatory variables in the model. That means
heteroskedasticity could still be present in your regression model, but those errors
(if present) are not correlated with the Y-values.
If you suspect that a small subset of values is causing heteroskedasticity, you can
run a modified Breusch-Pagan test on those values alone

What is a Variance Inflation Factor?

A variance inflation factor(VIF) detects multicollinearity in regression analysis.
Multicollinearity is when there’s correlation between predictors (i.e. independent
variables) in a model; it’s presence can adversely affect your regression results.
The VIF estimates how much the variance of a regression coefficient is inflated
due to multicollinearity in the model.
VIFs are usually calculated by software, as part of regression analysis. You’ll see a
VIF column as part of the output. VIFs are calculated by taking a predictor, and
regressing it against every other predictor in the model. This gives you the R-
squared values, which can then be plugged into the VIF formula. “i” is the
predictor you’re looking at (e.g. x1 or x2):
Interpreting the Variance Inflation Factor
Variance inflation factors range from 1 upwards. The numerical value for VIF tells
you (in decimal form) what percentage the variance (i.e. the standard
error squared) is inflated for each coefficient. For example, a VIF of 1.9 tells you
that the variance of a particular coefficient is 90% bigger than what you would
expect if there was no multicollinearity — if there was no correlation with other
predictors.
A rule of thumb for interpreting the variance inflation factor:
 1 = not correlated.
 Between 1 and 5 = moderately correlated.
 Greater than 5 = highly correlated.
Exactly how large a VIF has to be before it causes issues is a subject of debate.
What is known is that the more your VIF increases, the less reliable your
regression results are going to be. In general, a VIF above 10 indicates high
correlation and is cause for concern. Some authors suggest a more conservative
level of 2.5 or above.
Sometimes a high VIF is no cause for concern at all. For example, you can get a
high VIF by including products or powers from other variables in your regression,
like x and x2. If you have high VIFs for dummy variables representing nominal
variables with three or more categories, those are usually not a problem.

Regression Analysis 

Regression analysis is a widely used technique which is useful for evaluating

multiple independent variables. As a result, it is particularly useful for assess and
adjusting for confounding. It can also be used to assess the presence of effect
modification. Interested readers should see Kleinbaum, Kupper and Muller for
more details on regression analysis and its many applications.1

Simple Linear Regression 

Suppose we want to assess the association between total cholesterol and body mass
index (BMI) in which total cholesterol is the dependent variable, and BMI is the
independent variable. In regression analysis, the dependent variable is denoted Y
and the independent variable is denoted X. So, in this case, Y=total cholesterol and
X=BMI.

When there is a single continuous dependent variable and a single independent

variable, the analysis is called a simple linear regression analysis. This analysis
assumes that there is a linear association between the two variables. (If a different
relationship is hypothesized, such as a curvilinear or exponential relationship,
alternative regression analyses are performed.)

The simple linear regression equation is as follows:

 , where

 is the predicted or expected value of the outcome, X is the predictor , b0 is the
estimated Y-intercept, and b1 is the estimated slope. The Y-intercept and slope are
estimated from the sample data so as to minimize the sum of the squared
differences between the observed and the predicted values of the outcome, i.e., the
estimates minimize:

These differences between observed and predicted values of the outcome are
called residuals. The estimates of the Y-intercept and slope minimize the sum of
the squared residuals, and are called the least squares estimates.1

Multiple Linear Regression Analysis

Multiple linear regression analysis is an extension of simple linear regression

analysis, used to assess the association between two or more independent variables
and a single continuous dependent variable. The multiple linear regression
equation is as follows:

   ,

 where is the predicted or expected value of the dependent variable, X1 through

Xp are p distinct independent or predictor variables, b0 is the value of Y when all of
the independent variables (X1 through Xp) are equal to zero, and b1 through bp are
the estimated regression coefficients. Each regression coefficient represents the
change in Y relative to a one unit change in the respective independent variable. In
the multiple regression situation, b1, for example, is the change in Y relative to a
one unit change in X1, holding all other independent variables constant (i.e., when
the remaining independent variables are held at the same value or are fixed).
Again, statistical tests can be performed to assess whether each regression
coefficient is significantly different from zero.

Controlling for Confounding With Multiple Linear Regression

Multiple regression analysis is also used to assess whether confounding exists.

Since multiple linear regression analysis allows us to estimate the association
between a given independent variable and the outcome holding all other variables
constant, it provides a way of adjusting for (or accounting for) potentially
confounding variables that have been included in the model.

Suppose we have a risk factor or an exposure variable, which we denote X1 (e.g.,

X1=obesity or X1=treatment), and an outcome or dependent variable which we
denote Y. We can estimate a simple linear regression equation relating the risk
factor (the independent variable) to the dependent variable as follows:

where b1 is the estimated regression coefficient that quantifies the association
between the risk factor and the outcome.

Suppose we now want to assess whether a third variable (e.g., age) is a

confounder. We denote the potential confounder X2, and then estimate a multiple
linear regression equation as follows:

 .

In the multiple linear regression equation, b1 is the estimated regression coefficient
that quantifies the association between the risk factor X1 and the outcome, adjusted
for X2 (b2 is the estimated regression coefficient that quantifies the association
between the potential confounder and the outcome). As noted earlier, some
investigators assess confounding by assessing how much the regression coefficient
associated with the risk factor (i.e., the measure of association) changes after
adjusting for the potential confounder. In this case, we compare b1 from the simple
linear regression model to b1 from the multiple linear regression model. As a rule
of thumb, if the regression coefficient from the simple linear regression model
changes by more than 10%, then X2 is said to be a confounder.
Once a variable is identified as a confounder, we can then use multiple linear
regression analysis to estimate the association between the risk factor and the
outcome adjusting for that confounder. The test of significance of the regression
coefficient associated with the risk factor can be used to assess whether the
association between the risk factor is statistically significant after accounting for
one or more confounding variables. This is also illustrated below.

Dummy" Variables in Regression Models 

Independent variables in regression models can be continuous or dichotomous.

Regression models can also accommodate categorical independent variables. For
example, it might be of interest to assess whether there is a difference in total
cholesterol by race/ethnicity. The module on Hypothesis Testing presented
analysis of variance as one way of testing for differences in means of a continuous
outcome among several comparison groups. Regression analysis can also be used.
However, the investigator must create indicator variables to represent the different
comparison groups (e.g., different racial/ethnic groups). The set of indicator
variables (also called dummy variables) are considered in the multiple regression
model simultaneously as a set independent variables.

Autocorrelation
Informally, autocorrelation is the similarity between observations as a function of
the time lag between them.
Example of an autocorrelation plot

Above is an example of an autocorrelation plot. Looking closely, you realize that

the first value and the 24th value have a high autocorrelation. Similarly, the 12th
and 36th observations are highly correlated. This means that we will find a very
similar value at every 24 unit of time.

Notice how the plot looks like sinusoidal function. This is a hint
for seasonality, and you can find its value by finding the period in the plot above,
which would give 24h.

Seasonality
Seasonality refers to periodic fluctuations. For example, electricity consumption is
high during the day and low during night, or online sales increase during Christmas
before slowing down again.
Example of seasonality

As you can see above, there is a clear daily seasonality. Every day, you see a peak
towards the evening, and the lowest points are the beginning and the end of each
day.

Remember that seasonality can also be derived from an autocorrelation plot if it has
a sinusoidal shape. Simply look at the period, and it gives the length of the season.

Stationarity
Stationarity is an important characteristic of time series. A time series is said to be
stationary if its statistical properties do not change over time. In other words, it
has constant mean and variance, and covariance is independent of time.
Example of a stationary process

Looking again at the same plot, we see that the process above is stationary. The
mean and variance do not vary over time.

Often, stock prices are not a stationary process, since we might see a growing trend,
or its volatility might increase over time (meaning that variance is changing).

Ideally, we want to have a stationary time series for modelling. Of course, not all of
them are stationary, but we can make different transformations to make them
stationary.

How to test if a process is stationary

You may have noticed in the title of the plot above Dickey-Fuller. This is the
statistical test that we run to determine if a time series is stationary or not.

Without going into the technicalities of the Dickey-Fuller test, it test the null
hypothesis that a unit root is present.

If it is, then p > 0, and the process is not stationary.

Otherwise, p = 0, the null hypothesis is rejected, and the process is considered to be
stationary.

As an example, the process below is not stationary. Notice how the mean is not
constant through time.
Example of a non-stationary process

Modelling time series

There are many ways to model a time series in order to make predictions. Here, I
will present:

 moving average

 exponential smoothing

 ARIMA

Moving average
The moving average model is probably the most naive approach to time series
modelling. This model simply states that the next observation is the mean of all past
observations.

Although simple, this model might be surprisingly good and it represents a good
starting point.

Otherwise, the moving average can be used to identify interesting trends in the data.
We can define a window to apply the moving average model to smooth the time
series, and highlight different trends.
Example of a moving average on a 24h window

In the plot above, we applied the moving average model to a 24h window. The
green line smoothed the time series, and we can see that there are 2 peaks in a 24h
period.

Of course, the longer the window, the smoother the trend will be. Below is an
example of moving average on a smaller window.

Example of a moving average on a 12h window

Exponential smoothing
Exponential smoothing uses a similar logic to moving average, but this time, a
different decreasing weight is assigned to each observations. In other words, less
importance is given to observations as we move further from the present.

Mathematically, exponential smoothing is expressed as:

Exponential smoothing expression

Here, alpha is a smoothing factor that takes values between 0 and 1. It determines

how fast the weight decreases for previous observations.

Example of exponential smoothing

From the plot above, the dark blue line represents the exponential smoothing of the
time series using a smoothing factor of 0.3, while the orange line uses a smoothing
factor of 0.05.

As you can see, the smaller the smoothing factor, the smoother the time series will
be. This makes sense, because as the smoothing factor approaches 0, we approach
the moving average model.

Double exponential smoothing

Double exponential smoothing is used when there is a trend in the time series. In
that case, we use this technique, which is simply a recursive use of exponential
smoothing twice.

Mathematically:
Double exponential smoothing expression

Here, beta is the trend smoothing factor, and it takes values between 0 and 1.

Below, you can see how different values of alpha and beta affect the shape of the
time series.

Example of double exponential smoothing

Tripe exponential smoothing

This method extends double exponential smoothing, by adding a seasonal
smoothing factor. Of course, this is useful if you notice seasonality in your time
series.

Mathematically, triple exponential smoothing is expressed as:

Triple exponential smoothing expression

Where gamma is the seasonal smoothing factor and L is the length of the season.
 TIME SERIES

General Introduction

In the following topics, we will review techniques that are useful for analyzing
time series data, that is, sequences of measurements that follow non-random
orders. Unlike the analyses of random samples of observations that are discussed in
the context of most other statistics, the analysis of time series is based on the
assumption that successive values in the data file represent consecutive
measurements taken at equally spaced time intervals.
Detailed discussions of the methods described in this section can be found in
Anderson (1976), Box and Jenkins (1976), Kendall (1984), Kendall and Ord
(1990), Montgomery, Johnson, and Gardiner (1990), Pankratz (1983), Shumway
(1988), Vandaele (1983), Walker (1991), and Wei (1989).
  To index

Two Main Goals

There are two main goals of time series analysis: (a) identifying the nature of the
phenomenon represented by the sequence of observations, and (b) forecasting
(predicting future values of the time series variable). Both of these goals require
that the pattern of observed time series data is identified and more or less formally
described. Once the pattern is established, we can interpret and integrate it with
other data (i.e., use it in our theory of the investigated phenomenon, e.g., seasonal
commodity prices). Regardless of the depth of our understanding and the validity
of our interpretation (theory) of the phenomenon, we can extrapolate the identified
pattern to predict future events.
  To index

Identifying Patterns in Time Series Data

 Systematic pattern and random noise
 Two general aspects of time series patterns
 Trend Analysis
 Analysis of Seasonality
 For more information on simple autocorrelations (introduced in this section) and
other auto correlations, see Anderson (1976), Box and Jenkins (1976), Kendall
(1984), Pankratz (1983), and Vandaele (1983). See also:
 ARIMA (Box & Jenkins) and Autocorrelations
 Interrupted Time Series
 Exponential Smoothing
 Seasonal Decomposition (Census I)
 X-11 Census method II seasonal adjustment
 X-11 Census method II result tables
 Distributed Lags Analysis
 Single Spectrum (Fourier) Analysis
 Cross-spectrum Analysis
 Basic Notations and Principles
 Fast Fourier Transformations
SYSTEMATIC PATTERN AND RANDOM NOISE
As in most other analyses, in time series analysis it is assumed that the data consist
of a systematic pattern (usually a set of identifiable components) and random noise
(error) which usually makes the pattern difficult to identify. Most time series
analysis techniques involve some form of filtering out noise in order to make the
pattern more salient.
TWO GENERAL ASPECTS OF TIME SERIES PATTERNS
Most time series patterns can be described in terms of two basic classes of
components: trend and seasonality. The former represents a general systematic
linear or (most often) nonlinear component that changes over time and does not
repeat or at least does not repeat within the time range captured by our data (e.g., a
plateau followed by a period of exponential growth). The latter may have a
formally similar nature (e.g., a plateau followed by a period of exponential
growth), however, it repeats itself in systematic intervals over time. Those two
general classes of time series components may coexist in real-life data. For
example, sales of a company can rapidly grow over years but they still follow
consistent seasonal patterns (e.g., as much as 25% of yearly sales each year are
made in December, whereas only 4% in August).
This general pattern is well illustrated in a "classic" Series G data set (Box and
Jenkins, 1976, p. 531) representing monthly international airline passenger totals
(measured in thousands) in twelve consecutive years from 1949 to 1960 (see
example data file G.sta and graph above). If you plot the successive observations
(months) of airline passenger totals, a clear, almost linear trend emerges, indicating
that the airline industry enjoyed a steady growth over the years (approximately 4
times more passengers traveled in 1960 than in 1949). At the same time, the
monthly figures will follow an almost identical pattern each year (e.g., more people
travel during holidays than during any other time of the year). This example data
file also illustrates a very common general type of pattern in time series data,
where the amplitude of the seasonal changes increases with the overall trend (i.e.,
the variance is correlated with the mean over the segments of the series). This
pattern which is called multiplicative seasonality indicates that the relative
amplitude of seasonal changes is constant over time, thus it is related to the trend.
TREND ANALYSIS
There are no proven "automatic" techniques to identify trend components in the
time series data; however, as long as the trend is monotonous (consistently
increasing or decreasing) that part of data analysis is typically not very difficult. If
the time series data contain considerable error, then the first step in the process of
trend identification is smoothing.
Smoothing. Smoothing always involves some form of local averaging of data such
that the nonsystematic components of individual observations cancel each other
out. The most common technique is moving average smoothing which replaces
each element of the series by either the simple or weighted average
of n surrounding elements, where n is the width of the smoothing "window" (see
Box & Jenkins, 1976; Velleman & Hoaglin, 1981). Medians can be used instead of
means. The main advantage of median as compared to moving average smoothing
is that its results are less biased by outliers (within the smoothing window). Thus,
if there are outliers in the data (e.g., due to measurement errors), median smoothing
typically produces smoother or at least more "reliable" curves than moving average
based on the same window width. The main disadvantage of median smoothing is
that in the absence of clear outliers it may produce more "jagged" curves than
moving average and it does not allow for weighting.
In the relatively less common cases (in time series data), when the measurement
error is very large, the distance weighted least squares smoothing or negative
exponentially weighted smoothing techniques can be used. All those methods will
filter out the noise and convert the data into a smooth curve that is relatively
unbiased by outliers (see the respective sections on each of those methods for more
details). Series with relatively few and systematically distributed points can be
smoothed with bicubic splines.
Fitting a function. Many monotonous time series data can be adequately
approximated by a linear function; if there is a clear monotonous nonlinear
component, the data first need to be transformed to remove the nonlinearity.
Usually a logarithmic, exponential, or (less often) polynomial function can be
used.

ANALYSIS OF SEASONALITY

Seasonal dependency (seasonality) is another general component of the time series

pattern. The concept was illustrated in the example of the airline passengers data
above. It is formally defined as correlational dependency of order k between
each i'th element of the series and the (i-k)'th element (Kendall, 1976) and
measured by autocorrelation (i.e., a correlation between the two terms); k is usually
called the lag. If the measurement error is not too large, seasonality can be visually
identified in the series as a pattern that repeats every k elements.
Autocorrelation correlogram. Seasonal patterns of time series can be examined
via correlograms. The correlogram (autocorrelogram) displays graphically and
numerically the autocorrelation function (ACF), that is, serial correlation
coefficients (and their standard errors) for consecutive lags in a specified range of
lags (e.g., 1 through 30). Ranges of two standard errors for each lag are usually
marked in correlograms but typically the size of auto correlation is of more interest
than its reliability (see Elementary Concepts) because we are usually interested
only in very strong (and thus highly significant) autocorrelations.
Examining correlograms. While examining correlograms, you should keep in
mind that autocorrelations for consecutive lags are formally dependent. Consider
the following example. If the first element is closely related to the second, and the
second to the third, then the first element must also be somewhat related to the
third one, etc. This implies that the pattern of serial dependencies can change
considerably after removing the first order auto correlation (i.e., after differencing
the series with a lag o0f 1).

Partial autocorrelations. Another useful method to examine serial dependencies

is to examine the partial autocorrelation function (PACF) - an extension of
autocorrelation, where the dependence on the intermediate elements
(those within the lag) is removed. In other words the partial autocorrelation is
similar to autocorrelation, except that when calculating it, the (auto) correlations
with all the elements within the lag are partialled out (Box & Jenkins, 1976; see
also McDowall, McCleary, Meidinger, & Hay, 1980). If a lag of 1 is specified (i.e.,
there are no intermediate elements within the lag), then the partial autocorrelation
is equivalent to auto correlation. In a sense, the partial autocorrelation provides a
"cleaner" picture of serial dependencies for individual lags (not confounded by
other serial dependencies).
Removing serial dependency. Serial dependency for a particular lag of k can be
removed by differencing the series, that is converting each i'th element of the series
into its difference from the (i-k)''th element. There are two major reasons for such
transformations.
First, we can identify the hidden nature of seasonal dependencies in the series.
Remember that, as mentioned in the previous paragraph, autocorrelations for
consecutive lags are interdependent. Therefore, removing some of the
autocorrelations will change other auto correlations, that is, it may eliminate them
or it may make some other seasonalities more apparent.
The other reason for removing seasonal dependencies is to make the
series stationary which is necessary for ARIMA and other techniques.
ARIMA To index

GENERAL INTRODUCTION
The modeling and forecasting procedures discussed in Identifying Patterns in Time
Series Data involved knowledge about the mathematical model of the process.
However, in real-life research and practice, patterns of the data are unclear,
individual observations involve considerable error, and we still need not only to
uncover the hidden patterns in the data but also generate forecasts. The ARIMA
methodology developed by Box and Jenkins (1976) allows us to do just that; it has
gained enormous popularity in many areas and research practice confirms its
power and flexibility (Hoff, 1983; Pankratz, 1983; Vandaele, 1983). However,
because of its power and flexibility, ARIMA is a complex technique; it is not easy
to use, it requires a great deal of experience, and although it often produces
satisfactory results, those results depend on the researcher's level of expertise
(Bails & Peppers, 1982). The following sections will introduce the basic ideas of
this methodology. For those interested in a brief, applications-oriented (non-
mathematical), introduction to ARIMA methods, we recommend McDowall,
McCleary, Meidinger, and Hay (1980).

TWO COMMON PROCESSES

Autoregressive process. Most time series consist of elements that are serially

dependent in the sense that you can estimate a coefficient or a set of coefficients
that describe consecutive elements of the series from specific, time-lagged
(previous) elements. This can be summarized in the equation: xt =   +  1*x(t-1) + 
2*x(t-2) +  3*x(t-3) + ... + 

Where:
                  is a constant (intercept), and
  1,  2,  3   are the autoregressive model parameters.
Put into words, each observation is made up of a random error component (random
shock,  ) and a linear combination of prior observations.
Stationarity requirement. Note that an autoregressive process will only be stable
if the parameters are within a certain range; for example, if there is only one
autoregressive parameter then is must fall within the interval of -1 <   < 1.
Otherwise, past effects would accumulate and the values of successive xt' s would
move towards infinity, that is, the series would not be stationary. If there is more
than one autoregressive parameter, similar (general) restrictions on the parameter
values can be defined (e.g., see Box & Jenkins, 1976; Montgomery, 1990).
Moving average process. Independent from the autoregressive process, each
element in the series can also be affected by the past error (or random shock) that
cannot be accounted for by the autoregressive component, that is:
xt = µ +  t -  1* (t-1) -  2* (t-2) -  3* (t-3) - ...
Where:
 µ                is a constant, and
  1,  2,  3  are the moving average model parameters.
Put into words, each observation is made up of a random error component (random
shock,  ) and a linear combination of prior random shocks.
Invertibility requirement. Without going into too much detail, there is a "duality"
between the moving average process and the autoregressive process (e.g., see Box
& Jenkins, 1976; Montgomery, Johnson, & Gardiner, 1990), that is, the moving
average equation above can be rewritten (inverted) into an autoregressive form (of
infinite order). However, analogous to the stationarity condition described above,
this can only be done if the moving average parameters follow certain conditions,
that is, if the model is invertible. Otherwise, the series will not be stationary.

ARIMA METHODOLOGY

Autoregressive moving average model. The general model introduced by Box

and Jenkins (1976) includes autoregressive as well as moving average parameters,
and explicitly includes differencing in the formulation of the model. Specifically,
the three types of parameters in the model are: the autoregressive parameters (p),
the number of differencing passes (d), and moving average parameters (q). In the
notation introduced by Box and Jenkins, models are summarized as ARIMA (p, d,
q); so, for example, a model described as (0, 1, 2) means that it contains 0 (zero)
autoregressive (p) parameters and 2 moving average (q) parameters which were
computed for the series after it was differenced once.
Identification. As mentioned earlier, the input series for ARIMA needs to
be stationary, that is, it should have a constant mean, variance, and autocorrelation
through time. Therefore, usually the series first needs to be differenced until it
is stationary (this also often requires log transforming the data to stabilize the
variance). The number of times the series needs to be differenced to achieve
stationarity is reflected in the d parameter (see the previous paragraph). In order to
determine the necessary level of differencing, you should examine the plot of the
data and autocorrelogram. Significant changes in level (strong upward or
downward changes) usually require first order non seasonal (lag=1) differencing;
strong changes of slope usually require second order non seasonal differencing.
Seasonal patterns require respective seasonal differencing (see below). If the
estimated autocorrelation coefficients decline slowly at longer lags, first order
differencing is usually needed. However, you should keep in mind that some time
series may require little or no differencing, and that over differenced series produce
less stable coefficient estimates.
At this stage (which is usually called Identification phase, see below) we also need
to decide how many autoregressive (p) and moving average (q) parameters are
necessary to yield an effective but still parsimonious model of the process
(parsimonious means that it has the fewest parameters and greatest number of
degrees of freedom among all models that fit the data). In practice, the numbers of
the p or q parameters very rarely need to be greater than 2 (see below for more
specific recommendations).
Estimation and Forecasting. At the next step (Estimation), the parameters are
estimated (using function minimization procedures, see below; for more
information on minimization procedures see also Nonlinear Estimation), so that the
sum of squared residuals is minimized. The estimates of the parameters are used in
the last stage (Forecasting) to calculate new values of the series (beyond those
included in the input data set) and confidence intervals for those predicted values.
The estimation process is performed on transformed (differenced) data; before the
forecasts are generated, the series needs to be integrated (integration is the inverse
of differencing) so that the forecasts are expressed in values compatible with the
input data. This automatic integration feature is represented by the letter I in the
name of the methodology (ARIMA = Auto-Regressive Integrated Moving
Average).
The constant in ARIMA models. In addition to the standard autoregressive and
moving average parameters, ARIMA models may also include a constant, as
described above. The interpretation of a (statistically significant) constant depends
on the model that is fit. Specifically, (1) if there are no autoregressive parameters
in the model, then the expected value of the constant is  , the mean of the series;
(2) if there are autoregressive parameters in the series, then the constant represents
the intercept. If the series is differenced, then the constant represents the mean or
intercept of the differenced series; For example, if the series is differenced once,
and there are no autoregressive parameters in the model, then the constant
represents the mean of the differenced series, and therefore the linear trend
slope of the un-differenced series.
IDENTIFICATION PHASE

Number of parameters to be estimated. Before the estimation can begin, we

need to decide on (identify) the specific number and type of ARIMA parameters to
be estimated. The major tools used in the identification phase are plots of the
series, correlograms of auto correlation (ACF), and partial autocorrelation (PACF).
The decision is not straightforward and in less typical cases requires not only
experience but also a good deal of experimentation with alternative models (as
well as the technical parameters of ARIMA). However, a majority of empirical
time series patterns can be sufficiently approximated using one of the 5 basic
models that can be identified based on the shape of the autocorrelogram (ACF) and
partial auto correlogram (PACF). The following brief summary is based on
practical recommendations of Pankratz (1983); for additional practical advice, see
also Hoff (1983), McCleary and Hay (1980), McDowall, McCleary, Meidinger,
and Hay (1980), and Vandaele (1983). Also, note that since the number of
parameters (to be estimated) of each kind is almost never greater than 2, it is often
practical to try alternative models on the same data.
1. One autoregressive (p) parameter: ACF - exponential decay; PACF - spike
at lag 1, no correlation for other lags.
2. Two autoregressive (p) parameters: ACF - a sine-wave shape pattern or a set
of exponential decays; PACF - spikes at lags 1 and 2, no correlation for other
lags.
3. One moving average (q) parameter: ACF - spike at lag 1, no correlation for
other lags; PACF - damps out exponentially.
4. Two moving average (q) parameters: ACF - spikes at lags 1 and 2, no
correlation for other lags; PACF - a sine-wave shape pattern or a set of
exponential decays.
5. One autoregressive (p) and one moving average (q) parameter: ACF -
exponential decay starting at lag 1; PACF - exponential decay starting at lag 1.
Seasonal models. Multiplicative seasonal ARIMA is a generalization and
extension of the method introduced in the previous paragraphs to series in which a
pattern repeats seasonally over time. In addition to the non-seasonal parameters,
seasonal parameters for a specified lag (established in the identification phase)
need to be estimated. Analogous to the simple ARIMA parameters, these are:
seasonal autoregressive (ps), seasonal differencing (ds), and seasonal moving
average parameters (qs). For example, the model (0,1,2)(0,1,1) describes a model
that includes no autoregressive parameters, 2 regular moving average parameters
and 1 seasonal moving average parameter, and these parameters were computed
for the series after it was differenced once with lag 1, and once seasonally
differenced. The seasonal lag used for the seasonal parameters is usually
determined during the identification phase and must be explicitly specified.
The general recommendations concerning the selection of parameters to be
estimated (based on ACF and PACF) also apply to seasonal models. The main
difference is that in seasonal series, ACF and PACF will show sizable coefficients
at multiples of the seasonal lag (in addition to their overall patterns reflecting the
non seasonal components of the series).

PARAMETER ESTIMATION

There are several different methods for estimating the parameters. All of them
should produce very similar estimates, but may be more or less efficient for any
given model. In general, during the parameter estimation phase a function
minimization algorithm is used (the so-called quasi-Newton method; refer to the
description of the Nonlinear Estimation method) to maximize the likelihood
(probability) of the observed series, given the parameter values. In practice, this
requires the calculation of the (conditional) sums of squares (SS) of the residuals,
given the respective parameters. Different methods have been proposed to compute
the SS for the residuals: (1) the approximate maximum likelihood method
according to McLeod and Sales (1983), (2) the approximate maximum likelihood
method with backcasting, and (3) the exact maximum likelihood method according
to Melard (1984).
Comparison of methods. In general, all methods should yield very similar
parameter estimates. Also, all methods are about equally efficient in most real-
world time series applications. However, method 1 above, (approximate maximum
likelihood, no backcasts) is the fastest, and should be used in particular for very
long time series (e.g., with more than 30,000 observations). Melard's exact
maximum likelihood method (number 3 above) may also become inefficient when
used to estimate parameters for seasonal models with long seasonal lags (e.g., with
yearly lags of 365 days). On the other hand, you should always use the
approximate maximum likelihood method first in order to establish initial
parameter estimates that are very close to the actual final values; thus, usually only
a few iterations with the exact maximum likelihood method (3, above) are
necessary to finalize the parameter estimates.
Parameter standard errors. For all parameter estimates, you will compute so-
called asymptotic standard errors. These are computed from the matrix of second-
order partial derivatives that is approximated via finite differencing (see also the
respective discussion in Nonlinear Estimation).
Penalty value. As mentioned above, the estimation procedure requires that the
(conditional) sums of squares of the ARIMA residuals be minimized. If the model
is inappropriate, it may happen during the iterative estimation process that the
parameter estimates become very large, and, in fact, invalid. In that case, it will
assign a very large value (a so-called penalty value) to the SS. This usually
"entices" the iteration process to move the parameters away from invalid ranges.
However, in some cases even this strategy fails, and you may see on the screen
(during the Estimation procedure) very large values for the SS in consecutive
iterations. In that case, carefully evaluate the appropriateness of your model. If
your model contains many parameters, and perhaps an intervention component (see
below), you may try again with different parameter start values.

EVALUATION OF THE MODEL

Parameter estimates. You will report approximate t values, computed from the

parameter standard errors (see above). If not significant, the respective parameter
can in most cases be dropped from the model without affecting substantially the
overall fit of the model.
Other quality criteria. Another straightforward and common measure of the
reliability of the model is the accuracy of its forecasts generated based on partial
data so that the forecasts can be compared with known (original) observations.
However, a good model should not only provide sufficiently accurate forecasts, it
should also be parsimonious and produce statistically independent residuals that
contain only noise and no systematic components (e.g., the correlogram of
residuals should not reveal any serial dependencies). A good test of the model is
(a) to plot the residuals and inspect them for any systematic trends, and (b) to
examine the autocorrelogram of residuals (there should be no serial dependency
between residuals).
Analysis of residuals. The major concern here is that the residuals are
systematically distributed across the series (e.g., they could be negative in the first
part of the series and approach zero in the second part) or that they contain some
serial dependency which may suggest that the ARIMA model is inadequate. The
analysis of ARIMA residuals constitutes an important test of the model. The
estimation procedure assumes that the residual are not (auto-) correlated and that
they are normally distributed.
Limitations. The ARIMA method is appropriate only for a time series that
is stationary (i.e., its mean, variance, and autocorrelation should be approximately
constant through time) and it is recommended that there are at least 50
observations in the input data. It is also assumed that the values of the estimated
parameters are constant throughout the series.

Stationarity
Stationarity A common assumption in many time
 series techniques is that the data are
stationary.

A stationary process has the property

that the mean, variance and
autocorrelation structure do not
change over time. Stationarity can be
defined in precise mathematical
terms, but for our purpose we mean a
flat looking series, without trend,
constant variance over time, a
constant autocorrelation structure
over time and no periodic
fluctuations (seasonality).

For practical purposes, stationarity

can usually be determined from a run
sequence plot.
Transformations to If the time series is not stationary, we
Achieve can often transform it to stationarity
Stationarity with one of the following techniques.

1. We can difference the data.

That is, given the series 
, we create the new series

The differenced data will contain

one less point than the original
data. Although you can
difference the data more than
once, one difference is usually
sufficient.

2. If the data contain a trend, we

can fit some type of curve to
 the data and then model the
residuals from that fit. Since
the purpose of the fit is to
simply remove long term
trend, a simple fit, such as a
straight line, is typically used.
3. For non-constant variance,
taking the logarithm or square
root of the series may stabilize
the variance. For negative data,
you can add a suitable constant
to make all the data positive
before applying the
transformation. This constant
can then be subtracted from the
model to obtain predicted (i.e.,
the fitted) values and forecasts
for future points.

The above techniques are intended to

generate series with constant location
and scale. Although seasonality also
violates stationarity, this is usually
explicitly incorporated into the time
series model.

2. Introduction to ARIMA Models

So what exactly is an ARIMA model?

ARIMA, short for ‘Auto Regressive Integrated Moving Average’ is actually a

class of models that ‘explains’ a given time series based on its own past
values, that is, its own lags and the lagged forecast errors, so that equation can
be used to forecast future values.

Any ‘non-seasonal’ time series that exhibits patterns and is not a random
white noise can be modeled with ARIMA models.
An ARIMA model is characterized by 3 terms: p, d, q

where,

p is the order of the AR term

q is the order of the MA term

d is the number of differencing required to make the time series stationary

If a time series, has seasonal patterns, then you need to add seasonal terms and
it becomes SARIMA, short for ‘Seasonal ARIMA’. More on that once we
finish ARIMA.

So, what does the ‘order of AR term’ even mean? Before we go there, let’s
first look at the ‘d’ term.

3. What does the p, d and q in ARIMA model mean

The first step to build an ARIMA model is to make the time series stationary .

Why?

Because, term ‘Auto Regressive’ in ARIMA means it is a linear regression

model  that uses its own lags as predictors. Linear regression models, as you
know, work best when the predictors are not correlated and are independent of
each other.

So how to make a series stationary?

The most common approach is to difference it. That is, subtract the previous
value from the current value. Sometimes, depending on the complexity of the
series, more than one differencing may be needed.
 The value of d, therefore, is the minimum number of differencing needed to
make the series stationary. And if the time series is already stationary, then d
= 0.

Next, what are the ‘p’ and ‘q’ terms?

‘p’ is the order of the ‘Auto Regressive’ (AR) term. It refers to the number of
lags of Y to be used as predictors. And ‘q’ is the order of the ‘Moving
Average’ (MA) term. It refers to the number of lagged forecast errors that
should go into the ARIMA Model.

BOOKMARK ADD CHEYTHITIND

Autoregressive Integrated Moving Average Model

An ARIMA model is a class of statistical model for analyzing and forecasting time
series data.

ARIMA is an acronym that stands

for AutoRegressive Integrated Moving Average. It is a generalization of the
simpler AutoRegressive Moving Average and adds the notion of integration.

This acronym is descriptive, capturing the key aspects of the model itself. Briefly,
they are:

 AR: Autoregression. A model that uses the dependent relationship between

an observation and some number of lagged observations.
 I: Integrated. The use of differencing of raw observations (i.e. subtracting an
observation from an observation at the previous time step) in order to make the
time series stationary.
 MA: Moving Average. A model that uses the dependency between an
observation and residual errors from a moving average model applied to lagged
observations.
Each of these components are explicitly specified in the model as a parameter.

A standard notation is used of ARIMA(p,d,q) where the parameters are substituted

with integer values to quickly indicate the specific ARIMA model being used.
 The parameters of the ARIMA model are defined as follows:

 p: The number of lag observations included in the model, also called the lag
order.
 d: The number of times that the raw observations are differenced, also called
the degree of differencing.
 q: The size of the moving average window, also called the order of moving
average.
Stop learning Time Series Forecasting the slow way!
Take my free 7-day email course and discover how to get started (with sample
code).

Click to sign-up and also get a free PDF Ebook version of the course.

Start Your FREE Mini-Course Now!

Box-Jenkins Method
The Box-Jenkins method was proposed by George Box and Gwilym Jenkins in
their seminal 1970 textbook Time Series Analysis: Forecasting and Control.
The approach starts with the assumption that the process that generated the time
series can be approximated using an ARMA model if it is stationary or an ARIMA
model if it is non-stationary.

The 2016 5th edition of the textbook (Part Two, page 177) refers to the process
as a stochastic model building and that it is an iterative approach that consists of
the following 3 steps:

1. Identification. Use the data and all related information to help select a sub-
class of model that may best summarize the data.
2. Estimation. Use the data to train the parameters of the model (i.e. the
coefficients).
3. Diagnostic Checking. Evaluate the fitted model in the context of the
available data and check for areas where the model may be improved.
It is an iterative process, so that as new information is gained during diagnostics,
you can circle back to step 1 and incorporate that into new model classes.

Let’s take a look at these steps in more detail.

 1. Identification
The identification step is further broken down into:

1. Assess whether the time series is stationary, and if not, how many
differences are required to make it stationary.
2. Identify the parameters of an ARMA model for the data.
1.1 Differencing
Below are some tips during identification.

 Unit Root Tests. Use unit root statistical tests on the time series to
determine whether or not it is stationary. Repeat after each round of differencing.
 Avoid over differencing. Differencing the time series more than is required
can result in the addition of extra serial correlation and additional complexity.
1.2 Configuring AR and MA
Two diagnostic plots can be used to help choose the p and q parameters of the
ARMA or ARIMA. They are:
 Autocorrelation Function (ACF). The plot summarizes the correlation of
an observation with lag values. The x-axis shows the lag and the y-axis shows the
correlation coefficient between -1 and 1 for negative and positive correlation.
 Partial Autocorrelation Function (PACF). The plot summarizes the
correlations for an observation with lag values that is not accounted for by prior
lagged observations.
Both plots are drawn as bar charts showing the 95% and 99% confidence intervals
as horizontal lines. Bars that cross these confidence intervals are therefore more
significant and worth noting.

Some useful patterns you may observe on these plots are:

 The model is AR if the ACF trails off after a lag and has a hard cut-off in the
PACF after a lag. This lag is taken as the value for p.
 The model is MA if the PACF trails off after a lag and has a hard cut-off in
the ACF after the lag. This lag value is taken as the value for q.
 The model is a mix of AR and MA if both the ACF and PACF trail off.
2. Estimation
Estimation involves using numerical methods to minimize a loss or error term.
 We will not go into the details of estimating model parameters as these details are
handled by the chosen library or tool.

I would recommend referring to a textbook for a deeper understanding of the

optimization problem to be solved by ARMA and ARIMA models and
optimization methods like Limited-memory BFGS used to solve it.
3. Diagnostic Checking
The idea of diagnostic checking is to look for evidence that the model is not a good
fit for the data.

Two useful areas to investigate diagnostics are:

1. Overfitting
2. Residual Errors.
3.1 Overfitting
The first check is to check whether the model overfits the data. Generally, this
means that the model is more complex than it needs to be and captures random
noise in the training data.

This is a problem for time series forecasting because it negatively impacts the
ability of the model to generalize, resulting in poor forecast performance on out of
sample data.

Careful attention must be paid to both in-sample and out-of-sample performance

and this requires the careful design of a robust test harness for evaluating models.

3.2 Residual Errors

Forecast residuals provide a great opportunity for diagnostics.

A review of the distribution of errors can help tease out bias in the model. The
errors from an ideal model would resemble white noise, that is a Gaussian
distribution with a mean of zero and a symmetrical variance.

For this, you may use density plots, histograms, and Q-Q plots that compare the
distribution of errors to the expected distribution. A non-Gaussian distribution may
suggest an opportunity for data pre-processing. A skew in the distribution or a non-
zero mean may suggest a bias in forecasts that may be correct.

Additionally, an ideal model would leave no temporal structure in the time series
of forecast residuals. These can be checked by creating ACF and PACF plots of the
residual error time series.

The presence of serial correlation in the residual errors suggests further opportunity
for using this information in the model.

What is “Unit Root”?

A unit root (also called a unit root process or a difference stationary process) is
a stochastic trend in a time series, sometimes called a “random walk with drift”; If
a time series has a unit root, it shows a systematic pattern that is unpredictable.
The Mathematics Behind Unit Roots
The reason why it’s called a unit root is because of the mathematics behind the
process. At a basic level, a process can be written as a series
of monomials (expressions with a single term). Each monomial corresponds to
a root. If one of these roots is equal to 1, then that’s a unit root.
The math behind unit roots is beyond the scope of this site (although if you’re
really interested you can read about it in this pdf). All you really need to know if
you’re analyzing time series is that the existence of unit roots can cause your
analysis to have serious issues like:
 Spurious regressions: you could get high r-squared values even if the data
is uncorrelated.
 Errant behavior due to assumptions for analysis not being valid. For
example, t-ratios will not follow a t-distribution.
What is a Unit Root Test?
Unit root tests are tests for stationarity in a time series. A time series has
stationarity if a shift in time doesn’t cause a change in the shape of the distribution;
unit roots are one cause for non-stationarity.
These tests are known for having low statistical power. Many tests exist, in part,
because none stand out as having the most power. Tests include:
 The Dickey Fuller Test (sometimes called a Dickey Pantula test), which is
based on linear regression. Serial correlation can be an issue, in which case
the Augmented Dickey-Fuller (ADF) test can be used. The ADF handles
 bigger, more complex models. It does have the downside of a fairly high Type
I error rate.
 The Elliott–Rothenberg–Stock Test, which has two subtypes:
 The P-test takes the error term’s serial correlation into account,
 The DF-GLS test can be applied to detrended data without intercept.
 The Schmidt–Phillips Test includes the coefficients of
the deterministic variables in the null and alternate hypotheses. Subtypes are
the rho-test and the tau-test.
 The Phillips–Perron (PP) Test is a modification of the Dickey Fuller test,
and corrects for autocorrelation and heteroscedasticity in the errors.
 The Zivot-Andrews test allows a break at an unknown point in the intercept
or linear trend.