Multiple Regression

Prof. Andy Field

 Aims
• Understand When To Use Multiple Regression.
• Understand the multiple regression equation and
what the betas represent.
• Understand Different Methods of Regression
– Hierarchical
– Stepwise
– Forced Entry
• Understand How to do a Multiple Regression on
IBM SPSS
• Understand how to Interpret multiple regression.
• Understand the Assumptions of Multiple
Regression and how to test them

Slide 2
 What is Multiple Regression?
• Linear Regression is a model to predict
the value of one variable from another.
• Multiple Regression is a natural extension
of this model:
– We use it to predict values of an outcome
from several predictors.
– It is a hypothetical model of the relationship
between several variables.

Slide 3
 Regression: An Example
• A record company boss was interested in
predicting album sales from advertising.
• Data
– 200 different album releases
• Outcome variable:
– Sales (CDs and Downloads) in the week after release
• Predictor variables
– The amount (in £s) spent promoting the album
before release (see last lecture)
– Number of plays on the radio (new variable)
Slide 5
 Multiple Regression as an Equation

• With multiple regression the

relationship is described using
a variation of the equation of a
straight line.

y  b0  b1 X 1 b2 X 2    bn X n   i

Slide 6
 b0

• b0 is the intercept.
• The intercept is the value of the Y
variable when all Xs = 0.
• This is the point at which the
regression plane crosses the Y-
axis (vertical).
Slide 7
 Beta Values

• b1 is the regression coefficient for

variable 1.
• b2 is the regression coefficient for
variable 2.
• bn is the regression coefficient for nth
variable.
Slide 8
 The Model with Two Predictors

Slide 9
 Methods of Regression
• Hierarchical:
– Experimenter decides the order in which
variables are entered into the model.
• Forced Entry:
– All predictors are entered simultaneously.
• Stepwise:
– Predictors are selected using their semi-
partial correlation with the outcome.

Slide 10
 Hierarchical Regression

• Known predictors (based on past

research) are entered into the
regression model first.
• New predictors are then entered in a
separate step/block.
• Experimenter makes the decisions.

Slide 12
 Hierarchical Regression
• It is the best method:
– Based on theory testing.
– You can see the unique predictive
influence of a new variable on the
outcome because known predictors are
held constant in the model.
• Bad Point:
– Relies on the experimenter knowing
what they’re doing!

Slide 13
 Forced Entry Regression
• All variables are entered into the
model simultaneously.
• The results obtained depend on the
variables entered into the model.
– It is important, therefore, to have good
theoretical reasons for including a
particular variable.

Slide 14
 Stepwise Regression I

• Variables are entered into the model

based on mathematical criteria.
• Computer selects variables in steps.
• Step 1
– SPSS looks for the predictor that can explain
the most variance in the outcome variable.

Slide 15
 Stepwise Regression II
• Step 2:
– Having selected the 1st predictor, a
second one is chosen from the
remaining predictors.
– The semi-partial correlation is
used as a criterion for selection.

Slide 16
 Problems with Stepwise Methods

• Rely on a mathematical criterion.

– Variable selection may depend upon only
slight differences in the Semi-partial
correlation.
– These slight numerical differences can lead
to major theoretical differences.
• Should be used only for exploration

Slide 17
• The backward method is the opposite of the
forward method in that the computer begins by
placing all predictors in the model and then
calculating the contribution of each one by
looking at the significance value of the t-test. If a
predictor meets the removal criterion it is
removed from the model and the model is re-
estimated for the remaining predictors. The
contribution of the remaining predictors is then
reassessed.
• Less Type II error
 Doing Multiple Regression

Slide 19
 Doing Multiple Regression

Slide 20
Regression Statistics
Regression
Diagnostics
 Output: Model Summary

Slide 23
 R and R2
• R
– The correlation between the observed values
of the outcome, and the values predicted by
the model.
• R2
– Yhe proportion of variance accounted for by
the model.
• Adj. R2
– An estimate of R2 in the population
(shrinkage).

Slide 24
 Output: ANOVA

Slide 25
 Analysis of Variance: ANOVA
• The F-test
– looks at whether the variance
explained by the model (SSM) is
significantly greater than the error
within the model (SSR).
– It tells us whether using the regression
model is significantly better at
predicting values of the outcome than
using the mean.

Slide 26
 Output: betas

Slide 27
 How to Interpret Beta Values

• Beta values:
– the change in the outcome associated
with a unit change in the predictor.
• Standardised beta values:
– tell us the same but expressed as
standard deviations.

Slide 28
 Beta Values
• b1= 0.087.
– So, as advertising increases by £1,
album sales increase by 0.087 units.
• b2= 3589.
– So, each time (per week) a song is
played on the radio its sales increase
by 3589 units.

Slide 29
 Constructing a Model
y  b0  b1 X 1  b2 X 2
Sales  41124  0.087Adverts  3589plays

£1 Million Advertising, 15 plays

Sales  41124  0.087  1,000,000  3589 15
 41124  87000 53835
 181959

Slide 30
 Standardised Beta Values
• 1= 0.523
– As advertising increases by 1 standard
deviation, album sales increase by 0.523 of a
standard deviation.
• 2= 0.546
– When the number of plays on the radio
increases by 1 SD its sales increase by 0.546
standard deviations.

Slide 31
 Interpreting Standardised Betas

• As advertising increases by
£485,655, album sales increase by
0.523  80,699 = 42,206.
• If the number of plays on the radio
per week increases by 12, album
sales increase by 0.546  80,699 =
44,062.

Slide 32
 Generalization
• When we run regression, we hope to be
able to generalize the sample model to the
entire population.
• To do this, several assumptions must be
met.
• Violating these assumptions stops us
generalizing conclusions to our target
population.

Slide 33
 Straightforward Assumptions
• Variable Type:
– Outcome must be continuous
– Predictors can be continuous or dichotomous.
• Non-Zero Variance:
– Predictors must not have zero variance.
• Linearity:
– The relationship we model is, in reality, linear.
• Independence:
– All values of the outcome should come from a
different person.

Slide 34
 The More Tricky Assumptions
• No Multicollinearity:
– Predictors must not be highly correlated.
• Homoscedasticity:
– For each value of the predictors the variance of the
error term should be constant.
• Independent Errors:
– For any pair of observations, the error terms should
be uncorrelated.
• Normally-distributed Errors

Slide 35
• In designs in which you test several
groups of participants this assumption
means that each of these samples comes
from populations with the same variance
• In correlational designs, this assumption
means that the variance of the outcome
variable should be stable at all levels of
the predictor variable
Plots of standardized residuals against predicted values
 Checking Assumptions about Errors
• Homoscedacity/Independence of Errors:
– Plot ZRESID against ZPRED.
• Normality of Errors:
– Normal probability plot.
– Normally distributed errors : It is assumed that
the residuals in the model are random, normally
distributed variables with a mean of 0. This
assumption simply means that the differences
between the model and the observed data are
most frequently zero or very close to zero

Slide 39
Regression Plots
Homoscedasticity: ZRESID vs. ZPRED
Normality of Errors: Histograms
and P-P plots
 Multicollinearity

• Multicollinearity exists if predictors are

highly correlated.
• This assumption can be checked with
collinearity diagnostics.

Slide 43
• Tolerance should be more than 0.2
(Menard, 1995)
• VIF should be less than 10 (Myers,
1990)
• Independent errors: For any two
observations the residual terms should be
uncorrelated (i.e., independent).
• This assumption can be tested with the
Durbin–Watson test,. The test statistic
can vary between 0 and 4, with a value of
2 meaning that the residuals are
uncorrelated