Meeting 8: Analysis of Variance (ANOVA)

09/06

Lecture 6: Review & Analysis of Variance (ANOVA)

T-test → two groups

ANOVA → more than two groups

Hypothesis Test in Review

RQ about relationship
● Correlation (2 variables)
● Simple linear regression (2 variables)
● Multiple linear regression (3 or more variables)

RQ about differences between groups

Difference between two groups
● Independent samples t-test (between subject design)
● Paired samples t-test (within subject design)
Difference between two or more groups
● (factorial) ANOVA (between subject design)
● ANOVA for repeated measures (within subject design)

Remember: the t-test

Independent samples t-test
● DV is interval/ratio
● IV is nominal scale (indicates group)
● T-test compares 2 groups
○ Males vs. females
○ Experimental vs. control
Example
Ice cream sales
RQ: Whether selling ice cream depends on the location or not
Use independent samples t-test because we have two different groups

Check assumptions (Table)

1. Equality of Variance
○ Levene’s test
2 2 2 2
■ 𝐻0: σ1 = σ2 𝐻𝐴: σ1 ≠ σ2

■ Equal variance if 𝐻0 is not rejected

■ We cannot reject the null hypothesis so we have equal variance

2. Equality of means
○ Mean difference
○ We reject null hypothesis, the two means are different from each other
→ the selling of the ice cream depends on the location

T-test in Review
→ the test-statistic t turns the observed difference into a relative difference (can only be used to compare
two groups)
→ the observed difference is divided by a measure of variation called the standard error

Table
● Interpretation t-statistic: the observed difference is 2.8 standard errors above zero (= no
difference)
Analysis of variance
ANOVA
● DV is interval/ratio scale
● IV is nominal scale
● ANOVA compares 3 (or more) groups

Test statistic under the loop

Standard error:
● Measure of variation in difference of sample means
● = standard deviation of mean difference

𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑖𝑛 𝑚𝑒𝑎𝑛𝑠

Convert definition: t =
𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑖𝑛 𝑚𝑒𝑎𝑛𝑠

T-test is always for two groups, so we need to chance the formula and convert the definition
→ instead, we look at variance ( → ANOVA) we convert t-statistic to the F-statistic

Test statistic ANOVA

𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑣𝑎𝑟𝑖𝑎𝑛𝑐𝑒 𝑖𝑛 𝑚𝑒𝑎𝑛𝑠
F=
𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑣𝑎𝑟𝑖𝑎𝑛𝑐𝑒 𝑖𝑛 𝑚𝑒𝑎𝑛𝑠

Observed variance
● Variance between the group means
● = variance explained by the groups
● = model variance
● Model Mean Square = 𝑀𝑆𝑀

Expected variance
● Variance within the groups
● = variance not explained by the groups
● = residual variance
● Residual Mean Square = 𝑀𝑆𝑅

Compare the mean between the groups

Compare it to the groups mean

Assumptions of ANOVA
1. Observations are independent
2. Dependent variable is at least interval scale
3. Dependent variable is normally distributed in each group
4. Homogeneity of variances (equality of within-group variances)

Example: women’s confidence

RQ: Does watching beautiful men have the same effect on women’s confidence as watching beautiful
women?

Female assigned to 1 of 3 conditions

● Watch pictures of beautiful women (BW)
● Watch pictures of beautiful men (BM)
● Watch pictures of ordinary people (control group)

Compare the mean of the three groups

Data exploration (first step - visualize the groups)

You cannot conclude anything statistically from these graphs
We do see that there is a difference, but is it significant?

Steps in ANOVA

Step 1. Homogeneity
Levene’s test
● If not significant → assume homogeneity
 ● If significant, check rule of thumb

If both are met, we can still rely on

the result

Example → null hypothesis is not rejected, so homogeneity is assumed

Step 2. Effect of condition

Look at the F-statistic - whether the three groups are different

Step 3. Effect size

Measure of effect size (R-squared or Eta Squared)
→ (the proportion of explained variance)

Interpretation
● .01 small i.e. the conditions have a small effect
● .09 medium
● .25 largeF

Table: Condition has a medium to large effect size, η2 = .222

Step 4. Post-hoc tests
We know the groups are different, but which groups are different?
→ so we use the post-hoc tests a.k.a. pairwise comparisons to find it out

Post-hoc test
● T-tests, but with an adjusted p-values
● Have two groups - makes pairs and compare them
● Uses adjusted p-values
○ We have more than one pair
○ The Type I error gets inflated (adjusted p-value lessens this)

α → chance of making a Type I error: probability of correct decision = 0.95

α = 1 - P.C.D. = 1 - .95 = .05

This chances with the number of tests used

● The probability of correct decisions decreases with more tests
● α increases, the Type I error increases (it gets inflated, we should correct it)

To control our overall α, we can make use of post-hoc correction

Three main approaches
1. Bonferroni correction → if n. of groups is small (i.e. 3-4 groups)
2. Tukey’s HSD → if n. of groups is large (i.e. 20 groups)
3. LSD → no correction

Controlling for inflation of Type I error reduces power

Table
● Whenever a women looks at the picture of beautiful woman, she rates herself lower than when
looking at beautiful men
○ -1.5625 and it is significant
○ First row
○ The mean of beautiful women is lower than the mean of beautiful men
○ Beautiful women also rate lower than the control groups
● Between beautiful men and control groups, there is no difference
○ Significance is one
○ 2nd row
● When it is significant, there is a difference
● Positive CI → beautiful men has higher rates than beautiful women
● Both +/- CI → you cannot conclude anything

ANOVA is regression
ANOVA can be seen as a regression model with dummy variables
 Meeting 9: Factorial ANOVA

10/06

Lecture 7: Factorial ANOVA

Want to add more factors to an analysis → we should use another approach

Two-way ANOVA or Factorial ANOVA

When the second variable is a second factor

ANCOVA
When the second variable is an interval variable

Factorial ANOVA
ANOVA with more than one factor
In case of two factors
● Factor A with a levels
● Factor B with b levels

Mean row wise: the dot comes after the number

Column wise: the dot comes before 1
Can also write the last one in column wise

Main and interaction effects

Main effect A
● Are the means equal for all levels of A
Main effect B
● Are the means equal for all level of B
Interaction effect A and B
● Is the effect of A the same for all levels of B

Examples of interaction
1) a. There is no main effect b. Yes, because the lines are parallel

2) a. There is a main effect b. Yes, because they are parallel

→ you jump from condition A to B, there is the same increase

3) a. Yes, for males it changes, but does not change for females b. No

4) a. Yes, there is a change when you go from condition A to B

b. No, go into different direction
New Research Question
Have another group that is added → male
Interaction → you should also consider the effect

RQ: Does watching pictures have the same effect on men as it has on women?

Two-Way ANOVA
● Main effect condition
○ Are the self ratings the same in all three conditions
● Main effect gender
○ Are the self ratings the same for men and women
● Interaction
○ Are the differences between the conditions the same for men and women?

Steps in Two-Way ANOVA

Step 1. Levene’s Test

We cannot reject the null hypothesis, so we

can assume homogeneity
Step 2 - 4.

The interaction between gender and experimental condition was found to be not significant, F(2, 74) =
0.21, p = .808. So we rerun the model without the interaction

After the removal of the interaction term, both main effects were found to be significant. The difference
between men and women was significant, F(1, 76) = 13.13, p = .001, η2 = .147, indicating a medium
effect. The difference between the three experimental conditions was significant, F(2, 76) = 8.09, p =
.001, η2 = .176, indicating a medium effect.

Step 5. Post-hoc
Post-hoc tests revealed a significant difference between the experimental conditions ‘beautiful women’
and ‘beautiful men’ (p = .016) and between the experimental conditions ‘beautiful women’ and the
control group (p = .001). The difference between the control group and ‘beautiful men’ was not
significant.

Estimated marginal means

Top table
● Descriptive statistics of gender

1 means that it is completely not significant

Why is there no post-hoc test for gender?

● Gender only has two categories

Profile plot
● Here it is more parallel and no interaction
 Meeting 10: Analysis of Covariance (ANCOVA)

13/06

Lecture 8: Analysis of Covariance (ANCOVA)

ANOVA
→ comparing more than two groups

Factorial ANOVA
Only with a factor

Question → what if the DV is influenced by an interval or ratio IV as well?

Analysis of Covariance
ANCOVA → compares levels of a factor, while controlling for a covariate
● Factor: nominal/categorical IV
● Covariate: an interval/ ratio IV
● Adding a covariate to ANOVA and controlling it
= fix the covariate so the difference is equal between two lines

Why a covariate
● To correct for differences between factor levels on this covariate
● To increase power to detect differences between populations

ANCOVA is a regression model:

Predictor x is the covariate

Dummy variable(s) z represent the factor (0 & 1)
The slope of regression models are the same → they are parallel
In the graph, the covariate is controlled for so the distance is the same
● Equal slopes for z = 0 and z = 1
● For all values of x, difference z is the same (𝑏1)

Comparison of adjusted means

→ we can compare the group on factor z, while controlling for group differences on x

ANCOVA has two advantages over ANOVA

1. Correction for differences on the covariate
2. More power because within-group variance is reduced
a. Error w.r.t regression line instead of group mean

Example A

Means on DV y are the same but adjusted means are

different
Group two scores higher on the covariate x
x and y are positively related
With equal scores on x, group 1 scores higher on y
Example B

The means on DV y are different but adjusted means

are the same
They only use one regression line
Group 2 scores higher x
x and y are positively related
With equal scores on x, both groups score equal on y

ANCOVA: assumptions
1) DV y and covariate x are linearly related (quadratic term)
2) Homogeneity of variances
● As in ANOVA - Levene’s test
3) Homogeneity of regression slopes
● Regression slopes are parallel
● No interaction between factor and covariate
● If significant, include in the model

How to test for interaction

Add an interaction term to the model and see if it is

significant or not

Example: ANOVA
RQ: Does talking on the phone interfere with people’s driving skills

Factor: condition DV: reaction time in ms in simulation test

Example: including covariate age
→ ANCOVA model

Inclusion of age in ANCOVA model:

● Reduces error variance, thus increasing power to detect an effect
● Corrects for possible differences in age

Steps in ANCOVA
Step 1. Linear dependency
We see from the graph that they are linear in each condition

Step 2. Homogeneous slopes

Visual: check slopes of separate regression lines
● They are not parallel but go all in the same direction

Test significance: add interaction to the model

● Interaction is not significant - assume that the regression lines are parallel & run the ANCOVA
model with the interaction→ homogeneity of slopes satisfied
Step 3. Homogeneity of variances
Check Levene’s test
Levene’s test not significant, p = .273, homogeneity of variances satisfied

Step 4. ANCOVA
Model without interaction
● The effect of condition, after controlling for age, is significant
○ Age is a good covariate in this model (large effect): p < .001, η2 = .332
○ There are significant differences between the adjusted means of the three conditions, p =
.028, η2 = .120 (medium effect)

Step 5. Post-hoc tests

If 0 is in between the CI, it confirms the p-value
The only group that has different reaction times is between hand-held and control - the reaction time for
hand-held is higher than for the control
No significant difference between the hand-held and hands-free condition, p = .324.
No significant difference between hands-free and control condition, p = .842.
There was a significant difference between the hand-held and control condition, p = .025, 95% CI [9.28,
176.8]

Step 6. Effect Covariate

Older people are slower (look at B)
→ when age increases by one year, the reaction time increases with 7.35 ms
Two-way ANCOVA
ANCOVA can be extended with more factors – include sex to the model

Step 2. Homogenous regressions

None of the interactions are significant → model without interaction is better
Step 3. Homogeneity of variances
Equality may be assumed (p = .629)

Step 4.

Age is still a good covariate (p < .001)

No significant effect for sex: main effect (p = .187) and interaction effect (p = .219)
→ model without sex is better
 Meeting 11: Repeated Measures

15/06

Lecture 9: Repeated Measures Analysis

Within-Subjects designs
Mixed designs

Remember ANOVA
● Comparing two or more groups
● Groups are independent
● Often used in experimental design

⇒ Between Subjects Design

● Key element: separate groups of participants are used for the different treatment conditions

Within-subject design (repeated-measures)

→ Repeated measures design is a research design in which subjects are measured two or more times on
the DV – Rather than using different participants for each level of treatment, the participants are given
more than one treatment and are measured after each (same group but different measures)

● Same group of individuals participates in all of the treatment conditions

Counter-balancing = to avoid a learning effect, different orders of treatments may be used

Within-subject measurements of the same IV
● At different times
● Under different conditions
● Measurements between times are dependent

→ Special technique: analysis of difference scores

Advantage over between-subjects design:

● More power to detect effects
● More economical (fewer participants needed)
● Study develops over time

Example 1
RQ: Do the OCD-scores change significantly over time?

DV: score on OCD IV: time, there are three

→ measurements are not independent

For the third assumption: the variances are not equal

● Assume the dependence between measurement moments is equal
○ Assume equal variances of the difference scores
⇒ Sphericity
Sphericity
Tested with Mauchly’s test
● If not significant, we assume Sphericity: all the variance differences are equal

When assumption violated → look at the measure called Epsilon: use the value to determine which
alternative we should use

We look at the first row because the assumption was met (Mauchly’s test was not significant)
Pairwise comparisons
● The first treatment scores 3.55 times lower than the second treatment, (p = .006) and 5.83 times
lower than the third treatment (p < .001).
● Difference between time 1 and 2, and difference between 1 and 3 are significant
● The difference between time 2 and 3 is not significant

Example 2

→ assumption not satisfied: when looking at Epsilon, we should use Greenhouse-Geisser alternative
Because the assumption of sphericity was violated, the Greenhouse-Geisser alternative was used. No
significant difference was found, F (2.04, 22.39) = 3.13, p = .062. ANOVA is not significant.

Example 3
In total, there are 4 conditions
● Gambling with their own money instead someone else’s money
● The risk is low instead of high
○ Low risk is that you gain or lose a little

2 within-subject factors DV
Person (Self, Other) % of choices that can be considered hazardous
Risk (High, Low)

Assumption of sphericity
We have two levels, there is only one difference, so that is why there is a dot
→ assume sphericity
Test of within-subject effect
The effect of person is significant, p < .001
The effect of risk is significant, p < .001
The interaction between risk and person is significant, p = .008

Profile plot: they are not parallel → shows that there is interaction

Descriptive statistics
The CI is for the mean
We don’t have a post hoc test because there are only two groups so we know already where the difference
is
Mixed designs
Have both within-subjects and between-subjects factors

Example – classical experimental design

● Control group and experimental group (between-subjects factor)
● Pre-test and post-test (within-subjects factor)

Effects
● Main effects of within and between subject factor
● Interaction between within and between-subjects factor

Example 4A
RQ: Does sleep deprivation reduce depression symptoms?

Within-subject factor
● Time (before, after)
Between-subject factor
● Condition (sleep deprivation, control)
Dependent variable
● Symptoms of depression

To compare the symptoms of depression on the post-test between the two groups, we can control for the
differences on the pre-test
● When we only have two measurement moments, we can analyze the data using ANCOVA

Example 4B
RQ: Does sleep deprivation reduce depression symptoms?

Within-subject factor
● Time (before, mid, end, after)
Between-subject factor
● Condition (sleep deprivation, control)
Dependent variable
● Symptoms of depression
Sphericity
Mauchly’s test not significant → Sphericity assumption is met

Within-subjects effects
Look at the first row because Sphericity is met

Interaction is also significant: the changes over time for the control group differ significantly from how
the means change over time for the sleep deprivation group

Condition is significant, meaning that they are different from the control group
 Meeting 12: Multivariate Analysis of Variance

15/06

Lecture 10: Multivariate Analysis of Variance

→ comparing group means on various dependent variables

Example
There are 4 dependent variables

Multivariate analysis of variance

MANOVA
● Group differences on more than one DV
● Why not one ANOVA for each DV?
○ Inflates the Type I error if you increase the number of testing
○ Less power to detect group differences

Multivariate differences
2 DV and 2 groups

Example: IQ and breastfeeding

RQ: Does breastfeeding have an effect on IQ of children (at age 7)?
DVs
1. Verbal IQ
2. Performance IQ

Factor: 3 conditions
● No breastfeeding (control)
● Breastfeeding (physical hypothesis)
● Breast milk but no breastfeeding (social hypothesis)

Null hypothesis

The vector means of verbal and performance IQ

do not differ across the 3 conditions

Assumption of MANOVA
1) Homogeneous covariance matrices
● The covariance matrices of the DV’s are the same in each group
● Check with Box’s M-test (comparable to Levene’s test)

→ instead of having one variance, we have a covariance matrix because we have more
than one DV variables (this matrix should be the same in all conditions)

Covariance
● Measure the variation of both variables
● Measures joint variation
Steps in MANOVA

Step 1. Covariance Matrices

Test is not significant → may assume equal covariance matrices

(p = .740) (matrix of variance and covariance)

Step 2. Wilk’s Lambda

Significant differences between the three groups in multivariate space were found using Wilk’s Lambda
Test, F(4, 208) = 6.62, p < .001, η2 = .113, the various conditions have a medium effect

Step 3. Separate ANOVAs

We now run separate ANOVAs for the verbal and performance scores
First we check Levene’s test: yields an insignificant result so we can assume homogeneity
Interpretation = There is difference between three conditions in Verbal and Performance IQ

Continue to post-hoc test

Extensions MANOVA
Factorial MANOVA
● More than one factor

MANCOVA
● Inclusion of covariate(s)
Same principle
● Check multivariate effects
● If significant, interpret univariate models

Example: factorial MANOVA

3 DV
● Social dominance
● Sociability
● Worry
2 factors
● Gender
● Age group
Step 2:
● Multivariate main effects significant
● Multivariate interaction not significant
● Remove interaction term from the model

Without interaction
● Multivariate main effects significant
● Look at separate ANOVAs
 Meeting 13: Recap Module 2

20/06

Ancova
● Want parallel lines in the graph

Tips
● More than two tables (SPSS output)
○ Which of the following is correct
○ May try to confuse us
● Try to tell us a story about the experiment
○ Ask which statement is correct based on this report
○ Eta-squared
● True and false questions
○ Repeated measures
○ Compare two kind of anovas
● Plots
○ The one with the circles
● Two conditions
○ Whether it is possible to include one more factor or covariate to the analysis
● Profile plot
○ Which one is correct from the output
○ When the lines are close to each other → then there is no effect