Error bars

It’s good practice to give some indication of uncertainty or expected variability in experi-
mental results. You will need to report experimental results in your coursework. Many of
you will also write up experimental results in dissertations this year, and you will also want
to know how seriously to take numbers that you measure in your future work.
We will discuss some different “standard deviations” that you might see reported, or want
to report, including “standard errors on the mean”.

1 Standard errors on a mean

Imagine we are taking a series of experimental measurements { xn }nN=1 . These could be all
sorts of things: for example, the times taken for different runs of a program that you’re
testing, or the weights of some people who attend a particular gym.
We will assume that the measurements are taken independently from some unknown
distribution. In the first example, the test conditions for the computer program are stable,
and the times depend on independent random choices. In the second example, the people
were selected at random from members of the gym. These distributions are ‘unknown’ in
that we don’t have mathematical descriptions of them. However, we can draw samples
(gather data) from the distributions in these examples.
The mean µ and variance σ2 of the distribution are usually unknown, but we assume they
are finite. We can estimate them from the sample mean and variance, given N independent
samples:
N
1
µ ≈ x̄ =
N ∑ xn (1)
n =1
N
1
σ2 ≈ σ̂2 = ∑
N − 1 n =1
( xn − x̄ )2 . (2)

The ( N − 1) in the estimator for the variance, rather than N, (Bessel’s correction) is a small
detail you don’t need to worry about for this course.1
The estimator x̄ is itself a random variable: if we gathered a second dataset and computed
its mean in the same way, we would get a different x̄. For some datasets x̄ will be bigger
than the underlying true mean µ, sometimes it will be smaller. The mean of x̄ is the correct
answer µ. That is, x̄ is an unbiased estimator.
Using the rules of expectations and variances (see the note in the background section), we
can estimate the variance of x̄. We assume here that the observations are independent:
N
1
var[ x̄ ] =
N2 ∑ var[xn ] (3)
n =1
1
= Nσ2 = σ2 /N ≈ σ̂2 /N. (4)
N2
A “typical” deviation from the mean is given by the standard deviation (not the variance).
So we write: √
µ = x̄ ± σ̂/ N, (5)
to give an indication of how precisely we think we have measured the mean of the distribu-
tion with our N samples. Some papers might report ± two standard deviations.

1. The ( N − 1) normalization makes the variance estimator unbiased and is what the Matlab/Octave var function
does by default. NumPy’s np.var requires the option ddof=1 to get the unbiased estimator. However, if N is small
enough that this difference matters, you need to be more careful about the statistics than we are in this note.

MLPR:w2d Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 1

If the distribution over observations has finite mean and variance, then for large N the
Central Limit Theorem (CLT) tells us that x̄ will be approximately Gaussian distributed
close to its mean. We don’t assume that the data are Gaussian distributed. We just note
that the sum, and so average, of many values will be approximately Gaussian.
√ With that
interpretation, we expect the estimate x̄ to be within 1 standard error σ/ N about 2/3 of
the time, and within 2 standard errors about 95% of the time.
√ √
Care: we don’t evaluate the ‘true’ standard error σ/ N, but an approximation of it, σ̂/ N.
Moreover, the CLT will only be accurate for large N, and can’t be trusted several
√ standard
deviations away from the mean. In other words, the “error bar” µ = x̄ ± σ̂/ N gives some
indication of what means are plausible. Giving this statement is better than just stating
µ ≈ x̄, but more concrete statistical statements would require additional analysis.

2 Application to test set errors

The average test set loss,
M M
1 1
Ltest =
M ∑ L(y(m) , f (x(m) )) =
M ∑ Lm , (6)
m =1 m =1
is an estimate of the generalization error, the average loss we would see if we could gather
an infinite test set with the same distribution. How wrong might this estimate be?
We don’t assume that the individual losses, L(y, f (x)), are Gaussian-distributed, and they
often aren’t. For example, when performing classification we might report the 0–1 loss,
which is zero when we are correct and one when we make an error. In this example, the
distribution over the losses is a Bernoulli distribution.
However, we can compute the empirical mean and variance of any set of losses, and report
an estimate of the mean, with a standard error bar.
Before taking such an error bar on test performance too seriously we would need to think
about whether the theory above applies. Are the test cases independent? If the loss isn’t
bounded, is it likely to have a finite variance that we can reasonably estimate? Will future
inputs to our model come from the same distribution? Will the relationship between inputs
and outputs that we’re modelling continue to be the same?

3 Reliability of a method
A standard error on the test set loss indicates how much the future performance of a
particular fitted model might deviate from the performance we have estimated. It doesn’t say
whether the machine learning method would work well in future, if training were run again
to create a new model.
Readers of a paper may also want to know how variable the performance of a model can
be across different fits, that is, how robust is the method? The fitted models could vary
for multiple reasons: we might gather new data; some machine learning methods depend
on random choices; somewhat horrifyingly, even machine learning code using no random
numbers and running on the same data, often gives different results!2 To summarize one
of these effects, we could report the standard deviation of the models’ performances (not
a standard error on the mean) to indicate how much a future fit will typically vary from
average performance when something is changed.
Important: Papers are sometimes not clear on what their “error bars” are reporting. Some-
times they show the standard deviation of results under different conditions, other times
a standard error indicating uncertainty of the generalization error due to a finite test set.
Always try to be clear precisely what standard deviation/error you are reporting and why.

2. This phenomenon is mentioned in passing in https://arxiv.org/abs/1707.05589. Different single-point preci-

sion round-off errors occur when different choices are made by a parallel scheduler on a GPU, and the effect can be
as large as changing a random initialization!

MLPR:w2d Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 2

4 Which model is better?
If we fit two models, A and B, we can get a test set loss and standard error for both. If these
two error bars overlap, it is common — but usually wrong — to conclude that we can’t tell
whether A is better than B.
As an example, if A had a loss of 0.1 less than B on every single test case in a test set of 1,000
cases, I would be incredibly confident that A was better than B. The two standard errors for
the models’ performance could be larger (for example 0.5). It is possible to be sure of the
ordering of two models, without knowing how good either is very precisely.
To test model A against model B we could do a simple paired comparison. Construct the
difference in losses on each test case:

δm = L(y(m) , f (x(m) ; B)) − L(y(m) , f (x(m) ; A)). (7)

If the mean of the δ’s is several standard errors greater than zero, we would report that A is
the better model. (Non-examinable: you could perform a paired t-test if you wanted to turn
this idea into a formal hypothesis test.)

5 Test your understanding

I hope you’ll try computing error bars “for real”, when you next gather some data. However,
you can (and should!) check you know how on a toy example now.
For example, try generating 100 variables from some non-Gaussian distribution with a mean
you know. For example a Bernoulli distribution:
xx = 1 * (np.random.rand(100) < 0.3)
Now estimate the mean from the 100 data-points, along with a standard error bar on that
mean. While scipy has a standard error function, you should compute the error bar from
the standard deviation at first, so you definitely know what you are computing and why. If
you re-run your code several times, is the error bar usually reasonable?
Not everything you read will have an explicit suggestion of something to try at the end. You
should come up with these ideas yourself: I often look for ways to check my mathematical
understanding (which is often wrong). If there is a small special case where we know the
answer, trying it out on a computer is often a good idea.

6 Further Reading and Reflection

This course doesn’t get into any “proper” or sophisticated statistical analysis of results. For
example, if you wanted to do formal statistical tests on approximately normally distributed
quantities, a t-test takes into account that we can only estimate the variance of the distribution.
However, there is no point talking about t-tests, or other specific statistical tests, if it’s not
clear what you are doing and why!
This note reflects what is actually the most important first step: think about what sources of
variability are affecting your results, and get some idea of how big these variations can be.
Sometimes the difference in performance of two models is so much larger than any source
of variability, that careful testing is not required. If careful testing is required, you should
consider whether anyone cares about the size of the difference, even if it’s “statistically
significant”. If the performance of models is close, maybe we should pick the model that’s
easiest to implement or maintain, or takes less computer time.

MLPR:w2d Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 3