Statistics for Astronomy and

Physics students
Fall 2020

Thomas Nagler

Version: June 21, 2021

Thomas Nagler1
1
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International
(CC BY-ND 4.0) License.
 Contents
2 Probability basics 1
2.1 Sample spaces and events . . . . . . . . . . . . . . . . . . . . . . 1
2.2 Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 Conditional probability . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.6 Distribution functions . . . . . . . . . . . . . . . . . . . . . . . . 14
2.7 Probability mass and density functions . . . . . . . . . . . . . . . 15
2.8 Bivariate distributions . . . . . . . . . . . . . . . . . . . . . . . . 18
2.9 Conditional distributions . . . . . . . . . . . . . . . . . . . . . . . 20
2.10 Independence (ctd’) . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.11 Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.12 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.13 Variance and standard deviation . . . . . . . . . . . . . . . . . . . 26
2.14 Covariance and correlation . . . . . . . . . . . . . . . . . . . . . . 27

3 Descriptive statistics and exploratory data analysis 31

3.1 Sample averages and the law of large numbers . . . . . . . . . . . 31
3.2 Sample (co)variances . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 The empirical distribution function . . . . . . . . . . . . . . . . . 37
3.4 The histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5 Quantiles and the boxplot . . . . . . . . . . . . . . . . . . . . . . 40
3.6 A recipe for EDA . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.7 Case study: Colors of quasars vs. Galaxies . . . . . . . . . . . . . 46

4 Parametric statistical models 51

4.1 Discrete distribution families . . . . . . . . . . . . . . . . . . . . . 51
4.2 Continuous distribution families . . . . . . . . . . . . . . . . . . . 54
4.3 Multivariate normal distribution . . . . . . . . . . . . . . . . . . . 62
4.4 Regression models . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5 Parameter estimation 65
5.1 Motivation: distributions in times of Corona . . . . . . . . . . . . 65
5.2 A general framework . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3 The method of moments . . . . . . . . . . . . . . . . . . . . . . . 69
5.4 Maximum likelihood estimation . . . . . . . . . . . . . . . . . . . 70
5.5 Checking for misspecification and model fit . . . . . . . . . . . . . 75
5.6 Chi-square fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6 Uncertainty quantification 78
6.1 The central limit theorem . . . . . . . . . . . . . . . . . . . . . . 78
6.2 Asymptotic normality of estimators . . . . . . . . . . . . . . . . . 80

ii
 Contents

6.3 Asymptotic normality of the MLE . . . . . . . . . . . . . . . . . . 81

6.4 The delta method . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.5 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.6 The bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

7 Testing 90
7.1 Hypothesis testing: a quick walk through . . . . . . . . . . . . . . 90
7.2 Issues with hypothesis testing . . . . . . . . . . . . . . . . . . . . 93
7.3 Null and alternative hypotheses . . . . . . . . . . . . . . . . . . . 95
7.4 Test statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.5 Test errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.6 Significance and p-values . . . . . . . . . . . . . . . . . . . . . . . 98
7.7 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.8 Multiple testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.9 Some classes of tests . . . . . . . . . . . . . . . . . . . . . . . . . 101

8 Regression models 102

8.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.2 The linear regression model . . . . . . . . . . . . . . . . . . . . . 103
8.3 Generalized linear models . . . . . . . . . . . . . . . . . . . . . . 109
8.4 Non-linear models . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8.5 Model selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

9 Missing data 121

9.1 Setup and notation . . . . . . . . . . . . . . . . . . . . . . . . . . 121
9.2 Types of missing data . . . . . . . . . . . . . . . . . . . . . . . . 121
9.3 Censoring and truncation . . . . . . . . . . . . . . . . . . . . . . . 123
9.4 Correcting for missingness . . . . . . . . . . . . . . . . . . . . . . 124
9.5 Takeaways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

10 Bayesian inference 126

10.1 Frequentism vs. Bayesianism . . . . . . . . . . . . . . . . . . . . . 126
10.2 Bayesian model setup . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.3 Bayesian updating . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.4 Bayesian inference . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10.5 The choice of prior . . . . . . . . . . . . . . . . . . . . . . . . . . 130
10.6 Bayesian inference via simulation . . . . . . . . . . . . . . . . . . 132
10.7 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 134

iii
 Probability basics
2
Having observed some data X1 , . . . , Xn , what can we say about the
mechanism that generated them?

This question is the basic problem of statistics. In statistics, we assume that

the mechanism involves some form of randomness. This is helpful even if the
mechanism is genuinely deterministic. Here, we use ‘randomness’ to reflect our
lack of knowledge. In many cases, this lack of knowledge cannot be resolved —
because the mechanism is too complex for us to understand (e.g., interaction
of all particles in the universe) or because a critical piece of information is not
observed.
Before we can talk properly about statistics, we need a suitable language to
talk about randomness. Probability theory is a mathematical field concerned with
random phenomena and provides just that. In the following, we learn about
basic concepts, definitions, and properties that build the foundation for statistical
theory.

2.1 Sample spaces and events

In probability and statistics, random phenomena are often called experiment.

Definition 2.1. (i) The sample space Ω is the set of all possible outcomes
of an experiment.

(ii) Elements ω ∈ Ω are called outcomes or realizations.

(iii) Subsets of A ⊂ Ω are called events.

A more philosophical interpretation of the above is the following: The sample

space Ω can be seen as all possible (hypothetical) states of our world. An element
ω ∈ Ω is a particular state of our world. An event is a collections of states that
share some property.
Let’s make this more concrete.

Example 2.2. For the experiment consisting in tossing a coin twice,

Ω = {HH, HT, T H, T T },
Chapter 2 Probability basics 2

where H stands for heads and T for tails. The event that the first toss is heads
is A = {HH, HT }.

Example 2.3. Let ω be the outcome of a measurement of some quantity, for

instance stellar mass. Then Ω = (0, ∞). The event that the measurement is larger
than 0.5, but less than 1.2 is A = (0.5, 1.2).

Example 2.4. We ask a random person in the street what his month of birth
is. The sample space is Ω = {Jan, F eb, . . . , Dec}. The event that he was born in
spring is A = {M ar, Apr, M ay}.

Now consider two events: an earthquake (A) and a flood (B) defined on
the sample space Ω. The set A contains all states of our world ω, in which a
earthquake happens. The set B contains all ω, for which a flood happens.
What is the event of no earthquake happening?

Definition 2.5. For a given event A, let Ac = {ω ∈ Ω : ω ∈

/ A}. This is called
the complement of A and is thought of as the event ‘not A’.

To understand events, visualizations like the Venn diagram are often helpful.
Think of the rectangle as the sample space Ω. That is, all points ω inside the
rectangle together form the set Ω. The disk represents the event A. All points ω
inside the disk form the set A. The event Ac (‘a does not happen’) is defined as
all outcomes in Ω that are not part of the set A (the shaded area).

Remark 2.1. The complement of Ω is an empty set ∅. Conversely, the comple-

ment of an empty set ∅ is Ω. Convince yourself visually that this is true.

Example 2.6. We ask a random person in the street what his month of birth
is. The sample space is Ω = {Jan, F eb, . . . , Dec}. The event that he was born in
spring is A = {M ar, Apr, M ay}. The event that he was not born in spring is

Ac = {ω ∈ Ω : ω ∈/ A}
= {Jan, F eb, Jun, Jul, . . . , Dec}.

There are several ways to connect or disentangle two different events A and B.
First, what is the event that there is an earthquake or a flood?
Chapter 2 Probability basics 3

Definition 2.7. The union of the events A and B, which can be thought of
as an event ‘A or B’, is defined as

A ∪ B = {ω ∈ Ω : ω ∈ A or ω ∈ B or both}.

A B

The ‘or’ here is non-exclusive: A and B may both happen simultaneously, but
we also accept it if just one of them does.
Similarly, or a possibly infinite sequence of events A1 , A2 , A3 , . . .
[
Ai = {ω ∈ Ω : ω ∈ Ai for at least one i}.
i

Example 2.8. We ask a random person in the street what his month of birth
is. The sample space is Ω = {Jan, F eb, . . . , Dec}. The event that he was born
in spring is A = {M ar, Apr, M ay}. The event that he was born in summer is
B = {Jun, Jul, Aug}. The event that he was born in spring or summer is

A ∪ B = {M ar, Apr, M ay} ∪ {Jun, Jul, Aug}

= {M ar, Apr, M ay, Jun, Jul, Aug}

The second way to connect the two is: what is the event that there is both an
earth quake and a flood?

Definition 2.9. The intersection of A and B (an event ‘A and B’) is

A ∩ B = {ω ∈ Ω : ω ∈ A and ω ∈ B simultaneously}.

A B

Similarly, for an infinite sequence of events A1 , A2 , A3 , . . .

∞
\
Ai = {ω ∈ Ω : ω ∈ Ai for all i}.
i=1
Chapter 2 Probability basics 4

Example 2.10. We ask a random person in the street what his month of birth
is. The sample space is Ω = {Jan, F eb, . . . , Dec}. The event that he was born in
winter is A = {Dec, Jan, F eb}. The event that he was born in the first half of a
year is B = {Jan, . . . , Jun}. The event that he was born both in the winter and
in the first half of the year is {Jan, F eb}.

A ∩ B = {Dec, Jan, F eb} ∩ {Jan, . . . , Jun}

= {Jan, F eb}

Next, what is the event that an earthquake, bot no flood happens?

Definition 2.11. The set difference is defined as A \ B = {ω ∈ Ω : ω ∈

A, ω ∈
/ B}. It can be thought of as the event ‘A but not B‘.

A B

Example 2.12. We ask a random person in the street what his month of birth
is. The sample space is Ω = {Jan, F eb, . . . , Dec}. The event that he was born
in summer is A = {Jun, Jul, Aug}. The event that he was born in August is
B = {Aug}. The event that he was born in Summer but not in August is

A \ B = {Jun, Jul, Aug} \ {Aug} = {Jun, Jul}

2.2 Probabilities
Having set a mathematical framework to speak about events, we can move on to
probabilities of events.

2.2.1 The axioms of probability

In 1933, Andrey Kolmogorov defined the concept ‘probability’ formally as a
function satisfying three axioms. Loosely speaking, an axiom is a property that
is taken to be true with no questions asked. Axioms build the foundation of all
of mathematics: real numbers, sets, logic — and probability. Axioms were a big
topic around 100 years ago.1 The idea is to find a minimal set of principles that
everybody can agree on. Everything else needs to be deduced from these facts.
We need some more notation. We call sets A1 , A2 , A3 , . . . are disjoint, if
Ai ∩ Aj = ∅ whenever i 6= j. For example, A1 = [0, 1), A2 = [1, 2), A3 = [2, 3), . . .
are disjoint sets.
1
Considering the long history of mathematics, axioms are a modern concept.
Chapter 2 Probability basics 5

Definition 2.13. A function P that assigns a number P(A) to each event

A ⊆ Ω is a probability distribution or a probability measure, if it
satisfies:

(i) P(A) ≥ 0 for every A,

(ii) P(Ω) = 1,

(iii) If A1 , A2 , . . . are disjoint, then

∞
! ∞
[ X
P Ai = P(Ai ).
i=1 i=1

The axioms are supposed to reflect what we mean by the concept ‘probability’.
A (i) non-negative number, such that (ii) the set of all possible outcomes has
probability 1, and (ii) the probabilities of exclusive events (only one of them can
happen) sum up.2
There are two common interpretations of probability: frequencies and degrees
of belief.

• In the frequency interpretation, we think of repeating the exact same

experiment over and over again. Then P(A) is the frequency of outcomes
where the event A happens. For example, if we say that the probability of
heads is 1/2, we mean that if we flip a (fair) coin many times, the proportion
of times we get heads will tend to 1/2.

• In the degree-of-belief interpretation, P(A) is the observer’s strength of

belief that A is true. Like in “I’m 100% sure the Dutch national team wins
the world cup.”

In either case we require Axioms 1–3 to hold.

2.2.2 Properties of probabilities

The axioms imply many intuitive properties of the probability. You can verify
the following as an exercise:

• P(∅) = 0.

• A ⊂ B ⇒ P(A) ≤ P(B).

• 0 ≤ P(A) ≤ 1.

2
Admittedly, the third axiom is not that intuitive for non-mathematicians.
Chapter 2 Probability basics 6

• P(Ac ) = 1 − P(A).

• A ∩ B = ∅ ⇒ P (A ∪ B) = P(A) + P(B).
The following result is less trivial:

Lemma 2.14 (Inclusion-exclusion principle). For any events A and B,

P (A ∪ B) = P(A) + P(B) − P (A ∩ B) .

A B

We shall follow an old mathematical tradition, proof by picture.3 Visually,

we can think of the probability as areas (in the sense of 2d-volume). The area
enclosed by the rectangle is P(Ω) = 1. The probability of an event E, is the
area it covers in proportion to the area of the rectangle. For example, P(A) is
the area enclosed by the circle around A. We want to compute the area of the
shaded region in the figure above. Add P(A) (the area of the circle around A) to
P(B) (the area of the circle around B). Then we counted the area dark region
in the middle (A ∩ B) twice. Hence, we need to subtract P(A ∩ B).
Let’s do this formally. If you have trouble understanding the proof, try to
visualize every step.
Proof of Lemma 2.14. Note that A ∪ B = (A \ B) ∪ (A ∩ B) ∪ (B \ A). The sets
(A \ B), (A ∩ B), and (B \ A) are disjoint. Hence,
 
P(A ∪ B) = P (A \ B) ∪ (A ∩ B) ∪ (B \ A)

(third axiom) = P(A \ B) + P(A ∩ B) + P(B \ A)

= P(A \ B) + P(A ∩ B) + P(B \ A) + P(A ∩ B) − P(A ∩ B) .
| {z }
=0

We now use the third axiom again, but in the other direction. The sets (A \ B)
and (A ∩ B) are disjoint and (A \ B) ∪ (A ∩ B) = A. Hence,

P(A \ B) + P(A ∩ B) = P(A),

and similarly,

P(B \ A) + P(A ∩ B) = P(B).

3
That’s actually a classical mathematician’s joke. Mathematicians are funny.
Chapter 2 Probability basics 7

Using this in the equation above yields

P(A ∪ B) = P(A \ B) + P(A ∩ B) + P(B \ A) + P(A ∩ B) − P(A ∩ B)

= P(A) + P(B) − P(A ∩ B).

Let’s end this section with a few examples.

Example 2.15. Toss a fair coin twice. Let H1 be the event that head occurs on
toss 1 and let H2 be the event that head occurs on toss 2, then

P (H1 ∪ H2 ) = P(H1 ) + P(H2 ) − P (H1 ∩ H2 )

1 1 1 3
= + − = .
2 2 4 4
In the example, we used a hidden assumption to see that P (H1 ∩ H2 ) = 1/4.
More on that later.
Now suppose Ω = {ω1 , . . . , ωn }. Such sample spaces are called finite, because
they contain a finite number of possible outcomes.

Example 2.16. If we throw a die twice, then Ω has 36 elements:

Ω = {(i, j) : i, j ∈ {1, 2, . . . , 6}}.

If each outcome is equally likely, then P(A) = |A|/36. For instance, the probability
that the sum of the dice is 11 is equal to 2/36.

For a finite set A, let |A| denote the number of elements in A. If Ω is finite
and each outcome is equally likely, then

|A|
P(A) = .
|Ω|

This is called the uniform distribution on the set Ω.

Example 2.17. We ask a random person in the street what his month of birth
is. The sample space is Ω = {Jan, F eb, . . . , Dec}. The probability that he was
born in the winter is 3/12 = 1/4, if being born in each month is equally probable.

2.3 Independence
Colloquially, we speak of two independent events, when they have nothing to
do with each other. For example, the events ‘it will be raining tomorrow’ and
‘you read an article on mice yesterday’ are entirely unrelated. There is a formal,
probabilistic definition of such events.
Chapter 2 Probability basics 8

Definition 2.18. Events A and B are called independent, if

P(A ∩ B) = P(A)P(B).

We shall write A ⊥ B in this case.

To understand this definition, reconsider Example 2.15. Suppose we flip a fair

coin twice. From the many coins we have flipped in our lives, we think of the
two flips as independent. The outcome of the first flip tells us nothing about the
outcome of the second. Now what’s the probability that both flips turn heads?
Our intuition says 1/4 (1/2 for the first flip times 1/2 for the second). This is
exactly what Definition 2.18 says. If you are not convinced, we will see another,
equivalent definition of independence in the next section.
Independence does not only apply to a pair of events.

Definition 2.19. A collection of events {Ai : i ∈ I} is called independent, if

!
\ Y
P Ai = P(Ai )
i∈J i∈J

for any finite subset J ⊆ I.

Example 2.20. Toss a fair coin 10 times. Let A = ‘at least one head’ and Tj
be the event that tail occurs on the j th coin toss. Then

P(A) = 1 − P(Ac )
= 1 − P(all tails)
10
!
\
=1−P Ti
i=1
10
Y
(using independence) =1− P(Ti )
i=1
 10
1
=1− ≈ 0.999.
2

2.4 Conditional probability

Conditional probabilities are about if–then statements: if we know that B hap-
pened, what is the probability of an event A? As in: if we found someones
fingerprints one a dead body, what’s the probability we caught the murder? We
call this P(A | B), the conditional probability of A given B.
Chapter 2 Probability basics 9

Definition 2.21. If P(B) > 0, then the conditional probability of A given

B is defined by
P (A ∩ B)
P(A|B) = .
P(B)

The restriction P(B) > 0 is necessary for the fraction to be well defined. This
aligns with common sense: if something cannot possibly happen, it’s foolish to
talk about it’s consequences. Let’s get a visual intuition for the formula.

A B

We see the conventional probability P(A) as the area of the disk around A divided
by the area of the rectangle (Ω). Now suppose we know that the realization ω
lies in the disk B and forget about all the other cases. Then B acts as our new
Ω. Now we see the conditional probability P(A | B) is the area covered by A
(which, after forgetting everything else, is A ∩ B) relative to the area of the disk
around B. We therefore think P(A | B) as the fraction of times A occurs among
those in which B occurs.
For fixed B, P(A | B) is a proper probability measure – it satisfies all three
axioms:

P(A | B) ≥ 0, for any A ⊆ Ω,

P(Ω | B) = 1,
∞
! ∞
[ X
P Ai B = P (Ai | B) , for disjoint A1 A2 , . . . .
i=1 i=1

It does not behave like that as a function of B though. In general it is not true
that
P(A | B ∪ C) = P(A | B) + P(A | C).
Neither is P(A | B) = P(B | A) true in general (related to the ‘prosecutor’s
fallacy’4 ).
Our intuition often fails us when conditional probabilities are involved. There’s
a huge number of ‘fallacies’ and ‘paradoxes’. That’s why understanding the
concept of conditional probabilities is even more important.

Example 2.22. A medical test for COVID-19 has outcomes + and −. The joint
probabilities are
4
https://en.wikipedia.org/wiki/Prosecutor%27s_fallacy
Chapter 2 Probability basics 10

COV ID healthy
+ 1% 1%
− 0.2% 97.6%

The test appears to be pretty accurate:5

P(+ ∩ COVID) 1%
P(+ | COVID) = = ≈ 83%,
P(COVID) 1% + 0.2%
P(− ∩ healthy) 97.6%
P(− | healthy) = = ≈ 99%.
P(healthy) 97.6% + 1%

The first equation states that, if someone has the disease, the test detects the
disease in 83% of the cases. The second equation states that, if someone does not
has the disease, the test will correctly diagnose ‘no disease’ in 99% of the cases.
These conditional probabilities are the two common quality measures for medical
tests (called sensitivity and specificity). If you read ‘the test is 99% correct’ in a
newspaper, it refers to one of these conditional probabilities (or both). The above
accuracy numbers are in line with what we know about the COVID tests in use to
day.
Now suppose you go for a test and the test is positive. What is the probability
you have the disease?

P(COVID ∩ +) 1%
P(COVID | +) = = = 50%.
P(+) 1% + 1%

So if you get a positive test, you have a 50% chance to be healthy anyway. That’s
not intuitive at all, the test is correct at least 83% of the time!
Indeed it is correct on 83% of diseased patients and correct on 99% of healthy
patients. However, there are way more healthy people (P (healthy) = 98%) than
diseased ones (P (COVID) = 2%). Out of the huge number of healthy people,
1% are incorrectly diagnosed with the disease. If the whole population would get
tested, 98% × 1% ≈ 1% of the entire population would be tested positive despite
being healthy. Contrast this with the total number of diseased people: 1% of the
population. Hence, most of the people with a positive test are healthy.

The previous example is not just a brain teaser but important for medical
practice. False positives often have severe consequences (think: quarantine,
mental health issues, expensive medicine, unnecessary surgery). Testing a large
proportion of a population can therefore cause more harm than good if a) the
test is not reliable enough, or b) the disease is too rare. Especially b) is rarely
part of the public discourse, but at least as important as a).
Let’s continue with some useful results on conditional probabilities.

5
Here we use that for two events A, B, it holds P (A) = P (A ∩ B) + P (A ∩ B c ). This follows
from the third axiom because A can be partitioned in the two disjoint sets A ∩ B and A ∩ B c .
Chapter 2 Probability basics 11

Lemma 2.23.

(i) If A and B are independent, then

P(A | B) = P(A) and P(B | A) = P(B).

(ii) For any pair of events A and B,

P(A ∩ B) = P(A | B)P(B) = P(B | A)P(A).

Proof.
(i) Recall that independence of A and B is equivalent to P(A | B) = P(A)P(B).
By the definition of conditional probabilities (Definition 2.21), it holds

P(A ∩ B) indep. P(A)P(B)

P(A|B) = = = P(A).
P(B) P(B)

Repeat the same argument by interchanging the roles of A and B.

(ii) For the first equality, multiply both sides in Definition 2.21 with P(B).
Repeat the same argument by interchanging the roles of A and B.
The first part has an intuitive interpretation. Independence of A and B means
that the events are completely unrelated. Knowing about the outcome of B cannot
provide any additional information on the event A. Therefore, the conditional
probability P(A | B) must be the same as the unconditional probability P(A).
The second part is just a useful trick in computations.

Example 2.24. Draw two cards from a deck (of 52 cards), without replacement.
Let A be the event that the first draw is the Ace of Clubs and let B be the event
that the second draw is the Queen of Diamonds. Then
1 1
P(A ∩ B) = P(A)P(B | A) = × .
52 51
Note that A and B are not independent, because if A happens, the second draw
cannot be the Ace of Clubs (this card was removed from the deck).

There are two more useful formulas related to conditional probabilities. The
first, Bayes’ theorem 6 , is a direct consequence of Lemma 2.23.

Theorem 2.25 (Bayes’ theorem). Let A, B be events with P(A), P(B) > 0.
Then
P(B | A)P(A)
P(A | B) = .
P(B)
6
Named after Reverend Thomas Bayes, who invented the concept of conditional probability
in 1763. (Yes, that long ago!)
Chapter 2 Probability basics 12

Proof. We have

P(A ∩ B) [Lemma 2.23] P(B | A)P(A)

P(A | B) = = .
P(B) P(B)

This also tells us that P(A | B) = P(B | A) only if P(A) = P(B). Otherwise,
the two conditional probabilities are only proportional up to factor determined
by the relative probabilities.
The last result relates to partitions of the sample space. SA partition of Ω is a
(finite or infinite) sequence of disjoint events Ai , such that i Ai = Ω. The next
result shows that any unconditional probability can be computed from a sum of
weighted conditional probabilities.

Theorem 2.26 (Law of total probability). Let A1 , . . . , Ak be a partition of Ω.

Then for any event B,
k
X
P(B) = P(B | Ai )P(Ai ).
i=1

Proof. Because A1 , . . . , Ak is a partition of Ω, it holds

k
[ k

[
B =B∩Ω=B∩ Ai = (B ∩ Ai ).
i=1 i=1

Furthermore, because the events A1 , . . . , Ak are disjoint, also the events (B ∩

A1 ), . . . , (B ∩ Ak ) must be disjoint. The third axiom then yields
k
[ 
P(B) = P (B ∩ Ai )
i=1
k
X
= P(B ∩ Ai )
i=1
Xk
[Lemma 2.23] = P(B | Ai )P(Ai ).
i=1

Let’s see these two results in action:

Example 2.27. Suppose I divide my email into three categories:

• A1 = ‘spam’,

• A2 = ‘low priority’

• A3 = ‘high priority’.
Chapter 2 Probability basics 13

From previous experience I know that

P(A1 ) = 0.7, P(A2 ) = 0.2, P(A3 ) = 0.1.

Let B be the event that the email contains the word ‘free’. From previous experi-
ence,

P(B | A1 ) = 0.9, P(B | A2 ) = 0.01, P(B | A3 ) = 0.01.

Now suppose I receive a new email containing the word free. What is the proba-
bility, that it is spam? Bayes’ theorem and the law of total probability yield

P(B | A1 )P(A1 )
P(A1 | B) =
P(B)
P(B | A1 )P(A1 )
= P3
i=1 P(B | Ai )P(Ai )
0.9 × 0.7
=
(0.9 × 0.7) + (0.01 × 0.2) + (0.01 × 0.1)
= 0.995.

2.5 Random variables

Recall the basic problem of statistics:

Having observed some data X1 , . . . , Xn , what can we say about the

mechanism that generated them?
This says nothing about sample spaces and events, so what was all this fuzz
about? The key is the concept of a random variable.

Definition 2.28 (Random variable). A random variable is a mapping

X:Ω→R

that assigns a real number X(ω) to each ω ∈ Ω.

Example 2.29. Flip a coin 5 times. Let X(ω) be the number of heads in the
sequence ω. For example, if ω = HHT T H, then X(ω) = 3.

Example 2.30. Let Ω = {(x, y) : x2 +y 2 ≤ 1} be the unit disk. Consider drawing

a point at random from Ω. A typical outcome is of the form ω = (x, y). Some
examples of random variables are X(ω) = x, Y (ω) = y, Z(ω) = x + y.

Random variables provide the link between sample spaces and events and the
data. In general, a random variable is any quantity whose actual value is random,
Chapter 2 Probability basics 14

i.e., dependent on a realization ω ∈ Ω. We then view the data X1 , . . . , Xn as n

realizations of a random variable X(ω).
In fact, we rarely mention the sample space Ω in statistics and work with random
variables directly. For example, we define X = X(ω) to be the luminosity of a
randomly chosen star in a galaxy. There are several choices we could make for
the underlying sample space (ω is the ID of a star, or ω is the luminosity itself,
or . . . ). If we’re just interested in the luminosity, the exact choice of the sample
space becomes irrelevant. For working with probabilities, we just need to know
that there is some sample space underlying the experiment.

2.6 Distribution functions

Random variables take on real (R) values by definition. That makes our life a lot
easier, because we can characterize probability distributions by more traditional
mathematical functions taking real numbers as arguments (as opposed to sets as
in Definition 2.13).

Definition 2.31. The cumulative distribution function (CDF) of the

random variable X is the function FX : R → [0, 1] defined by

FX (x) = P(X ≤ x) = P({ω : X(ω) ≤ x}).

Example 2.32. Suppose we flip a fair coin (so P(H) = P(T ) = 1/2) twice
and let X be the number of heads. Then P(X = 0) = P(X = 2) = 1/4 and
P(X = 1) = 1/2. The corresponding CDF is


 0 x<0

1/4 0 ≤ x < 1
FX (x) =


 3/4 1 ≤ x < 2
x ≥ 2.

1

The graph of this function is shown below:

Chapter 2 Probability basics 15

1.0
●

0.8
●

0.6
FX(x)
0.4
●
0.2
0.0

−1 0 1 2 3
x

It is a step function that jumps at 0, 1, 2, and 3 (the possible outcomes). The

size of the jump at x is equal to the probability P(X = x). The dots indicate that
at any jump point x, F (x) is equal to the value after the jump.

CDFs have a few useful properties. While Definition 2.31 gives a probabilistic
definition of CDFs, such functions can also be characterized analytically.

Theorem 2.33. A function F : R → [0, 1] is a CDF for some probability P

if and only if F satisfies the following conditions:

(i) F is non-decreasing: x1 < x2 ⇒ F (x1 ) ≤ F (x2 ),

(ii) limx→−∞ F (x) = 0, limx→∞ F (x) = 1,

(iii) F is right-continuous: F (x) = F (x+ ), where F (x+ ) = limy↓x F (y).

The proof isn’t hard but quite boring. Let’s turn to more interesting things.

2.7 Probability mass and density functions

CDFs are a bit hard to interpret, since we’re accumulating probabilities up to a
value x. It would be much easier if we would just know how probable a specific
value of x is. Probability mass and density functions do just that. To define
them, we need to discern discrete and continuous random variables.

Definition 2.34. A random variable X is called discrete, if it can take only

countably7 many different values, say {x1 , x2 , . . .}.

7
A set is countable if we can assign each of its elements with a natural number. Examples are
finite sets, e.g., {0, 1}, or countably infinite sets like N, Z, or Q.
Chapter 2 Probability basics 16

Definition 2.35. We define the probability mass function (PMF) of a

discrete variable X by
fX (x) = P(X = x).

The following properties follow directly from the definition of probabilities and
the CDF:
• For all x we have X
fX (x) ≥ 0, fX (xi ) = 1.
i

• The CDF is related to fX by

X
FX (x) = P(X ≤ x) = fX (xi ).
i : xi ≤x

In fact, we implicitly derived the CDF in Example 2.32 from the PMF fX . Here’s
another common example:

Example 2.36 (Bernoulli distribution). Consider a coin flipping experiment and

define X by X(H) = 1, X(T ) = 0. Then P(X = 1) = p and P(X = 0) = 1 − p
for some p ∈ [0, 1]. This is called the Bernoulli(p) distribution and has PMF

1 − p for x = 0

f (x) = p for x = 1

0 otherwise,


The CDF is 
0
 for x < 0
F (x) = 1 − p for 0 ≤ x < 1

1 for x ≥ 1.


The Bernoulli distribution is super common in statistics because it applies to

any random variable that only has two possible outcomes (win/lose, pass/fail,
cat/dog, . . . ). Later in this course, we’ll see more examples of common discrete
distributions.

Definition 2.37. A random variable X is called continuous, if the cdf FX (x)

is continuous.

Maybe it’s not obvious, but this definition already implies that X can take
uncountably many values. (Otherwise the CDF must jump somewhere.) In this
case, the concept of a PMF is meaningless, because P(X = x) = 0 for all x.8
Instead, we use a slightly different concept, a density function.
8
The sum of uncountably many strictly positive numbers is always infinite. Hence, P(X =
x) > 0 is only possible for countably many x.
Chapter 2 Probability basics 17

Definition 2.38. If FX (x) is differentiable, the function fX (x) = FX0 (x) is

called the probability density function (PDF).

From this definition it follows that

• fX (x) ≥ 0,
R∞
• −∞ fX (t)dt = 1,
Z x
• FX (x) = fX (t)dt.
−∞

The density fX (x) is not a probability. Instead it is interpreted as a relative

likelihood. Regions where fX is large are more likely; regions where fX is small
are less likely. (You’ve probably seen the concept a density as mass per unit
volume. The idea here is similar.)

Example 2.39 (Uniform distribution). Suppose that a < b and X has PDF
(
1/(b − a) for a < x < b
fX (x) =
0 otherwise.
R∞
Clearly, fX (x) ≥ 0 and −∞ fX (x)dx = 1. A random variable with such a PDF is
said to have a Uniform(a, b) distribution. The CDF is given by

0
 x<a
FX (x) = (x − a)/(b − a) a ≤ x ≤ b

1 x > b.


The PDF and CDF of the Uniform(0, 1) distribution are shown below:
PDF of Uniform(0, 1) CDF of Uniform(0, 1)

1.0 1.0
FX(x)
fX(x)

0.5 0.5

0.0 0.0

−0.5 0.0 0.5 1.0 1.5 −0.5 0.0 0.5 1.0 1.5
x x

The graph of the PDF indicates that all values in the interval (0, 1) are equally
likely, which explains the distribution’s name.

Note that, unlike a PMF, a PDF can be larger than 1 (and even unbounded!).
Chapter 2 Probability basics 18

Example 2.40. LetR f (x) = (2/3)x−1/3 for 0 < x < 1 and f (x) = 0 otherwise.
∞
Then f (x) ≥ 0 and −∞ f (x)dx = 1, but f is unbounded.

This reminds us to not misinterpret a PDF as a probability, but a relative

likelihood.
Now that we know about continuous random variables, we can state a few
more properties of the CDF.

Lemma 2.41. Let F be the CDF of a random variable X. Then

(i) P(X = x) = F (x) − F (x− ), where F (x− ) = limy↑x F (y);

(ii) P(x < X ≤ y) = F (y) − F (x);

(iii) P(X > x) = 1 − F (x);

(iv) If X is continuous, then

F (b) − F (a) = P(a < X < b)

= P(a < X ≤ b)
= P(a ≤ X < b)
= P(a ≤ X ≤ b).

2.8 Bivariate distributions

Similar concepts apply to the joint distribution of multiple random variables.
Joint distributions come into play when we are interested in events that depend
on several variables. For example, the probability that tomorrow there is no rain
(X) and the temperature (Y ) is more than 20 degrees. For simplicity, we shall
only consider joint distributions of two random variables here, but the concept
naturally extends to higher dimensions.

Definition 2.42. We define the joint CDF as

FX,Y (x, y) = P(X ≤ x and Y ≤ y) = P(X ≤ x, Y ≤ y).

We must again differentiate discrete and continuous random variables.

2.8.1 Discrete variables

Definition 2.43. Let X and Y be discrete random variables. The joint PMF
is
f (x, y) = P(X = x and Y = y) = P(X = x, Y = y).
Chapter 2 Probability basics 19

Example 2.44. Here is a bivariate distribution for random variables X and Y

both taking values 0 or 1:
Y =0 Y =1
X = 0 1/9 2/9 1/3
X = 1 2/9 4/9 2/3
1/3 2/3 1

For instance, fX,Y (1, 1) = P(X = 1, Y = 1) = 4/9.

From a joint distribution, we can also extract the distributions of the individual
variables. The latter are called marginal distributions.

Definition 2.45. If (X, Y ) has the joint mass function fX,Y , the marginal
mass function for X is
X X
fX (x) = P(X = x) = P(X = x, Y = y) = f (x, y).
y y

Likewise, the marginal mass function for Y is

X X
fY (y) = P(Y = y) = P(X = x, Y = y) = f (x, y).
x x

As you can see, we extract the marginal PDFs by summing the joint PMF over
all possible values of the other variable. For example, in Example 2.44 we have
fX (0) = 1/3 and fY (1) = 2/3.

2.8.2 Continuous variables

Similarly, we define a joint PDF for continuous random variables.

Definition 2.46. Let X and Y be continuous random variables. A function

f (x, y) is called the joint PDF of (X, Y ), if

(i) f (x, y) ≥ 0 for all (x, y);

R∞ R∞
(ii) −∞ −∞ f (x, y)dxdy = 1;

(iii) for any set A ⊂ R × R,

Z Z


P (X, Y ) ∈ A = f (x, y)dxdy.
A

(Compare this to the properties stated after Definition 2.38.)

Chapter 2 Probability basics 20

Example 2.47. Let

(
1 if 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
f (x, y) =
0 otherwise.

This is the PDF of the uniform distribution on the unit square. Suppose we want
to compute

P(X ≤ 1/2, Y ≤ 1/2) = FX,Y (1/2, 1/2) = P(X < 1/2, Y < 1/2).

Integration gives that FX,Y (1/2, 1/2) = 1/4.

Marginal densities are constructed like in the discrete case, just replacing the
sum by an integral.

Definition 2.48. For continuous random variables (X, Y ) with a joint PDF
f (x, y) the marginal PDFs are
Z Z
fX (x) = f (x, y)dy, fY (y) = f (x, y)dx.

The corresponding marginal CDFs are denoted by FX and FY .

Example 2.49. Let

(
e−(x+y) for x ≥ 0 and y ≥ 0
fX,Y (x, y) =
0 otherwise.

Integration gives that fX (x) = e−x .

Example 2.50. Let

(
x+y if 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
fX,Y (x, y) =
0 otherwise.

Integration gives that fY (y) = 1/2 + y for 0 ≤ y ≤ 1 and fY (y) = 0 otherwise.

2.9 Conditional distributions

We can also apply the concept of conditional probability to random variables and
associated functions. The conditional PMF of a discrete random variable X,
given that a discrete random variable Y takes the value y, is

P(X = x, Y = y) fX,Y (x, y)

fX|Y (x|y) = P(X = x | Y = y) = = ,
P(Y = y) fY (y)
Chapter 2 Probability basics 21

provided fY (y) > 0. The conditional PDF of a continuous random variable X,

given that a continuous random variable takes the value y is
( f (x,y)
X,Y
fY (y)
, for fY (y) > 0
fX|Y (x | y) =
0, for fY (y) = 0.

Then Z
P(X ∈ A | Y = y) = fX|Y (x | y)dx,
A

so fX|Y (· | y) is a proper density function for the conditional probability.

Example 2.51. Let

(
x+y if 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
fX,Y (x, y) =
0 otherwise.

Since fY (y) = y + 1/2 for 0 ≤ y ≤ 1,

fX,Y (x, y) x+y

fX|Y (x | y) = =
fY (y) y + 1/2

for 0 ≤ x ≤ 1, 0 ≤ y ≤ 1 and fX|Y (x|y) = 0 otherwise. Thus,

  Z 1/4  
1 1 1 11
P X< |Y = = fX|Y x| dx = .
4 3 0 3 80

2.10 Independence (ctd’)

We call two random variables independent, if all events associated with the two
variables are independent.

Definition 2.52. Two random variables X and Y are independent (denoted

X ⊥ Y ), if for every A and B

P(X ∈ A, Y ∈ B) = P(X ∈ A)P(Y ∈ B).

The interpretation is the same as for events: two variables are independent, if
their outcomes are entirely unrelated (do not influence each other).
Independence can also be characterized using densities (this follows immediately
from their definitions).

Theorem 2.53. Let X and Y have joint PDF (or PMF) fX,Y . Then X ⊥ Y
if and only if fX,Y (x, y) = fX (x)fY (y) for all (x, y).

Example 2.54. Let X and Y have the joint distribution as in the following table:
Chapter 2 Probability basics 22

Y =0 Y =1
X = 0 1/4 1/4 1/2
X = 1 1/4 1/4 1/2
1/2 1/2 1
Then X and Y are independent, which can be verified by the previous theorem.
For example, f (0, 0) = 1/4 = fX (0)fY (0), and similarly for other cases.

Example 2.55. Let X and Y be independent and both have the same PDF
(
2x if 0 ≤ x ≤ 1
f (x) =
0 otherwise.
Suppose we want to find P(X + Y ≤ 1). Thanks to independence,
(
4xy if 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
f (x, y) = fX (x)fY (y) =
0 otherwise.

So Z Z
1
P(X + Y ≤ 1) = f (x, y)dxdy = .
x+y≤1 6

Let’s back up for a second and think about the bigger picture. Our goal in
this course is to learn from data. These data are modeled as random variables
X1 , . . . , Xn . In many circumstances it is reasonable to assume these random
variables are independent. For example, if Xj is the jth flip of a coin, we have no
reason to assume that the outcome of one flip is affecting another. The situation
is similar in other repeated experiments or some measurements taken on distinct
objects (e.g., stars, galaxies, . . . ).
We need to be a little bit more precise here. Independence of X1 , dots, Xn is
more than just pairwise independence of all Xi , Xj :

Definition 2.56. X1 , . . . , Xn are independent, if for every A1 , . . . , An

n
Y
P(X1 ∈ A1 , . . . Xn ∈ An ) = P(Xi ∈ Ai ).
i=1

This definition is stronger than pairwise independence. The interpretation is

as follows: the joint outcome of (Xi1 , . . . , Xik ) is entirely unrelated to the joint
outcome of (Xj1 , . . . , Xj` ), whenever no variable appears twice, i.e., {i1 , . . . , ik } ∩
{j1 , . . . , j` } = ∅.
There is another assumption commonly made in statistics: that each of the
variables X1 , . . . , Xn has the same distribution, i.e. FX1 = . . . , FXn .

Definition 2.57. If X1 ∼ F, . . . , Xn ∼ F are independent, we say that

X1 , . . . , Xn are iid (independent and identically distributed).
Chapter 2 Probability basics 23

If X1 , . . . , Xn are iid, we also write X1 , . . . , Xn ∼ F . If F has density f we also

write X1 , . . . , Xn ∼ f .

2.11 Transforms
Sometimes we know the distribution of random variable X, but are interested in
the distribution of a transformation r(X). The following result comes in handy.

Theorem 2.58. Suppose X is a continuous random variable, r is a invert-

ible9 and differentiable10 function and Y = r(X). Then

fX (r−1 (y))
fY (y) = 0 −1 .
|r (r (y))|

Similar results exist for discrete variables and joint densities. You can look
them up yourself11 whenever there’s a need. The above result will be good enough
for this course.

Example 2.59. Let Y = X 3 . The function r(x) = x3 is strictly increasing and,

hence, invertible.

100

50
r(x)

−50

−100

−5.0 −2.5 0.0 2.5 5.0

x

dr(y)
Now r−1 (y) = y 1/3 and dy
= 3y 2 . So

fX (y 1/3 ) fX (y 1/3 )
fY (y) = = .
|3(y 1/3 )2 | 3 |y 2/3 |

2.12 Expectation
Distributions are functions and, thus, fairly complex objects. Instead we can
also look at summaries of the distribution. The most important summary is the
expect value E[X]. It tells us what value a random variable X we can expect to
see on average. For the formal definition, we once again need to discern discrete
and continuous cases.
10
A function is invertible if and only if it is strictly monotone (increasing or decreasing).
10
Whenever you can compute the derivative, it is differentiable. This will rarely be an issue.
11
https://en.wikibooks.org/wiki/Probability/Transformation_of_Probability_
Densities
Chapter 2 Probability basics 24

Definition 2.60. The expected value (also mean or first moment) of a

random variable X is defined as
X
Z 

 xf (x) if X is discrete
x∈Ω
E[X] = xdF (x) = Z
Ω
 xf (x)dx if X is continuous.


Ω

R
The integral Ω xdF (x) has a precise measure theoretic meaning that you don’t
need to worry about. You may just treat it as short hand notation for one of the
two cases on the right.
In both cases, the expected value is an average over all possible outcomes of X,
weighted by the likelihood of occurrence. Another way to think about it is the
following approximation: repeat
Pn the same experiment many times and average all
1
outcomes, then E[X] ≈ n i=1 Xi , for a large number of iid draws X1 , . . . , Xn
with the same distribution as X. While this is only an approximation, it helps
to understand the meaning of the number E[X]. Let’s see some examples.

Example 2.61. Let X ∼ Bernoulli(p). Then E[X] = 1 × p + 0 × (1 − p) = p. So

if we flip a fair coin many times, count heads as 1 and tails as 0, we expect to
see a value of 0.5 on average.

Example 2.62. Let X denote the outcome of a single throw of a fair die. Then
E[X] = (1 + 2 + .. + 6) × 1/6 = 3.5.

Example 2.63. Let X ∼ Uniform(a, b). Then

b
b 2 − a2 (b + a)(b − a)
Z
1 a+b
E[X] = xdx = = = .
(b − a) a 2(b − a) 2(b − a) 2

Hence, if we draw uniformly from the interval (1, 2) many times, we expect to see
a value of 1.5 on average.

As you can see, computing the expectation of a random variable can be fairly
easy. It’s similarly easy to compute the expectation of a transformed random
variable.

Theorem 2.64. Let Y = r(X). Then

Z
E[Y ] = E[r(X)] = r(x)dFX (x).

Example 2.65. Let X ∼ Bernoulli(p) and Y = X 2 . Then

E[Y ] = E[X 2 ] = 12 × p + 02 × (1 − p) = p.
Chapter 2 Probability basics 25

Example 2.66. Let X ∼ Uniform(0, 1) and let Y = r(X) = eX . Then

Z Z 1
x
E[Y ] = E[r(X)] = e f (x)dx = ex dx = e − 1.
R 0

The same also works when more than one random variable is involved, just
that we need to use the joint PMF/PDF as a weight in the sum/integral. For
example, for two continuous random variables X1 , X2 and Y = r(X1 , X2 ),
Z
E[Y ] = E[r(X1 , X2 )] = r(x1 , x2 )fX1 ,X2 (x1 , x2 )dx1 dx2 .

The expectation has the nice property of being linear, which just means that
you can pull the sum and any constants out of it. If you think about our
approximation of E[X] as averaging over many draws from an experiments, this
makes sense (and it follows immediately from Definition 2.60).

Theorem 2.67. If X1 , . . . , Xn are random variables and a0 , a1 , . . . , an are

constants, then " #
Xn n
X
E a0 + ai X i = a0 + ai E[Xi ].
i=1 i=1

(In particular, E[X1 + X2 ] = E[X1 ] + E[X2 ].)

iid
Example 2.68. Let Y1 , . . . , Yn ∼ Bernoulli(p) and X = ni=1 Yi . Then we say
P
X has Binomial(n, p)-distribution. By linearity of the expectation, it holds
n
X
E[X] = E[Yi ] = np.
i=1

While sums of random variables are easy to handle, this is not in general
true for products. A convenient and common special case is when variables are
independent.

Theorem 2.69. If X1 , . . . , Xn are independent random variables, then

" n # n
Y Y
E Xi = E[Xi ].
i=1 i=1

This might fail without independence assumption!

Proof. Let’s just look at the continuous case and n = 2. If X1 , X2 are independent,
the joint density fX1 ,X2 is just the product of marginal densities fX1 × fX2 . Hence,
Chapter 2 Probability basics 26

by Theorem 2.64
Z
E[X1 X2 ] = x1 x2 fX1 ,X2 (x1 , x2 )dx1 dx2
Z
= x1 x2 fX1 (x1 )fX2 (x2 )dx1 dx2
Z Z
= x1 fX1 (x1 )dx1 × x2 fX2 (x2 )dx2

= E[X1 ] × E[X2 ],

where we used independence in the second equality.

2.13 Variance and standard deviation

The expectation is important, but alone it gives a very limited view on the
distribution. Imagine a lottery that always pays out the expected value: you pay
€10 for your ticket and you immediately get €8 back. Nobody would play that
game! What makes lotteries exciting (for some people at least) is the variability
of the outcome. The most common measures for variability are the variance and
standard deviation.

Definition 2.70. Let X be a random variable with mean E[X] = µ.

• The variance of X is defined as

Z
2
V[X] = E[(X − µ) ] = (x − µ)2 dF (x).

• The standard deviation of X is sd[X] =

p
V[X].

Both variance and standard deviation are one-number summaries of the dis-
tribution. They answer the question: “how much does X fluctuate around its
mean”? In the extreme case V[X] = 0, there is no variability at all and X is just
a constant. While the variance is easier to calculate with, the standard deviation
is easier to interpret. Because we square what’s in the expectation, we need to
take the square root of the result to bring it back to the original scale/units.
Keep that in mind when reporting or reading about these measures.
Chapter 2 Probability basics 27

Theorem 2.71. The variance has the following properties:

(i) V[X] = E[X 2 ] − E[X]2 .

(ii) If a and b are constants, then V[aX + b] = a2 V[X].

(iii) If X1 , . . . , Xn are independent (!) and a1 , . . . , an , b are constants, then

" n # n
X X
V ai X i + b = a2i V[Xi ].
i=1 i=1

You can verify this an exercise. The key ingredient is linearity of the expectation
Theorem 2.67.

Example 2.72. Let Y ∼ Bernoulli(p). Recall that E[Y 2 ] = p and E[Y ] = p.

Therefore V[Y ] = p − p2 = p(1 − p).
Pn
Example 2.73. Suppose X ∼ Binomial(n, p). Recall that X = i=1 Yi for
Y1 , . . . , Yn iid Bernoulli(p) random variables. Therefore
n
X
V[X] = V[Yi ] = np(1 − p).
i=1

2.14 Covariance and correlation

Now we know about the two most important summaries of the distribution of
a single random variable. When there’s another variable, a new concept comes
into play: the dependence between variables. The only thing we know about
dependence so far is the concept of independence: two variables are completely
unrelated. It’s not hard to imagine a situation where variables are related. For
example, a person’s body weight tends to be related to her height. Again, the
complete picture of the relationship is captured by the joint distribution (or
PMF/PDF), but there are one-number summaries of the dependence.

Definition 2.74. Let X and Y be random variables with means µX and µY

and standard deviations σX and σY , respectively.

• The covariance between X and Y is defined as

Cov[X, Y ] = E [(X − µX )(Y − µY )] .

• The correlation between X and Y is defined as

Cov[X, Y ]
ρX,Y = ρ(X, Y ) = .
σX σY
Chapter 2 Probability basics 28

Similar to the variance, the covariance is a bit harder to interpret (just think about
its units), mainly because it mixes two things: i) the individual variability of X
and Y , ii) the dependence between X and Y . The correlation is a standardized
version that takes out the variability part. That makes it a pure measure of
dependence, which is typically what we want.

Theorem 2.75. (i) The covariance satisfies Cov[X, Y ] = E[XY ] −

E[X]E[Y ].

(ii) The correlation satisfies ρ(X, Y ) ∈ [−1, 1].

(iii) If Y = aX + b for some constants a and b, then

(
1, if a > 0
ρ(X, Y ) = .
−1, if a < 0.

(iv) If X and Y are independent, then ρ(X, Y ) = 0.

(The converse is in general false.)

The first property is useful mainly for calculations. The second tells us that the
correlation is standardized to the interval [−1, 1]. The sign and magnitude of the
correlation tell us what kind of dependence we are dealing with. Let’s consider
the three extreme cases. As shown in (iii), the correlation has absolute magnitude
1 when two variables are perfectly linearly related. The sign tells us how:

• ρ(X, Y ) > 0: X and Y tend to be both large or both small at the same
time.

• ρ(X, Y ) < 0: large values of X tend to occur with small values of Y and
vice versa.

Finally, if ρ(X, Y ) = 0, we say that Y and X are uncorrelated. This is always

the case when X and Y are independent.
Keep in mind that correlation is only a measure of linear dependence. There are
other forms of dependence where Y and X are perfectly related (e.g., Y = X 2 ),
but the correlation is not 1. Similarly, there are cases where ρ(X, Y ) = 0, but
the variables are not independent.
Now that we know about covariances, we can drop the independence assumption
in Theorem 2.71 (iii):

Theorem 2.76. For random variables X1 , . . . , Xn and constants a1 , . . . , an ,

" n # n
X X X
V ai X i = a2i V[Xi ] + 2 ai aj Cov[Xi , Xj ].
i=1 i=1 i<j
Chapter 2 Probability basics 29

We see that, in general, we need an extra correction term for the variance of
a sum. To understand why, consider the case where X1 = −X2 . Then clearly
X1 + X2 = 0, so there is no variability at all. If we would just sum up the
variances of X1 and X2 , we would get 2V[X1 ] 6= 0. The covariance term in the
theorem above fixes this.

2.14.1 Conditional expectation

The final concept that we need from probability theory is the conditional ex-
pectation. As you might expect, it is the expectation related to a conditional
distribution. Knowing this, its mathematical definition is just what you expect:12

Definition 2.77. The conditional expectation of X given Y = y is

(P
xfX|Y (x|y) discrete case
E[X|Y = y] = R x
xfX|Y (x|y)dx continuous case.

Recall that the unconditional expectation E[X] is interpreted as “the value of

X we expect to see on average”. Now suppose we have already observed some
other information Y = y. Which value do we expect now on average? That’s it,
the conditional expectation E[X | Y = y]. Let’s make this a bit more concrete
and go back to the example with body height (X) and weight (Y ). If we draw a
random person from the Dutch population, we expect his height to be roughly
1.80m (E[X] ≈ 1.80). What if I told you this person weighs 50kg (Y =≈ 50).
How tall do you expect this person to be now (E[X | Y = y])?
The concept should be easy to grasp, the mathematically, the conditional ex-
pectation is a bit trickier than the unconditional one. The conditional expectation
is different if we change the information we condition on. (If I tell you the person
weighs 90kg the answer will be different than above.) So while µ = E[X] is a
number, µ(y) = E[X | Y = y] is a function.
Now, what happens if we plug the random variable Y into the function µ(y)?
Well we get another random variable defined as Z = µ(Y ). This random variable
is typically denoted as E[Y | X] (which, admittedly, is a bit confusing but you’ll
get used to it). Of course, this variable has a distribution and we can compute
its expectation.

Theorem 2.78 (Law of total expectation/Tower rule). It holds

E[E[X|Y ]] = E[X].

The Tower rule is a convenient tool for computations as we’ll see in a minute.
Let’s first verify that the formula makes sense. Suppose we know the exact
12
You see what I did there?
Chapter 2 Probability basics 30

function µ(y) stating what body height to expect given every possible value of
the body weight y > 0. Now we start drawing random people from the population,
check their weight Y , and compute the expected height Z = µ(Y ). On average,
our guesses Z should equal the average height in the population E[X].

Example 2.79. Suppose Y ∼ Uniform(0, 1). After we observe Y = y, we draw

X|Y = y ∼ Uniform(y, 1). Note that fX|Y (x|y) = 1/(1 − y) and hence
Z 1
1 1+y
E[X|Y = y] = x dx = .
y 1−y 2

So E[X|Y ] = (1 + Y )/2. This is a random variable, whose value is (1 + y)/2 after

Y = y is observed. To compute E[X], we write
 
1+Y 1 3
E[X] = E[E[X|Y ]] = E = (1 + E[Y ]) = .
2 2 4

More generally, it also holds

E[E[r(X, Y )|Y ]] = E[r(X, Y )].

and there is a similar result for the variance (”law of total variance”). We won’t
need it for this course, but it’s good to know it exists.
 Descriptive statistics and
3
exploratory data analysis
The last chapter introduced the fundamentals of probability theory to set the
stage for the main objective of this course: doing statistics (or ‘analyzing data’).
Recall the basic problem of statistics:

Having observed some data X1 , . . . , Xn , what can we say about the

mechanism that generated them?

By now we know that the data X1 , . . . , Xn are modeled as random variables. We

have learned that they are characterized by a probability distribution. And if
this distribution is known, we know how to interpret and summarize it.
But how do we know the distribution of the data? Well, we don’t — and that’s
what distinguishes probability theory from statistics. The data we observe gives
us some clues how the distribution may look like and we wish to extract as much
information as possible. The first step is to summarize and explore the data.
Here the aim is to ‘get a feeling’ for the data before we to do actual modeling
and inference. This process is called exploratory data analysis (EDA).

3.1 Sample averages and the law of large numbers

Suppose our data X1 , . . . , Xn are iid random variables from an unknown distri-
bution F . We start with the most basic task: learning about their expectation.
Because the data are iid, there is a number µ ∈ R such that E[Xi ] = µ for all
i = 1, . . . , n. Recall that µ is called expectation because that’s the value we
expect the Xi ’s to take on average. Now the problem is that we can’t know µ
because we don’t know F . But is there an observed quantity that approximates
µ? The average of the Xi ’s is an obvious candidate.

Definition 3.1. For data X1 , . . . , Xn , the sample average or sample

mean is defined as
n
1X
X̄n = Xi .
n i=1
Chapter 3 Descriptive statistics and exploratory data analysis 32

Intuitively, we expect X̄n to be a reasonable approximation of µ. And our

intuition is right: we shall see that X̄n → µ as n → ∞ (in a certain sense). In
plain words: as we get more data, X̄n gets closer and closer to µ. There is a
little twist to the story though. Because X1 , . . . , Xn are random variables, X̄n is
a random variable, too! So we first need a notion of convergence that accounts
for randomness:

Definition 3.2 (Convergence in probability). Let Y1 , . . . , Yn , be a sequence of

random variables and Y another random variable. We say that Yn converges
to Y in probability or Yn →p Y , if for every  > 0,


lim P |Yn − Y | >  = 0.
n→∞

In plain words, Yn →p Y means: as we get more and more data (n → ∞), the
probability that Yn is  away from Y goes to 0. You could also write the definition
the other way around: for every  > 0,


lim P |Yn − Y | ≤  = 1.
n→∞

So with probability going to 1, Yn and Y become arbitrarily close to each other.

The general definition above involves a random variable Y as the limit. In
most cases of interest, the limit Y is actually a constant (i.e., a random variable
with zero variance). Generally speaking, most rules for “usual” limits also
apply for limits in probability. For example, Yn →p Y and Xn →p X imply
Yn + Xn →p Y + X, and so on. We won’t go into detail here; you may just assume
that all the rules you know from analysis apply.
Now we’re all set to state what I like to call the fundamental theorem of
statistics.

Theorem 3.3 (The law of large numbers, LNN). Let PnX1 , . . . , Xn be iid random
1
variables with E[Xi ] = µ < ∞ and define X̄n = n i=1 Xi . Then

X̄n →p µ.

While we can’t know µ, X̄n is something we observe. The LNN implies that
the sample mean X̄n is a reasonable approximation of µ. Hence, X̄n gives us
a “feeling” what the actual mean µ might be. The LNN makes this intuition
mathematically precise. It allows us to learn about the expected value of an
unknown random mechanism just from seeing the data.

Example 3.4. Let’s illustrate the LLN with a small experiment: We simulate
X1 , . . . , Xn ∼ Bernoulli(0.5) and compute X̄n for each n. We repeat this exper-
iment three times. By the law of large numbers, we expect the three resulting
sequences to converge to the expected value E[X1 ] = 0.5. The results are shown
in Figure 3.1. Each line (color) corresponds to a sequence X̄1 , X̄2 , X̄3 , . . . , one
Chapter 3 Descriptive statistics and exploratory data analysis 33

1.00

0.75

Xn
0.50

0.25

0.00
0 1000 2000 3000
n

Figure 3.1: The law of large numbers in action (Example 3.4). Each line corre-
sponds to a sequence X̄n after simulating from n iid Bernoulli(0.5)
random variables.

line for each repetition of the experiment. We see that for small n, X̄n can be
quite far away from the mean. As we increase the amount of data, all three lines
seem to stabilize around 0.5. However, the three lines are different, reflecting
the randomness of our sample. The green line lies mainly above 0.5, the others
mainly below. The LLN states that, despite this randomness, it becomes less and
less likely that one of the lines ends up away from 0.5.

Remark 3.1. Just as a side note: There is a stronger version of the LLN called
the strong law of large numbers (SLLN). It involves a different notion of con-
vergence called almost sure convergence. It states that the the probability that
the sequence Xn converges to µ is exactly 1: P(limn→∞ Xn = µ) = 1. Here
we’re making a probability statement about the limit limn→∞ Xn . Convergence
in probability is a statement about convergence of probabilities. For us, conver-
gence in probability will be enough, but it’s good to have heard about almost sure
convergence.

3.1.1 Estimators and consistency

The statement “X̄n is a good approximation of µ” is made mathematically precise
by Xn →p . In that case, we say that X̄n is a consistent estimator for µ. Let us
put this in a slightly more abstract setting.

Definition 3.5 (Estimator). If X1 , . . . , Xn is our data, any quantity that can

be expressed as g(X1 , . . . , Xn ) for some function g is called an estimator.

Less formally, an estimator is any number that you compute from data.
Chapter 3 Descriptive statistics and exploratory data analysis 34

Definition 3.6 (Consistency). Let θ be an unknown quantity that we are

interested in. An estimator θbn is called consistent for θ if

θbn →p θ.

Example 3.7. Let θ = E[X]. Then θbn = X̄n is a consistent estimator.

3.2 Sample (co)variances

The sample mean is an estimator for the expectation. We can similarly find
estimators for other summaries of a distribution.

Definition 3.8.
n n
1X
2 1X 2
• Sample variance: Sn2 = Xi − X̄n = X − (X̄n )2 .
n i=1 n i=1 i

• Sample standard deviation: Sn =

p
Sn2 .

Theorem 3.9. If X1 , . . .p
, Xn are iid samples from a distribution F , it holds
2
Sn →p V[X] and Sn →p V[X], for a random variable X ∼ F .

Proof. By the LLN X̄n →p E[X]P and, thus, (X̄n )2 →p E[X]2 . Similarly (defining
Yi = Xi2 ), the LLN gives that n1 ni=1 Xi2 →p E[X 2 ]. In combination this yields
n
1X 2
Sn2 = X − (X̄n )2 →p E[X 2 ] − E[X]2 = V[X].
n i=1 i
p p
Finally, because the square root is a continuous function, Sn = Sn2 →p V[X].

Similarly, we can define estimators of the covariance and correlation for two-
dimensional iid data (X1 , Y1 ), . . . , (Xn , Yn ):
Chapter 3 Descriptive statistics and exploratory data analysis 35

Definition 3.10.
n
1X
• Sample covariance: Cn =



Xi − X̄n Yi − Ȳn .
n i=1

Cn
• Sample correlation: Rn = q q ,
2 2
Sn,X Sn,Y
where
n n
2 1X
2 2 1X
2
Sn,X = Xi − X̄n , Sn,Y = Yi − Ȳn .
n i=1 n i=1

Again using the LLN, we find that Cn and Rn are consistent estimators for
Cov[X, Y ] and ρ(X, Y ), respectively.

3.2.1 Using summaries in EDA

The interpretation of the sample quantities is similar to their population versions1

• The sample mean is a measure of location.

• Sample variance and standard deviation are measures of variability.

• Sample covariance and correlation are measures of dependence.

Computing these summaries at the very start of a data analysis is a good idea.
They give us a “feeling” of the behavior of certain variables: location, variability,
and dependence.
Let’s see a few examples. Figures 3.2 to 3.4 show scatterplots of two variables
X and Y : each dot represents one sample (Xi , Yi ) in the data set (with Xi drawn
on the x-axis and Yi on the y-axis). The figure captions contain the sample means,
standard deviations, and correlations computed from these data sets. As you
can see from Figure 3.2, X is located around 0.5, Y around 3. The variability
of X is much larger than the variability of Y , which is reflected in the sample
standard deviations. The correlation is around 0, so there’s not much dependence
going on. In Figure 3.3, the sample means and standard deviations remain the
same, but now we have a correlation of 0.62. We can see this dependence in
the scatterplot by the upward trend in the data: when X is small, Y tends to
be small; when X is large, Y tends to be large. This is what we call positive
dependence. In Figure 3.4 the correlation changes to −0.82. So now we have
negative dependence which is reflects the downward trend: when X is small, Y
tends to be large; when X is large, Y tends to be small.

1
Everything computed from the actual (unknown) distribution F is called a “population
version”. Everything computed from data is called a “sample version”.
Chapter 3 Descriptive statistics and exploratory data analysis 36

10

●
●● ●

5 ● ●
● ● ●● ●
●
●
●
● ●
●
● ●●
●

●
●
●
●

● ● ● ● ● ●
●●
● ● ●● ●●● ● ● ● ●●
●
● ●● ●
● ● ● ● ●● ●●
● ● ● ● ● ●●● ●
● ● ● ●
●
● ●● ● ● ● ●
● ●● ● ● ● ●● ●
● ●● ● ● ● ●● ●● ● ● ● ●
● ●● ●● ● ●● ● ● ●
● ●● ● ● ● ●●
● ● ●
● ●●● ● ● ● ● ●
●
●
● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●
● ● ●●●● ●●● ● ●
● ● ● ● ●●
● ● ●●
● ●●
●● ● ● ●
● ● ● ● ● ● ● ●
● ● ●
● ● ●

Y
●

−5

−10
−10 −5 0 5 10
X

Figure 3.2: Example with X̄n = 0.54, Ȳn = 3.02, Sn,X = 2.85, Sn,Y = 1.02,
Rn = −0.04.

10

●
● ●
●
5 ● ● ●
●
●
●●

●
●

●●
● ●
●
●
●

●
●
●

●
● ● ●●●
●● ● ● ● ● ●
● ● ● ●
●●
● ●●● ●●● ●●
● ● ● ●
● ● ● ●● ● ● ●● ●● ●● ● ●
● ● ● ●●
● ● ●● ● ● ●● ●● ●
●●● ● ● ● ●
● ● ●● ●
● ●
●
● ● ● ●● ● ● ● ●
● ●● ● ●● ●●●●●●●
● ● ●●
●● ● ●
● ● ● ● ●● ●●
●●● ● ● ● ●
●● ● ●
● ● ●● ● ●
● ● ● ●
●●● ● ● ●● ●
● ●●
●
● ●● ● ● ● ● ●● ●● ●● ●●
● ●
● ●
● ●
● ● ●
●
0
Y

−5

−10
−10 −5 0 5 10
X

Figure 3.3: Example with X̄n = 0.54, Ȳn = 3.02, Sn,X = 2.85, Sn,Y = 1.02,
Rn = 0.62.

10

● ●●

5 ●
●
● ●
●
● ●
●
●
●● ● ● ● ●
●

● ●●● ●● ●● ●● ● ● ●●
● ●● ● ●
● ● ● ● ●● ● ● ●●●
● ●
● ● ●● ● ● ●● ●●
●●● ●
● ●● ●●● ●
● ●
●● ● ●●●● ●● ● ●●● ●
●●
● ● ●● ● ●● ● ●● ● ●
●● ● ●●●
● ● ● ●● ● ● ●
● ●● ●
● ●
●● ● ●●● ● ●● ● ● ●
●●●●● ●● ● ●
●
● ●
● ● ● ●
● ●●●● ● ● ● ● ● ● ●
● ●● ● ● ● ● ●
● ●
● ●●
● ● ●
● ● ●● ● ●
● ● ●
● ●●● ● ●
●

0
Y

●
●

−5

−10
−10 −5 0 5 10
X

Figure 3.4: Example with X̄n = 0.54, Ȳn = 3.02, Sn,X = 2.85, Sn,Y = 1.02,
Rn = −0.82.
Chapter 3 Descriptive statistics and exploratory data analysis 37

Figure 3.5: The Datasaurus: all scatterplots have the same mean, standard devi-
ation, and correlation.

Beware of the Datasaurus!

Summarizing the data into a few numbers is nice, because it gives us a quick
overview of what’s going on. But never forget that this is a simplification! Two
data sets with the same summaries can be wildly different. The Datasaurus has
established itself as the mascot of this piece of wisdom. All scatterplots in the
figure have the same means, standard deviations, and correlations. Yet the data
sets couldn’t be more different. Take this as a cautionary tale: summarizing your
data first is fine, but always make plots to check what’s really going on.

3.3 The empirical distribution function

Instead of estimating summaries of a distribution F , we can also estimate the
distribution function itself. To do so, we define the indicator function
(
1, A is true,
1(A) =
0, A is not true.

So how does this help to estimate F ? You can verify that F (x) = P(X ≤
x) = E[1(X ≤ x)]. So estimating F isn’t much different from estimating an
expectation (except that we have to estimate one expectation for every x ∈ R).
Chapter 3 Descriptive statistics and exploratory data analysis 38

1.00 ●

●
0.75

●
Fn(x)

0.50
●

0.25
●

observations
0.00
0.0 0.4 0.8
x

Figure 3.6: The empirical cumulative distribution function of a simulated data

set X1 , . . . , X5 . Orange crosses indicate the observations.

Definition 3.11 (Empirical cumulative distribution function, ECDF). Let

iid
X1 , . . . , Xn ∼ F . Then the ECDF Fn is defined as
n
1X
Fn (x) = 1(Xi ≤ x).
n i=1

1
Pn
(Similarly, we can define Fn (x, y) = n i=1 1(Xi ≤ x, Yi ≤ y) as the ECDF
of a bivariate distribution.)

The formula is very intuitive. Recall that 1(Xi ≤P x) = 1 for exactly those Xi
with Xi ≤ x. (Otherwise 1(Xi ≤ x) = 0). Hence, ni=1 1(Xi ≤ x) is counting
how many observations Xi are less than x. Dividing by n gives us the proportion
of samples that are less than x. Intuitively, this proportion should be a good
approximation of the probability that X ≤ x.
The ECDF Fn is a function very similar to the CDF of a discrete random
variable. In fact, it is the CDF of a discrete random variable X̃ that puts
probability mass 1/n on each data point. As a result, the ECDF is right continuous
and jumps by 1/n at every observation. You can see this in an example with
n = 5 in Figure 3.6. The crosses represent the location of our five observations
X1 , . . . , X5 . The ECDF starts out at 0 until we encounter the first observation
(coming from the left). At each of the observations the ECDF jumps up by
1/5 = 0.2 until it reaches 1, from where on it remains constant.
We can show that the ECDF Fn (x) is a consistent estimator for F (x). In fact
it is consistent uniformly, i.e. for all values of x at the same time.
Chapter 3 Descriptive statistics and exploratory data analysis 39

n = 10 n = 100 n = 1000
1.00 ● 1.00 ●
●
●●
● 1.00 ●
●●
●
●
●
●
●●
● ●
●
●●
●
●●
● ●
●
●
●
●
●
● ●
● ●●
●
●
●
● ●
●
●●
● ●
●
●
●
●
●
●
●● ●
●
●
●
● ●
●
●
●●
● ●
● ●
●●
●
●●
● ●●
●
●
●●
● ●
0.75 0.75 ●
●
●●
0.75 ●●
●
●
●
●
●
●
●
●
●
● ●●● ●
●
●●
●
●● ●●
●
●
●●
● ●●
●
●
●
●
●
● ●
●●
● ●
● ●●
●
●●
● ●● ●●
●
●
●
●
●
●● ●
Fn(x)

Fn(x)

Fn(x)
●
●●
●●● ●●
●
●●
● ●
0.50 ● 0.50 ●
●
●●
● 0.50 ●
●
●
●
●
●
●●
●
●●
●● ●●
●
●
●
●
● ●
●
●●
●
● ● ●
●
●
●●● ●
●
●
●
●
●
●
● ●
●●
● ●
●●
● ●
●
●
●
●
●
●
● ●
● ●
●
●
● ●
●●
●● ●
●●
0.25 0.25 ●● 0.25 ●
●
●●
●●● ●
●
●
●
●
● ●
● ●
●
●
●●
●
●●● ●
●●
●●
●●
●
●
●
● ●
●
●●
●
●●
● ●
●
●
●
●
●
● ●
● ●●
●
●
●
● ●
●
●● ●
●●
●
●
●●
● ●
●
●
●
●
●●
● ●●
●
0.00 0.00 0.00 ●
●

0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8

x x x

Figure 3.7: Uniform convergence of the ECDF Fn to the CDF F of a Uniform(0, 1)

distribution.

iid
Theorem 3.12 (Glivenko-Cantelli theorem). Let X1 , . . . , Xn ∼ F . Then

max |Fn (x) − F (x)| →p 0.

x

In particular, Fn (x) →p F (x) for all x ∈ R.

This convergence is visualized in Figure 3.7. We simulate data sets from the
Uniform(0, 1) distribution of increasing size n. The true CDF is F (x) = x which
is shown as as the straight diagonal line. For n = 10 we are fairly close in some
regions , but far away in others (around x = 0.7). As n increases we get closer
and closer to the true CDF. And we do so in away that is uniform in the sense
that there’s no region where our approximation remains bad. For n = 1000 the
true CDF F and the ECDF Fn are hardly distinguishable.

3.4 The histogram

As mentioned in the last chapter, the CDF F is mathematically important, but
it’s hard to interpret the graphs. The same is true for the ECDF. Instead,
we would much rather have an estimate of the PDF/PMF, which are easier to
interpret. The histogram does just that:

(i) For some x0 < xK , we divide the interval (x0 , xK ] into K bins Bk =
(xk−1 , xk ] of equal size δ = x1 − x0 = x2 − x1 = . . .

(ii) We compute Nk = ni=1 1(Xi ∈ Bk ), the number of observations that fall

P
into each bin Bk , k = 1, . . . , K.

(iii) For x ∈ Bk , the histogram is defined as

Nk
hn (x) = .
n×δ
Chapter 3 Descriptive statistics and exploratory data analysis 40

This process is visualized in Figure 3.8. The data is shown as crosses in the
top panel. Then the interval (0, 3.5] is divided into 7 bins of equal size (mid
panel). Then we count the number of observations per bin to compute the relative
frequencies in step 3 (bottom panel).
The histogram is extremely powerful. With a single glance we get a good feeling
for the shape of the entire distribution. Two important characteristics to look out
for are skew and modality. Skew is related to symmetry: a distribution is called
right-skewed if the histogram/density is leaning to the left and left-skewed if it is
leaning to the right. (I know that’s confusing, but it is what it is.) Modality tells
us about potential clusters (showing up as bumps in the graph). Each bump is
called a mode; for example, a density with two distinct bumps is called bimodal.
You can see some exemplary graphs in Figure 3.9.

Example 3.13. Figure 3.10 shows a histogram for the metallicity of globular
clusters in the Milky way (relative to the sun) with K = 10 bins. We see that
the distribution is slightly right skewed and potentially bimodal (with one bump
around -1.5 and a possible second one at -0.9). But why did I choose 10 bins? In
Figure 3.11 we see histograms for the same data, but this time with K = 2 (left)
and K = 100 (right). There’s not much we can learn from two bins, the graph is
hiding most of the information in two huge blocks. On the other hand, the graph
with 100 bins is extremely erratic and we wouldn’t expect the true density to have
a shap with that many peaks and troughs.

This example illustrates that the number of bins is crucial for getting meaningful
information out of a histogram. Choosing this number is more or less guesswork
unfortunately. A good rule of thumb is K ≈ 2n1/3 . (There’s some theory behind
this, but every data set is different.) In practice, we usually try a few values and
see what works best.
Let’s conclude with a theorem on the consistency of the histogram. This
(almost) follows from the Glivenko-Cantelli theorem.

iid
Theorem 3.14. Let X1 , . . . , Xn ∼ f , and K → ∞, K/n → 0 as n → ∞.

(i) If f is a PDF, then hn (x) →p f (x) for all x.

(ii) If f is a PMF, then δhn (x) →p f (x) for all x.

3.5 Quantiles and the boxplot

The boxplot is another visual summary of the distribution and often used to
detect outliers (very unusual observations). It is based on quantiles, which we
have to define first.
Chapter 3 Descriptive statistics and exploratory data analysis 41

0.8

0.6

density

0.4

0.2

0.0
0 1 2 3
x
1.00 B1 B2 B3 B4 B5 B6 B7

0.75
density

0.50

0.25

0.00
0 1 2 3
x
1.00 B1 B2 B3 B4 B5 B6 B7

0.75
density

0.50

0.25

0.00
0 1 2 3
x

Figure 3.8: Construction of a histogram: the data is shown the top panel, the
interval (0, 3.5] is divided into bins (mid panel), relative frequencies
drawn in the bottom panel.
Chapter 3 Descriptive statistics and exploratory data analysis 42

right−skewed symmetric left−skewed

0.6
0.4

0.4
0.3 0.4
density

density

density
0.2
0.2 0.2
0.1

0.0 0.0 0.0

0 2 4 6 −2.5 0.0 2.5 0 2 4 6
x x x

unimodal bimodal trimodal

0.25
0.4
0.20 0.15

0.3
0.15
density

density

density
0.10
0.2
0.10

0.05
0.1 0.05

0.0 0.00 0.00

−2.5 0.0 2.5 −3 0 3 6 −5 0 5 10
x x x

Figure 3.9: Common shapes of a distribution. Histograms with the the true
density superimposed as orange line.

0.6
density

0.4

0.2

0.0
−2 −1 0
log metallicity

Figure 3.10: Histogram for the metallicity of globular clusters in the Mikly way
(relative to the sun).
Chapter 3 Descriptive statistics and exploratory data analysis 43

2 bins − to smooth 100 bins − too erratic

2.0
0.20

1.5
0.15
density

density
1.0
0.10

0.05 0.5

0.00 0.0
−4 −2 0 −2.5 −2.0 −1.5 −1.0 −0.5 0.0
log metallicity log metallicity

Figure 3.11: Histograms for the metallicity of globular clusters in the Mikly way
(relative to the sun) with too few and too many bins.

Definition 3.15 (Quantile). Let F be a CDF. The corresponding p-quantile

is defined as

Q(p) = min{x : F (x) ≥ p}.

The definition is a bit weird, so some discussion is in place. First consider the
case where F is continuous. Then Q(p) = F −1 (p) is just the inverse function of
the CDF F . The weirdness only comes in for discrete distributions, where the
CDF is not strictly increasing and, thus, no inverse exists. Conceptually, the
definition above is equivalent to an inverse function though.
Quantiles answer the question: which value of X is not exceed with a probability
of p? For example, if Q(0.01) = −5, then the probability that X is less than -5 is
1%. More generally, the Q(p) divides the real line into two parts: the first part,
X ≤ Q(p) has probability p, and the second part,X > Q(p) has probability 1 − p.
Given data X1 , . . . , Xn , we define the sample p-quantile as:

Qn (p) = Fn−1 (p) = min{x : Fn (x) ≥ p}.

Let’s write this in a more intuitive way. Denote dae as the smallest integer k
with k ≥ a (‘rounding up’). Now Qn (p) is defined such that

• dnpe of the observations are less or equal to Qn (p),

• dn(1 − p)e of the observations are larger or equal to Qn (p),

So a proportion of at most p of the data points is less than Qn (p), and a proportion
of at most 1 − p is larger than Qn (p). To compute this number, do the following:
First sort the data X1 , . . . , Xn in ascending order. This gives an ordered data set
X(1) , . . . , X(n) , where X(k) denotes the kth smallest observation. Then set2

Qn (p) = X(dnpe) .
2
Most software/books make an adjustment if np is an integer.
Chapter 3 Descriptive statistics and exploratory data analysis 44

Some quantiles have special names:

• Q(1/2): median,

• Q(1/4), Q(3/4): lower quartile and upper quartile.

Similar to the mean, the median is another measure of location. It is defined

such that half of the data/population is less than this number, and half of the
data/population is larger than this number. So the median is really the “average
guy”. Similar to the variance, the inter-quartile range IQR = Q(3/4) − Q(1/4)
is a measure of variation. Such quantile-based measures are called robust because
they’re not sensitive to individual observations. Let’s see this in an example:

Example 3.16 (Robustness of median). Suppose we have data on the monthly

net income (in 1000 EUR):

X1 = 3, X2 = 2.6, X3 = 3.6, X4 = 4.6, X5 = 1.7.

We compute X̄n = 3.1 and Qn (1/2) = 3. What happens to the mean and median
if we change X4 = 300? Well the median doesn’t change at all, but the mean
increases by an order of magnitude.

The observation X4 = 300 is called an outlier, because it is so different from

all the other data points. The presence of a few outliers drastically changes the
sample mean and variance, but the median and IQR are only little affected (if
at all). In most countries, the majority of the population has a similar income,
but there’s the infamous ‘top 1%’ that earns muuuuch more than all the others.
In that case, the vast majority of people would earn way less than the average
income, so the sample average is not really representative of an average person.
However, the median income always represents this average person: 50% earn
more, 50% earn less.
The boxplot is a visual tool that i) gives a visual summary of the distribution,
and ii) helps to identify potential outliers. An exemplary boxplot is shown in
Figure 3.12. The line in the middle of the box indicates the median, a measure
of location. The upper and lower boundary of the box are the upper and lower
quartiles; the distance between them tell us something about the variability of
the data. The lines leaving the box are called whiskers. (Their exact definition
differs between implementations.) All data points that exceed the whiskers are
drawn as dots. These points are potential outliers.
In the income example above, the outliers are a feature of our reality, so we
should not ignore them (and not call them outliers). In other cases, outliers
may come from faulty measurements or the guy entering the data having fat
fingers. Such outliers we should filter out before proceeding with the data analysis.
Detecting potential outliers is an important part of EDA. After having detected
one, we must always ask ourselves:

(i) whether it is due to random variation or faulty measurement,

Chapter 3 Descriptive statistics and exploratory data analysis 45

potential outliers

Q(3/4) + 1.5 * IQR

upper quartile

median

lower quartile

Q(1/4) − 1.5 * IQR

potential outliers

Figure 3.12: The components of the boxplot explained.

(ii) how it affects further analyses,

(iii) whether or not to remove it from the data.

Deciding whether to keep or remove a potential outlier is always a judgment call

and should be based on domain knowledge.

3.6 A recipe for EDA

Now that we’ve learned about all these summaries, let’s put them together.
Whenever you start your data analysis, it’s a good idea to follow the following
recipe:

1. High-level summary: how many observations, which variables (units), missing

values.

2. For each variable, compute and interpret summary statistics for location and
scale.

3. For each variable, plot and interpret boxplot and/or histograms. Consider:
skewness, modality, outliers

4. If more than one variable, also consider their dependence:

• pair-wise scatterplots,
• compute correlation,
 Chapter 3 Descriptive statistics and exploratory data analysis 46

• check for potential outliers.

Now let’s walk through this process with some real data.

3.7 Case study: Colors of quasars vs. Galaxies

We shall perform an EDA for data about the colors of quasars and galaxies. The
data was obtained from

http://cas.sdss.org/dr16/en/tools/search/sql.aspx

by the following SQL query:

1 SELECT TOP 1000 p .u , p .g , p .r , s .z , s . class
2 FROM photoobj AS p
3 JOIN specobj AS s
4 ON s . bestobjid = p . objid
5 WHERE p . u BETWEEN 0 AND 19.6
6 AND p . g BETWEEN 0 AND 20
7 AND s . class <> ’ UNKNOWN ’
8 AND s . class <> ’ STAR ’
9 AND s . class <> ’ SKY ’
10 AND s . class <> ’ STAR_LATE ’

Our goal is to get a feeling for what is going on in this data. What follows
will be a very brief version of the process, mainly because I know too little about
astronomy to tell you something interesting. Maybe you see some more interesting
things?

3.7.1 High-level summary

• There are n = 1 000 observations,

• There are 5 variables:

– u: u-band apparent brightness (magnitude relative to sun),
– g: g-band apparent brightness (magnitude relative to sun),
– r: r-band apparent brightness (magnitude relative to sun),
– z: redshift,
– class: galaxy (nG = 863) or quasar (nQ = 137).

• There are no missing values.

To assess the colors, we define the following new variables: ‘u - g’ = u -

g (green-ness) and ‘g - r’ = g - r (red-ness). We can then forget about the
original u, g, r variables.
Chapter 3 Descriptive statistics and exploratory data analysis 47

3.7.2 EDA for individual variables

Variable u - g
Galaxies Quasars
mean std. dev. mean std. dev.
1.44 0.37 0.30 0.28

We see that galaxies in this data set tend do be greener than quasars and also
the variability seems slightly larger for galaxies.

2.0
2.0

1.5 1.5
density

u−g

1.0
1.0

0.5
0.5

0.0

0.0
0.0 0.5 1.0 1.5 2.0 2.5 GALAXY QSO
u−g class

class GALAXY QSO

We can clearly identify the two populations in the u -g histogram and boxplot
(galaxies and quasars). Above u - g= 1, we find almost no quasars, below
this value we find almost no galaxies. The histogram for the quasars is roughly
symmetric and unimodal. The histogram for the galaxies is bimodal, indicating
that there may be two sub-populations of galaxies. The boxplots show a few
potential outliers, but the points don’t seem to crazy. It is certainly plausible
that a Quasar has u - g ≈ 1.5. Without further reasons, we should keep them
in the data set.

Variable g - r
Galaxies Quasars
mean std. dev. mean std. dev.
0.70 0.32 0.21 0.22

We see that galaxies are redder than quasars on average and also more variable.
Chapter 3 Descriptive statistics and exploratory data analysis 48

1.5
0

1.0
−2
density

g− r
0.5 −4

−6
0.0
−5.0 −2.5 0.0 GALAXY QSO
g− r class

class GALAXY QSO

In the boxplot there is a clear outlier galaxy with g - r = 6. This looks very
suspicious and it definitely affects our data analysis. Optimally, we would now
look this galaxy up online and check whether it is really that red. And if it is, we
still need to ask ourselves if we want to keep it or focus on more ‘normal’ galaxies.
If we exclude it, we should also recompute all summaries and graphs above. But
let’s move on for now.

Variable z
Galaxies Quasars
mean std. dev. mean std. dev.
0.08 0.05 1.30 0.69

Unsurpringly, Quasars tend to be further away from us than galaxies. The

variablity in redshift is also much larger. That makes sense.

4
4

3
3
redshift
density

2
2

1 1

0 0
0 1 2 3 4 GALAXY QSO
redshift class

class GALAXY QSO

Chapter 3 Descriptive statistics and exploratory data analysis 49

We see that more clearly in the histogram and boxplot. While all galaxies
are fairly close (z < 0.2), the quasars are spread out wider with a center around
1.5 and one quasar with redshift of more than 4. This quasar also shows up as
potential outlier in the boxplot, but z ≈ 4 is certainly a plausible value for a
quasar.

3.7.3 Removing outliers

The outlier for g - r is weird. Also the quasar with z > 4 seems very different
from the rest of the objects under study. These outliers would distort the
correlation heavily because it is not robust. Hence, I will remove both observations
from the data set in what follows.

3.7.4 Joint behavior

The following graph contains scatterplots for all possible pairs of variables. At
the top you can read the correlation (R(C) =correlation in class C).

R(G) = 0.86, R(Q) = 0.37 R(G) = 0, R(Q) = −0.19 R(G) = 0.25, R(Q) = −0.42
● ● ●

● ●

● ●
●
● ● ● ● ●
●
● ●
●
●● ● ● ● ● ● ●
● ●● ● ● ● ●● ● ●
●
●● ● ●
● ● ●
● ●● ●● ● ● ●
●● ●● ●
● ● ● ● ● ● ●
●●● ● ● ●● ●
● ● ● ●●
1.0 ● ●
●
●
●●●●
●● ● ●●
●●
●●● ●●●● ●●●
●
●
● ●●●●●
●●
●● ●●
●●●
●
● ●
● ● ●●
●
●

●
●
●● ● ●
●
●
● ●
● ●●
●
●
●●
● ●● ●
● ●
● ●●
●
●
●●●●
●●●●●
●
●●●●
●
●
●
●
● ●
●
●●
●●●●
●● ●● ●●
●
●
●●● ●●
●●
●
●
●●●●
● ●
●●
●
2 ●
●
● ●
● ● ●
● ● ●
●
●
2 ●
●
●●
● ●●
●●
●
●
●

● ● ●●
● ● ● ● ●●
● ● ●
● ● ●●● ● ● ●● ●●
● ●● ●
● ● ● ●● ● ●●
●
●●●
●●●●●●
● ●● ●
●● ●● ● ● ● ●●● ●● ● ● ●
●●
●
●●
●
● ● ● ●● ● ●●●
● ●●
● ●
●●
●
●●
●
●
● ●●
●
●●
● ●
● ● ● ● ●●●● ● ●●● ● ●
●● ●
● ● ●● ●● ● ● ● ● ●● ●
● ● ●●● ●● ●●● ●● ●
● ● ●● ●●● ● ●
● ●●●● ● ●● ●● ● ●
● ● ●● ●
●● ●●●●● ● ●
● ● ●● ● ● ● ●
●● ●● ●●●
●
●
●● ●●●● ● ●
● ● ●● ● ●● ●● ● ● ●●●
● ●●●●● ●
● ● ●●
●●●
● ●●●●●●
●
●
●
● ● ● ●●
● ●●●● ● ● ●
● ● ●●●●● ●● ● ● ● ● ●
redshift

redshift

● ●● ●●●● ● ● ●
● ● ●● ● ● ●
●● ●●
●● ●
●
●
● ●● ●
● ●●● ●● ● ● ● ● ● ●● ● ●
● ●●● ● ●● ●● ●
●● ● ●● ● ● ●●
● ● ●● ●● ● ●● ● ●● ●
●●●●
g− r

● ●● ●●●● ●● ● ●
●● ●●●●
●
●
●●●● ●● ● ● ● ● ●
● ● ● ●●●● ● ●●●●●●●●●● ●● ● ● ●
●
● ●
● ●●●●●
●● ●● ●● ●●●●● ●
●●
●
●
●●● ● ● ● ● ●
● ● ●●●● ● ●
●●●●
●
● ●●● ●● ●
● ●● ● ●●● ● ●● ●●● ● ● ● ● ● ● ●
● ●● ●● ●●
● ●
●● ●●●●
●
●●●
● ●●
●
● ●
● ● ● ● ● ●
● ● ●
●
●
●
●●●●●●● ● ● ● ● ● ● ●
●● ●
●● ●
●●
●● ●●● ●●● ●● ●● ● ●
●
● ● ● ● ●
●●●●●●●●
0.5 ●
●
●

● ● ●●
●
●●●●●●
● ●
●●●●
● ● ●
●

●
● ●●
●●●
●● ●●●●
●
●

●
●●
●●●●
●●
●●
●
●● ●
● ●
● ●●●
●●
●●
●●
●
● ● ●
●
● ●●
●
●

●
● ●
●
●
●
●
●
● ●● ●●●●● ●
●●● ●
● ●● ●● ● ●● ●
● ●
●
●
●
● ●
● ●●
● ●
●●● ●
● ● ● ●● ●
●●● ● ●
●
●
●● ● ●●
● ●
●
●
1 ● ●
● ● ●
●
●
●●
1 ●
● ●
● ●
●●

● ● ● ● ● ●
● ●● ● ● ● ● ●
● ● ●
●● ● ●
● ● ●
●●
● ● ● ●● ● ● ● ●●
● ●● ● ●● ●● ●
● ● ● ● ● ● ●● ●
● ●● ●● ●● ● ● ●
● ● ● ●● ● ● ●
●
● ● ●●
● ●
● ●● ● ● ● ● ● ●
●●● ● ● ● ● ● ● ● ●
● ● ●
● ● ● ● ● ● ●
● ● ● ●
● ●
●
● ● ●● ● ● ● ●
● ● ● ● ● ● ● ●
● ●● ●
●
●● ●
● ●●● ● ●● ● ● ● ●
● ● ● ●
● ● ● ●●●●● ● ● ● ●
● ● ●● ● ● ●
● ● ●● ●●● ● ●●
● ● ● ● ● ● ● ● ● ● ●
●● ●● ● ● ●
● ● ● ● ● ● ●
● ●
● ●
0.0 ●
●
●● ● ● ●● ●
● ● ●
●
● ●
● ●

●
●
● ●
●
● ● ●
●
● ●●● ● ● ●●●●●
● ● ● ●●● ●
●●● ●● ●● ●● ● ●
●
●
● ●

● ●●
●● ●
● ●●●● ● ● ●
●●
● ●
●
● ●●● ●
●
● ● ●
●
●●●
● ●●●●
●●●● ●●●●
●●●●● ●●●
●●● ●●●● ●●● ●● ●
●● ●● ●
●● ● ●●● ● ●●
● ●
●● ●
● ●●
●●●●●●●
●●●●●●●●●
● ●●●
●
●●●●
●●●
●●●●
●● ●
●●●●
●●
●●●●●●●●● ●●● ● ●●●
● ●● ●●●
●●● ●● ●
● ●●●●●●●●●
●●● ● ●●●
●
●●
● ●●
● ●● ●● ●●● ● ●● ●●
●● ●● ● ● ●●●●●●●●● ●● ●
●● ●● ●●
●● ●●●●● ●●●● ●
● ●● ●●● ●●●●●●●●● ●● ●
●●
●
● ● ●
● ●●
●●
● ● ● ● ●●●●●●●●● ●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●
●●●
●
●●
●
●
●
●
●●●●●
●● ●●
●
●●●● ●●
●
●●● ● ●● ●●●●
●
●●●●
●●
● ●
● ●●
●
●●●
●● ●●●
●●●● ● ● ● ● ●●
●●●●● ● ● ●●●
●
●●
●● ●●●
●●● ●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●●●
●
●●
●●
●
●
●
●●
● ●
●
●
●●
●●●●
●●●●
●
●●● ●●●
●
● ●●
●●● ●●●
●●●●●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
● ●
●●● ● ●
● ● ● ● ●● ●●●
●
●●●
●●●
●●●
●●●●
●●●
●●
●●●
●●●
●● ● ● ●●●●
●● ●
●●
●
●● ●
● ●●
●●●●●
● ● ●
●
●●●
●●●
● ●
●●●●
●● ●
●
●●
●●
●●
●
●
●●
●●
●●
●
●●
●●●
●
●●
●●
●● ●●
●● ●●
●●
●●●●● ● ●●●
●●●
●●
● ●
●●●●●
● ●
●●
●
●●
●●●
● ●
●●
● ●
●
● ●
●●●● ●
●●●
●●
●
●●
●●
●●●●
● ●
●●
●●●●
●
● ●
●●●
●●
●●●
●
●●●
●
●●●●
●●
●
●●●
●
●
●●
●●
●●
●
●●
●●●
●●●● ● ●
● ●
●● ●●
● ●
●
●● ●●●
●●
●
●●● ●
●●
●●
●
●
●●
●●
●
●●
●●
●●
●●●
●
●
●
●
●
●●●●
●●
●●●
●●●●●
●
●●●●
●●●●●●●● ● ●●
●●●
●●
●
●
●
●●●
●●
●●
● ●
●●●●●
●●
●
●●
●●
●● ●
●
● ●
●●●●●● ● ● ● ●● ● ●●●● ● ●●● ●●●
●●
●
●
●
●● ●●
●●
●
●
●●
●
● ●
●
●●●
●●●
● ●
●●●●●
● ●
●●●●
● ●
● ●
●● ●●
●●
●●●●
●●
●●
● ●●
●● ●●
●●●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●●
● ●
●●●● ●
● ●
●
● ● ●
●● ●● ●●●●
● ●●●
●●●
●
●
●●
●
●●
●●●●●
●●
●●
●●●●
●● ●●
●
● ●●
● ●●●
● ●
●●
●
●● ●●
●● ●
● ●
●
● ● ● ●●
●●
●● ●
●
● ●●●
●●●●●
●●●●●
● ●
● ● ●● ● ●
●● ●● ●●●●
●●●
●●●
●●●
●●●
●●
● ●●●
●
●●
●
●●●
● ●●●
●
●●
●
●●●●●
●●●●
●●
●
●●
●
●● ●●●
● ●● ●●●
●●
●
●●●●
●●●●
●
●● ● ●
●
●
0 ● ● ● ●● ●●
●●
●●●●
●
●●
●
●●
● ●
●● ●
●●●●
●
●
●●
●●● ●
●
●●
● ●●
●●● ● ●
● ●
●●● ●●
● ●
● ●
0 ● ●●●●● ●
● ● ●● ● ●
● ●
●
●●●
●
●●
●●
● ●
●● ●
● ●●●●
●
● ●●●●
●● ●● ●
● ●
● ● ●●●
●
● ●●●
●

0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0
u−g u−g g− r

class ●
GALAXY ●
QSO

We see there is a strong positive dependence between colors of galaxies, but

much less for quasars. The dependencies between redshift and color are weaker
and negative for galaxies. So the further away a galaxy is the less green (or red)
it tends to be. The pairwise scatterplots show some galaxies falling far away from
the bulk. These should also be considered potential outliers.

3.7.5 Wrap up
This was a quick walk through the steps of an EDA. As a statistician, I can
compute numbers and draw graphs. But that’s only useful in combination with
domain knowledge. As an astronomer, you should always try to interpret the
result in the numbers and graphs in context. Always ask yourself if what you
Chapter 3 Descriptive statistics and exploratory data analysis 50

see is in line with your expectation. If something seems implausible, dig deeper.
Zoom into a graph, compute summaries for interesting subsets of the data etc.
At the end of this process you should i) have a feeling for what’s going on in
the data, and ii) trust that the data set you continue with is suitable for future
modeling steps. We’ll learn more about that in the next chapters.
 Parametric statistical models
4
Let’s once again recall the basic problem of statistics.
Having observed some data X1 , . . . , Xn , what can we say about the
mechanism that generated them?
In the last chapter we learned how to get a ‘feeling’ for this mechanism. We can
now try and come up with plausible mechanisms that could have generated the
data. Since we don’t know the mechanism, what we come up with are just models
of reality. A statistical model involves randomness and is hence characterized
by a probability distribution
 (or density). A parametric statistical model is a
family of distributions Fθ : θ ∈ Θ that is characterized by a parameter vector
θ ∈ Θ ⊆ Rp . Once we have a parametric model, the main question is which value
of the parameter θ fits the data best. This will be the topic of the next chapter.
In the current chapter, we shall introduce some essential statistical models.
This includes parametric families for discrete and continuous distributions as
well as multi-dimensional data and prediction problems. At the end of this
chapter, you should have heard about the most common models and know in
which situations they may or may not apply. That’s admittedly a bit boring, but
it’s the last thing we need in preparation for all the exciting things that follow in
the next chapters.

4.1 Discrete distribution families

4.1.1 The Bernoulli distribution
Definition 4.1. We say that X follows a Bernoulli distribution with pa-
rameter p ∈ (0, 1) or X ∼ Bernoulli(p) if the PMF is given as

1 − p for x = 0

f (x) = p for x = 1

0 otherwise,


or alternatively f (x) = px (1 − p)1−x for x ∈ {0, 1}. It holds E[X] = p and

V[X] = p(1 − p)

We have already seen the Bernoulli distribution earlier in the course when
speaking about coin flips. We can use the (arbitrary) encoding ‘0 = heads, 1 =
Chapter 4 Parametric statistical models 52

tails’ to define a random variabl X that represents the outcome of a single coin
flip. We say that X follows a Bernoulli distributionwith parameter p ∈ (0, 1)
or X ∼ Bernoulli(p). The parameter p = P(X = 1) is called success probability.
Note that the interpretation of this parameter depends heavily on your coding of
the categories (1 = heads vs. 1 = tails).
We rarely flip coins in reality, but the distribution is everywhere nevertheless.
Its quite common to put study subjects into two categories:
• yes or no answers,
• dead or alive people,
• radio-quiet and radio-loud galaxies,
• red-sequence and blue-sequence galaxies,
• metal-rich or metal-poor globular clusters
All these categories can be recoded (arbitrarily) to a binary variable that only
takes values 0 or 1. When checking the categorie of a random object, we’re again
faced with the Bernoulli distribution.

4.1.2 Binomial distribution

iid
Definition
Pk 4.2. Suppose Y1 , . . . , Yk ∼ Bernoulli(p). Then we say that X =
i=1 Yi follows a Binomial distribution with parameters n and p, or X ∼
Binomial(n, p). We have
(

n x
x
p (1 − p)n−x for x = 0, . . . , n
f (x) =
0 otherwise,

E[X] = np and Var[X] = np(1 − p).

If we throw a coin 10 times, how many heads do we get? That’s again a toy
problem of course, but we can replace the coin with other variables. Out of 100
people receiving cancer treatment, how many survive? Out of 50 random galaxies
under study, how many are radio-quiet? In all these questions, we take a sum
of Bernoulli variables, so the Binomial distribution arises naturally. It has two
parameters: the number of trials n and the success probability p.

Example 4.3. Suppose that a proportion p of all galaxies are radio-quiet. Pick
50 galaxies at random and let X be the number of radio-quiet galaxies. Then
X ∼ Binomial(50, p).

The PMF is visualized for varying parameter choices in Fig. 4.1.1 Note that
all PMFs are zero for x > n and that they have peak near E[X] = pn.
1
The dashed lines are only added as visual guides. The PMF is only defined where the dots
are.
Chapter 4 Parametric statistical models 53

●
0.3

● ●
0.2 ●
●
●
parameters
● ● n = 10, p = 0.2
f

● n = 10, p = 0.5
●
● ● ●
● n = 20, p = 0.7
●
0.1
●

●
●

● ●
● ●
●

● ●
●
● ● ●
0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15 20
x

Figure 4.1: Probability mass function of the Binomial distribution for varying
parameter choices.

4.1.3 Poisson distribution

Definition 4.4. We say that X follows a Poisson distribution with pa-
rameter λ > 0, written X ∼ Poisson(λ), if

λx
f (x) = e−λ , x = 0, 1, 2, . . . .
x!

One may check E[X] = λ, Var[X] = λ. Graphs of the PMF are shown in Fig. 4.2.
Note that f (x) > 0 for all x ∈ N.
The Poisson distribution can be derived formally as: the distribution of the
number of events occurring in a fixed period (area/volume/. . . ), if they occur at
a fixed rate and independently of the time since the last event. Such situations
arise often in The Poisson distribution often arises when modeling rare events:
• the number of mutations on a strand of DNA per unit length,
• telephone calls arriving in a system,
• number of radioactive decays in a given time interval.
The distribution has a few interesting properties:
• When n is large and p is small, the Binomial(n, p) distribution is well
approximated by the Poisson(λ) distribution with λ = np.
• If X1 ∼ Poisson(λ1 ) and X2 ∼ Poisson(λ2 ) are independent, then X1 +X2 ∼
Poisson(λ1 + λ2 ).
Chapter 4 Parametric statistical models 54

●
0.8

0.6

parameters
● λ = 0.2
f

0.4
● ●
● λ=1
● λ=5

0.2 ● ● ●
●
● ●

●
●
● ●
● ●
● ● ●
● ● ● ●
0.0 ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15
x

Figure 4.2: Probability mass function of the Poisson distribution for varying
parameter choices.

Example 4.5. A distant quasar emits 1064 photons per second in the X-ray band,
but at a earth-orbiting X-ray telescope only ca. 10−3 photons arrive per second.
In a typical observation period of 104 seconds, only around 101 of the n ≈ 1068
photons emitted during the observation period arrive, giving p ≈ 10−67 . The
number of photons arriving can be thought as Binomial(n, p)-distributed, and the
latter is well-approximated by the Poisson(λ) distribution with λ = np ≈ 10.

4.2 Continuous distribution families

4.2.1 Exponential distribution
Definition 4.6. We say that X has an exponential distribution with rate
parameter2 λ > 0, written X ∼ Exp(λ), if
(
λe−λx x ≥ 0
f (x) =
0 otherwise.

It holds E[X] = 1/λ, Var[X] = 1/λ2 . A graph is shown in Fig. 4.4

The exponential distribution is widely applied to model times between random

events. For example,

2
Some authors use a different parametrization with ‘scale’ parameter α = 1/λ.
Chapter 4 Parametric statistical models 55

2.0

1.5

parameters
λ = 0.5
1.0
f

λ=1
λ=2

0.5

0.0
0 1 2 3 4 5
x

Figure 4.3: Probability density function of the Exponential distribution for vary-
ing parameter choices.

• the time it takes before your next telephone call,

• the time between clicks of a geiger counter,

• the lifespan of a bulb.

4.2.2 Gamma distribution

The Gamma distribution generalizes the exponential distribution. First we need
to define the Gamma function:
Z ∞
Γ(α) = y α−1 e−y dy, α > 0.
0

Definition 4.7. X has a Gamma distributionwith parameters shape α > 0

and scale β > 0, written X ∼ Gamma(α, β), if
(
α
1
xα−1 e−x/β x ≥ 0
f (x) = β Γ(α)
0 otherwise.

It hods E[X] = αβ, Var[X] = αβ 2 .

Chapter 4 Parametric statistical models 56

2.0

1.5

parameters
λ = 0.5
1.0
f

λ=1
λ=2

0.5

0.0
0 1 2 3 4 5
x

Figure 4.4: Probability density function of the Exponential distribution for vary-
ing parameter choices.

1.00

0.75

parameters
high shape, high scale

0.50 high shape, low scale

f

low shape, high scale

low shape, low scale

0.25

0.00
0.0 2.5 5.0 7.5 10.0
x

Figure 4.5: Probability density function of the Gamma distribution for varying
parameter choices.
Chapter 4 Parametric statistical models 57

Note that the Gamma(1, β) distribution is a Exp(1/β) distribution. Fig. 4.5

shows the density for varying parameter choices. As you can see, the Gamma
distribution is quite flexible. For small values of the shape parameter, the PDF
is strictly decreasing. For larger values, the PDF shows a bump.
Because of its flexibility, the Gamma distribution is quite popular for modeling
strictly positive, continuous variables. Typical examples are:

• aggregate insurance claims,

• the amount of rainfalls accumulated in a reservoir.

Example 4.8. The gamma distribution has been studied extensively in extra-
galatic astronomy with respect to the distribution of luminosities, where it is
known as the Schechter luminosity function. According to the Schechter function
the number of stars or galaxies within a luminosity bin of fixed width at luminosity
` is proportional to `α exp(−l/L∗ ).

4.2.3 Pareto distribution (power law)

Definition 4.9. The Power law or Pareto distribution with shape α > 0
and truncation point ξ > 0, written as Pareto(α, ξ) is defined through the PDF
( α
ξ
α xα+1 for x ≥ ξ,
f (x) =
0 otherwise.

A graph of the Pareto density is shown in Fig. 4.6. The truncation parameter ξ
determines the smallest value that the random variable X can take. In particular,
P(X < ξ) = 0. From there on, the density strictly decreases with what is called
polynomial decay or a polynomial tail : f (x) ∝ x−(α+1) . The smaller the value
of alpha, the slower is the decay. Contrast this to exponential tails found in the
exponential and Gamma distributions, where (approximately) f (x) ∝ exp(−ax)
which goes to zero much faster. The type of decay determines how probable very
large values of the random variable X are. When the density decays slowly, large
values of X occur at relatively high frequency. In fact we have,
( (
∞, α≤1 ∞, α≤2
E[X] = αξ , Var[X] = αξ 2
α−1
, α > 1. (α−1)2 (α−2)
, α > 2.

So when α is very small, large values of X are so probable that the expectation
(or variance) is infinite. This is a bit of a mathematical oddity. The interpretation
is that large values or so frequent that taking the average of such numbers doesn’t
yield a stable result, no matter how many numbers we average.
Typical application domains are similar to the Gamma distribution. But
now we put more probability mass on very large values of the random variable.
Examples are:
Chapter 4 Parametric statistical models 58

1.5

1.0
parameters
α=1
f

α=2
α=3
0.5

ξ=2
0.0
0.0 2.5 5.0 7.5 10.0
x

Figure 4.6: Probability density function of the Pareto distribution for varying
parameter choices.

• the Angstrom exponent in aerosol optics,

• Pareto’s law of income distribution,

• extreme value theory (stock market crashes, natural disasters),

• populations of cities (Gibrat’s law),

Example 4.10. Imagine taking a random sample of stars, which are just entering
the main sequence. The masses of such stars are called initial masses. The
probability density of their masses is called the initial mass function (IMF). Let
us measure mass m in multiples of 1 solar mass. Salpeter discovered that the
number of stars with mass m appears to decrease as a power law (at least for the
larger stars).

4.2.4 Normal distribution

The next family is more widely known: the normal distribution. Another common
name is Gaussian distribution, because it was discovered by Carl Friedrich Gauss
around the year 1800 as a by-product of his astronomical studies.
Chapter 4 Parametric statistical models 59

0.4

0.3

parameters
µ = 0,σ = 1
0.2
f

µ = 0,σ = 2
µ = 2,σ = 1

0.1

0.0
−6 −3 0 3 6
x

Figure 4.7: Probability density function of the Normal distribution for varying
parameter choices.

Definition 4.11. X has a normal distribution with mean parameter µ ∈ R

and variance parameter σ 2 > 0, denoted X ∼ N (µ, σ 2 ), if

(x − µ)2
 
1
f (x) = √ exp − , x ∈ R.
σ 2π 2σ 2

As you might expect from the parameter names, it holds E[X] = µ, Var[X] = σ 2 .
If µ = 0 and σ = 1, we shall say that X has a standard normal distribution.
The PDF and CDF of the standard normal random variable are conventionally
denoted by φ(z) and Φ(z), respectively.
Fig. 4.7 shows graphs of normal density functions. The role of the parameters
is quite obvious. The densities have bell shape, symmetric around a peak at µ.
So this parameter is used to shift the location of the distribution. The spread of
the distribution is determined by the variance parameter σ 2 , where larger values
spread the probability mass out more.
The normal distribution is considered the most important distribution in statis-
tics. Before the change to Euros, it was even featured prominently on the most
common Deutsche Mark bill. (In Fig. 4.8 you can see during my PhD defence,
explaining what a PDF is using the graph on the bill.) It is called ‘normal’
because so many things we observe appear to approximately follow a normal dis-
tribution. This is also the case in astronomy (example: the near-infrared K-band
distribution of globular cluster magnitudes in the Milky Way Galaxy). There’s
even a mathematical reason for that. We will later see that (most) averages of
random variables are approximately normal. This is known as the central limit
Chapter 4 Parametric statistical models 60

Figure 4.8: The normal distribution was featured prominently on Deutsche Mark
bills.

theorem, the second fundamental ‘law’ in statistics (the first being the law of
large numbers). Many quantities are sums or averages of smaller contributions.
That’s true for both things that we observe and things that we compute (just
look back to the previous chapter). We’ll learn more about that later.
A particularly common application domain for the normal distribution are
measurement errors.

Example 4.12. Consider experiments of measuring the mass m of the (anti-

electron) neutrino. The ith experiment yields a measurement Mi . In many
experiments, Mi is computed as a difference two big and similar quantities, none
perfectly known. Also some experiments report a negative value. A sensible first
approach is to model Mi ∼ N (m, σ 2 ), where σ quantifies the precision.

The normal distribution has many convenient and fascinating properties.

Proposition 4.13.

(i) If X ∼ N (µ, σ 2 ), then Z = (X − µ)/σ ∼ N (0, 1).

(ii) If Z ∼ N (0, 1), then X = µ + σZ ∼ N (µ, σ 2 ).

(iii) If X1 ∼ N (µ1 , σ12 ) and X2 ∼ N (µ2 , σ22 ) are independent, then

X1 + X2 ∼ N (µ1 + µ2 , σ12 + σ22 ).

The first property tells us that, by shifting and scaling a normal random variable,
we can transform it to a standard normal one. This also works the other way
around: by shifting and scaling a standard normal variable, we obtain a normal
variable with arbitrary mean and variance (second property). Finally, if we add
Chapter 4 Parametric statistical models 61

up to independent normal random variables, the result is again normal. A few

more useful facts for computations:
• If X ∼ N (µ, σ 2 ), then it follows from the previous proposition
 
a−µ b−µ
P(a < X < b) = P <Z<
σ σ
   
b−µ a−µ
=Φ −Φ ,
σ σ

so all the probabilities for X can be computed from Φ (the CDF of a

standard normal).
• Because of symmetry of the distribution, φ(x) = φ(−x) and Φ(−x) =
1 − Φ(x).
Oddly, Φ does not have a closed form, meaning that it’s impossible to write it
down (so don’t even try to integrate the density function yourself). In practice,
Φ is therefore always computed using a (highly accurate) numeric approximation.
You’ll never need to worry about that though, these approximation algorithms
are implemented in any reasonable software — even on fancy hand calculators.
Because the normal distribution is so central, it has also been studied most
thoroughly. There are many more properties we could list here, but let’s not
overdo it. As a good rule of thumb: whenever you want to know something about
the normal distribution, Google will have the answer.

4.2.5 Normal mixtures

Mixtures are models that combine several distributions into a new one. Such dis-
tributions arise naturally whenever a data set contains multiple sub-populations.
Mixtures of normal distributions are especially popular. Let’s see this in an
example.
On the right you see the histogram of the u - g
color computed from the data set used in our EDA
last chapter. The data consists of ≈ 87% quasars
and ≈ 13% galaxies. The two sub-populations 90

(quasars and galaxies) are easy to distinguish from

this graph. Now suppose that the u - g color of
2 2 60
count

galaxies is N (µG , σG ) of quasars is N (µQ , σQ )

Then the overall color distribution is a weighted
mix of the two normal distributions. We write this 30
as
2 2
0.87N (µG , σG ) + 0.13N (µQ , σQ ) 0
0.0 0.5 1.0 1.5 2.0 2.5
u−g
One might argue that the quasars could be split
class GALAXY QSO
further into two sub-sub-populations. We could
Chapter 4 Parametric statistical models 62

model that by a mixture of three normal distribu-

tions. Conceptually, that’s no different than a mixture of two, but let’s stick to
the latter for simplicity.
Let’s make these models more formal.

Definition 4.14. For two groups with proportions α, 1 − α, means µ1 , µ2 ,and

variances σ1 , σ2 , Gaussian mixture distribution function is
   
x − µ1 x − µ2
F (x) = αΦ + (1 − α)Φ ,
σ1 σ2

and we write X ∼ αN (µ1 , σ12 ) + (1 − α)N (µ2 , σ22 ).

The corresponding density function is

   
x − µ1 x − µ2
f (x) = αφ + (1 − α)φ
σ1 σ2

The generalization to a mixture of K groups with proportions α1 , . . . , αK is

straightforward.

4.3 Multivariate normal distribution

A multivariate distribution is the joint distribution of a vector X ∈ Rd of random
variables. In the last chapter we only considered joint distributions of two
variables, but you can probably figure out yourself how to adapt the definitions
to d variables. The normal distribution also has a generalization to this case.

Definition 4.15. A random vector X is said to have multivariate normal

(Gaussian) distribution with

• mean vector µ ∈ Rd ,

• covariance matrix Σ ∈ Rd×d ,

written X ∼ N (µ, Σ), if its joint density is

1 1 > Σ−1 (x−µ)
f (x) = e− 2 (x−µ) .
(2π)d/2 |Σ|1/2

You can check that the joint density of the multivariate normal simplifies to
the density of the univariate normal when d = 1. The interpretation of the
parameters is the same. The mean vector µ shifts location, the covariance matrix
determines the spread in every direction. Of course, the covariance matrix also
contains information about the dependence between the components of X.
Chapter 4 Parametric statistical models 63

Theorem 4.16. Suppose X ∼ N (µ, Σ). Then it holds:

(i) E[X] = µ (i.e., E[Xk ] = µk for all k),

(ii) Cov[Xj , Xk ] = Σj,k for all j, k,

(iii) a> X ∼ N (a> µ, a> Σa) for all a ∈ Rd ,

(iv) If Cov[Xj , Xk ] = 0, then Xj and Xk are independent.

The first two properties are unsurprising. The third one generalizes the fact that
the sum of two independent normals is again normal. It states that any linear
combination of components of a multivariate normal vector is again normal. The
statement from earlier is recovered for d = 2, a = (1, 1), and Cov[X1 , X2 ] = 0.
But why does Cov[X1 , X2 ] = 0 mean that the variables are independent? For
the multivariate normal distribution, variables are independent if and only if
they are uncorrelated (fourth property). Note that this is a specific feature of
the normal distribution. For most other distributions, zero correlation does not
imply independence.

4.4 Regression models

A statistical model can consist of more than just a distribution. We shall briefly
discuss such a model to give you a taste of what’s coming in the second part
of the course. A regression model is a model concerning several variables. In
particular, we are interested how one of them, say Y , relates to some others, say
X.
As an example, let’s ask how does brightness (Y ) of a galaxy depend on its
mass (X)? A simple model would be

Y = β0 + β1 X + ,

where β0 , β1 are model parameters and  ∼ N (0, σ 2 ). This model can be written
equivalently as follows: the conditional distribution of Y given X = x is

N (µ(x), σ 2 ), where µ(x) = β0 + β1 x.

More generally, if X ∈ Rd , a linear regression model is formulated as

Y = β0 + β > X + , E[ | X] = 0.

The model parameters β0 ∈ R, β ∈ Rd are considered unknown. A further

generalization are generalized linear models: Let Fθ be a distribution function
with parameter θ ∈ Θ, then

Y | X = x ∼ Fθ , where θ = θ(X) = g(β0 + β > X),

Chapter 4 Parametric statistical models 64

where g : R → Θ is a fixed, known function. Also here, β0 ∈ R, β ∈ Rd are the

unknown model parameters.
Generally speaking, regression models are models for conditional distributions.
They are immensely important in many scientific fields. Common uses cases are:

• accounting for measurement errors,

• augmenting physical laws with randomness/uncertainty,

• prediction.

We will talk about such models in more detail in a few weeks.

 Parameter estimation
5
In the previous chapter we learned about common families of statistical distri-
butions. These families are characterized by one or more parameters. In this
chapter, we learn how to find parameter values that best fit observed data.

5.1 Motivation: distributions in times of Corona

When Corona hit western Europe (March 9–13, 2020), the financial markets
plummeted. The tweet in Figure 5.1 shows data for the Dow Jones Industrial
Average, the most important stock index for the US industry. The observed
quantities are weekly index returns (the percentage change of the index from one
week to another). Judging from recent losses, the crash is just as bad as the 2008
financial crisis.
The tweet in Figure 5.1 alerts us that the Dow Jones made a ‘7.7 standard
deviation move’: the losses that week were 7.7 times as large as the sample
standard deviation. Using the ‘standard deviation scale’ implicitly assumes a
normal distribution for the returns. Under the normal distribution, the probability
of such a move is 1 − Φ(7.7) < 10−14 . Put differently, we expect to see such an
event every 1014 weeks, or 2 · 1012 years. Even worse: in the last 120 years, we
already encountered four similar events. Man, how unlucky are we!
A better explanation than ‘bad luck’ is ‘bad model’. Extreme events like market
crashes aren’t normal — neither in the colloquial nor statistical sense. In fact,
one can prove mathematically that extreme events follow a Pareto distribution.1
To compute a crash probability under the Pareto(ξ, α) model, we need to specify
the parameters. The parameter ξ we can choose (only look at losses larger than
ξ). But we also need to know the shape parameter α. In this chapter, we learn
how to find the parameter that fits the data best.
We shall compute a more realistic probability later. For now, take this story
as a warning: statistical models need to be chosen wisely. That’s why you should
have an idea which model is suitable for which kind of problem.

1
This is the key result of an area of statistics called extreme value theory. The theory says that
events exceeding a large threshold approximately follow a generalized Pareto distribution.
It is well established that the generalized model simplifies to the usual Pareto model for
financial returns.
Chapter 5 Parameter estimation 66

Figure 5.1: Random tweet from March 14, 2020.

5.2 A general framework

5.2.1 Parametric statistical models
We can write the models from the last chapter in a more generic form. The key
commonality is that we specify a (possibly conditional) PDF or PMF fθ with
parameter θ. A parametric statistical model F is a collection of such functions:

F = fθ : θ ∈ Θ

Example 5.1. The width of lines in electromagnetic spectra approximately follows

a N (µ, σ 2 ) distribution. Then fθ is the density of a N (µ, σ 2 ) random variable,
where θ = (µ, σ 2 ) and Θ = R × (0, ∞).

A statistical model is therefore a collection of many possible distributions, each

corresponding to a different value of the parameter θ. Finding the parameter
value that best matches the data is called parameter estimation or model fitting.

5.2.2 Estimators and consistency

In Chapter 3, we already touched upon the concept of an estimator. Let’s recall
some important facts. In everything that follows, we assume that the data
X1 , . . . , Xn are iid random variables.

Definition 5.2 (Estimator). Any quantity that can be expressed as

g(X1 , . . . , Xn ) for some function g is called an estimator.
Chapter 5 Parameter estimation 67

Definition 5.3 (Consistency). Let θ be an unknown quantity that we are

interested in. An estimator θbn is called consistent for θ∗ if ‘θbn converges to
θ∗ in probability:
θbn →p θ∗ .

Example 5.4. Let θ∗ = E[X]. Then the sample average θbn = X̄n is a consistent
estimator.

A statistical model F is called correctly specified, if the (unknown) true density

(or PMF) f ∗ is contained in F, i.e., f ∗ ∈ F. If the model is correctly specified,
it is possible to construct estimators θbn that are consistent for the parameter θ∗ .
That is, we can learn the true distribution from the data if there are sufficiently
many.2 The subscript n indicates that the estimator is different for any sample
size n, but is often dropped for convenience.

5.2.3 Bias, variance, and MSE

Recall that estimators are functions of random variables. As a consequence, an
estimator is itself a random variable. It thus makes sense to speak about the
expectation and variance of an estimator.
The expectation E[θ] b is related to a concept we call bias.

Definition 5.5 (Bias). The bias of an estimator θb is defined as

bias[θ] b − θ∗ .
b = E[θ]

b = θ∗ : on average, the estimator θb is equal to

Optimally, we would like to have E[θ]
the true parameter. In this case, bias[θ]
b = 0 and we call the estimator unbiased.

Example 5.6. Let θ∗ = E[X] and θbn = X̄n . Then

 X n  n  
1 1X n
E[θ] = E[X̄n ] = E
b Xi = E Xi = E[X1 ] = E[X] = θ∗ .
n i=1 n i=1 n

Hence, the sample average X̄n is an unbiased estimator for the true mean E[X].

Although unbiasedness is desirable, it is not necessary for consistency. For

example, one can show that the sample variance is biased:
n−1
E[Sn2 ] = V[X] 6= V[X].
n
2
This seems like dark magic to some people. To reflect that, statisticians put a magician’s hat
on the parameter θ and denote the estimator by θbn . (That’s my interpretation at least.)
Chapter 5 Parameter estimation 68

In fact, there are problems for which it is impossible to construct an estimator

that is both consistent and unbiased (but that topic is too advaced for now).
Normally, we’re satisfied with asymptotically unbiased estimators. That is,
b → θ∗ as n → ∞. Indeed, for the sample variance it
estimators θb for which E[θ]
holds limn→∞ E[Sn ] → V[X].3
2

Unbiasedness is not sufficient for consistency either. The estimator θb is a

random variable, but the limiting value in Definition 5.3 is not. In particular,
the limiting value has zero variance. Hence, we would also like the variance of
the estimator to vanish asymptotically, i.e.,
b → 0,
V[θ] as n → ∞.

Any asymptotically unbiased estimator with vanishing variance is consistent.

Theorem 5.7. Let Y1 , Y2 , . . . be a sequence of random variables. If E[Yn ] → y

for some y ∈ R and V[Yn ] → 0, then Yn →p y.

Example 5.8. We already know that the sample average is unbiased: E[X̄n ] =
E[X] for all n. Furthermore, we can compute V[X̄n ] = n1 V[X] → 0. Hence,
X̄n →p E[X]. (This is one way to prove the law of large numbers.)

Instead of considering bias and variance separately, we can also look at a single
measure for the quality of an estimator.

Definition 5.9 (Mean squared error, MSE). The mean squared error of
an estimator θb is defined as

MSE = E[(θb − θ∗ )2 ].

The squared error (θb − θ∗ )2 is a measure for the accuracy of θ.

b The MSE tells us
4
how accurate we are on average. It turns out that the MSE is just a composition
of bias and variance.

Theorem 5.10. It holds

E[(θb − θ∗ )2 ] = bias[θ]
b 2 + V[θ].
b

Proof. Note that

θb − θ∗ = θb − E[θ] b −θ∗ = (θb − E[θ])

b + E[θ] b − θ∗ ).
b + (E[θ]
| {z }
=0

3
The expectation is not a random variable, so this convergence is in the usual, non-probabilistic
sense.
4
To preserve units, we may take the root of MSE.
Chapter 5 Parameter estimation 69

Then using the binomial theorem,

E[(θb − θ∗ )2 ] = E[ (θb − E[θ]) b − θ) 2 ]

b + (E[θ]
b 2 ] + 2 E[(θb − E[θ])]
= E[(θb − E[θ]) b − θ∗ ) + (E[θ]
b E[θ] b − θ∗ )2
| {z }
=0

= E[(θb − E[θ]) 2 b − θ∗ )2 .
b ] + (E[θ]
| {z } | {z }
=V[θ]
b b2
bias[θ]

5.3 The method of moments

We shall now turn to specific examples of estimators for the parameter of a
statistical model. We start with a simple and intuitive method, the method of
moments (MOM). The kth moment of a random variable X is simply E[X k ].
E[X] is the first moment; E[X 2 ] = V[X] + E[X]2 is the second moment, and so
on.
In the previous chapter, we have seen explicit expressions for the mean and
variance of random variables adhering to models fθ (exponential, Poisson, etc.).
In addition, we know that we can estimate the mean and variance of a random
variable consistently by the sample mean X̄n and sample variance Sn2 . We can
therefore define the estimated parameter θb such that theoretical and estimated
mean and variance match.

Example 5.11. Assume X ∼ Bernoulli(p). We know that E[X] = p. The

MOM estimator is defined as pb = X̄n . Consistency of this estimator follows from
consistency of X̄n .

Example 5.12. Assume X ∼ N (µ, σ 2 ), i.e., θ = (µ, σ 2 ). We know that E[X] =

µ and V[X] = σ 2 . We define the MOM estimator θb = (b b2 ) by
µ, σ

µ
b = X̄n , b2 = Sn2 .
σ

Consistency of this estimator follows from consistency of X̄n and Sn2 .

These examples were a bit boring: the parameters are equal to the mean and
variance. It’s often more involved, though.

Example 5.13. Assume X ∼ Gamma(α, β), i.e., θ = (α, β). We know that
E[X] = αβ and V[X] = αβ 2 . We define the MOM estimator θb = (b
α, β)
b by
solving the system of equations

α
bβb = X̄n , bβb2 = Sn2 .
α
Chapter 5 Parameter estimation 70

You can verify that the solution is

X̄n2 S2
α
b= , βb = n .
Sn2 X̄n

Consistency of θb again follows from consistency of X̄n and Sn2 .

In the examples above, θ was at most two-dimensional. Hence, we used two

moments to fit the parameters to data. In general, if theta is p-dimensional, we
need to match p theoretical
Pn and sample moments to another. The kth sample
1 k
moment is just n i=1 Xi , which convergences to E[X k ] by the law of large
numbers. This ensures consistency of the MOM estimator also in the more
general case.

5.4 Maximum likelihood estimation

The method of moments was especially popular in times where data analysis was
performed with nothing but pencil and paper. It presupposes that the theoretical
moments of a distribution are known and have a simple form. We shall now
discuss a more general method: maximum likelihood estimation.

5.4.1 Motivation
Suppose we have decided on a statistical model F = {fθ : θ ∈ Θ} and want to
construct an estimator for the parameter θ. Consider the following question:

Given observed data X1 , . . . , Xn , which value for the true parameter

θ∗ is most likely?

It seems natural to use the answer to this question as an estimator θ.

b There is a
subtle conceptual issue, however. The question asks for a probabilistic assessment
of a fixed (not random!) parameter θ∗ . But when there’s no randomness, prob-
abilities become trivial. The true parameter always takes the same (unknown)
value, every other value has probability zero.
To fix this, we would need to think of the parameter θ as a random variable.
This requires a different conceptual framework, called Bayesian paradigm. We
will touch on this later in the course. To resolve the paradox in the current
framework, we must ask a different question. Let’s reverse the one above:

Given a parameter value θ, how likely is it that we observe the data

X1 , . . . , X n ?

The data are random variables, so it’s adequate to assess them probabilistically.
We then define an estimator θb as the value θ under which the observed data are
most likely: we maximize the likelihood of the data given the parameter.
Chapter 5 Parameter estimation 71

5.4.2 The likelihood function

We first need to define what we mean by likelihood of the data. Suppose a
random variable X has PDF/PMF f . The value f (x) measures how likely it
is that the random variable X takes the value x. It will be unnecessary to
distinguish between continuous and discrete variables in what follows, so let’s
just say ‘density’ when we mean either PDF or PMF.
When we say ‘likelihood of the data’, we therefore mean ‘joint density of all
observations X1 , . . . , Xn ’. Now assume that the data are iid with Xi ∼ fθ∗ . So
what’s the joint density of the random vector X = (X1 , . . . , Xn )? Take a moment
to think about this.
Because the components of X are independent, the joint density is just the
product of marginal densities (see Theorem 2.47). Hence, we define the likelihood
function as
n
Y
L(θ) = fθ (Xi ).
i=1

L is a function of the model parameter θ and tells us how likely it is to observe

the data X1 , . . . , Xn if the true model were fθ .

5.4.3 The maximum likelihood estimator

The maximum-likelihood estimator (MLE) θb is defined as the value θ∗ that
maximizes the function L. In mathematical notation:
n
Y
θb = arg max L(θ) = arg max fθ (Xi ). (5.1)
θ∈Θ θ∈Θ
i=1

The method is extremely general. Everything we need to compute the estimator

is knowledge of the density function. Another formulation of the MLE will be
more convenient. Define the log-likelihood `(θ) = ln L(θ) and the MLE as
n
X
θb = arg max `(θ∗ ) = arg max ln fθ (Xi ) (5.2)
θ∈Θ θ∈Θ
i=1

Convince yourself that the two defintions (5.1) and (5.2) are in fact equivalent
(the logarithm is a strictly increasing function).
So what’s the advantage of taking the logarithm? Recall that, to find the
maximum of a function, we equate the first derivative of the function to zero.
The derivative of a product of n terms is unwieldy (think applying the product
rule dozens of times). The derivative of a sum is just the sum of derivatives. This
simplifies computations a lot. If the maximization problem is solved numerically,
a sum also tends to be more stable than a product, but that’s a topic for another
course.
Chapter 5 Parameter estimation 72

Remark 5.1. Suppose we are not actually interest in the parameter θ∗ , but some
transformation τ ∗ = g(θ∗ ) of it. If θb is the MLE for θ∗ , then g(θ)
b is the MLE for
∗
τ . This property is called equivariance of the MLE.

5.4.4 Computing the MLE

In many cases, the MLE can be computed theoretically. The strategy is always
the same:

Step 1. Compute the log-likelihood function `(θ).

Step 2. Compute the first derivative with respect to all components of θ (θ may
be multidimensional).

Step 3. Equate the derivatives to zero. For k-dimensional θ, we get the system
of equations

∂`(θ) ∂`(θ)
= 0, ..., = 0. (5.3)
∂θ1 ∂θk

Step 4. Define the MLE θb as the value of θ that solves (5.3).

iid
Example 5.14. Suppose X1 , . . . , Xn ∼ Bernoulli(p). Then θ = p, and fθ (x) =
px (1 − p)1−x . We have
n
X
`(p) = ln fθ (Xi )
i=1
n
X  
Xi 1−Xi
= ln p (1 − p)
i=1
n
X n
X
= Xi ln p + (1 − Xi ) ln(1 − p).
i=1 i=1

Taking the derivative with respect to the parameter p yields

n n
d`(p) 1X 1 X !
= Xi − (1 − Xi ) = 0.
dp p i=1 1 − p i=1
Chapter 5 Parameter estimation 73

To solve the above equation, multiply both sides with p(1 − p), which gives
n
X n
X
(1 − p) Xi − p (1 − Xi ) = 0
i=1 i=1
n
X n
X n
X n
X
⇔ Xi − p Xi − p 1+p Xi = 0
i=1 i=1 i=1 i=1
Xn Xn
⇔ Xi − p 1=0
i=1 i=1
n
X
⇔ Xi − pn = 0
i=1
n
1X
⇔ p= Xi = X̄n .
n i=1

Hence the MLE and MOM estimator coincide: θb = pb = X̄n .

Example 5.15. Consider the regression model

Y = β0 + β > X + ,

with  ∼ N (0, σ 2 ) is independent of X. Recall that this is equivalent to saying

Y | X = x = N (β0 + β > x, σ 2 ). Let’s assume for simplicity that the variance σ 2
is known. The statistical model for Y is then F = {fθ : θ ∈ Θ}, where fθ is the
density of a N (β0 + β > X, σ 2 ) random variable and θ = (β0 , β). Note that
n
X n
X
2`(β0 , β) = 2 ln fθ (Yi ) = −n ln(2πσ ) − (Yi − β0 − β > Xi )2 /σ 2 .
2

i=1 i=1

Maximizing this expression is equivalent to minimizing

n
X
(Yi − β0 − β > Xi )2 .
i=1

That’s why the MLE under a Gaussian likelihood is also referred to as (ordinary)
least-squares estimator or OLS for short. The OLS estimator can be computed
theoretically, but let’s reserve that for another time.

When the statistical model is too complicated, it may be hard to derive the
MLE theoretically. If that’s the case (or you just feel lazy), the MLE can
be computed using numerical optimization algorithms (e.g., scipy.optimize).
Theoretical expressions are much faster to compute, however, so they are still
useful in practice (and for exam problems).
Let’s get back to our Corona crash example from the beginning. We can also
compute the MLE for the Pareto distribution.
Chapter 5 Parameter estimation 74

iid
Example 5.16. Suppose X1 , . . . , Xn ∼ Pareto(ξ, α) and ξ is known. Then θ = α
and
αξ α
fα (x) = α+1 , for x > ξ.
x
The log-likelihood is
n
X
`(α) = n ln(α) + nα ln(ξ) − (α + 1) ln(Xi ).
i=1

Taking the derivative we get

n
∂`(α) n X !
= + n ln(ξ) − ln(Xi ) = 0.
∂α α i=1
Pn
Solving for α gives the MLE α
b = n/ i=1 ln(Xi /ξ).

Now we can fit the parameter and compute a probability for a crash as extreme
as last week. I downloaded data for Dow Jones returns for the last 35 years from
yahoo finance5 . Let’s set ξ = 0.05: every week with a loss larger than 5% is
considered extreme. By this definition, 39 of the weeks (≈ 2.1% of the data) were
larger than than ξ and the MLE gives α b ≈ 2.8. In the tweet, the weekly loss was
a whopping 17%. The probability of an event at least as extreme is

0.021P(X > 0.17) = 0.021 1 − Fξ,bα (0.17)) ≈ 0.0007.

Thus, we expect a crash like this every 1/0.0007 ≈ 1400 weeks or every 1/(52 ×
0.0007) ≈ 27 years. While this is still unlikely, the event is orders of magnitude
more probable as you would expect under a normal distribution. It also aligns
well with what we observed over the last 120 years. Seems like we’re not that
unlucky after all.

5.4.5 Consistency
The MLE enjoys several nice theoretical properties. In some sense, it is even the
best possible estimator (think: no other consistent estimator can have a smaller
MSE), but that’s beyond the current scope. For now, we shall content ourselves
with the fact that the MLE is consistent.

iid
Theorem 5.17. Suppose X1 , . . . , Xn ∼ fθ∗ for some fθ∗ ∈ F. Under some
regularity conditions6 , then the MLE θb exists7 and is consistent.

5
https://finance.yahoo.com/
7
The ‘regularity conditions’ are mostly unproblematic. Their main purpose is to exclude
pathological cases; for example, densities fθ that aren’t continuous in θ.
7
The ‘exists’ refers to the fact that the likelihood actually has a maximum.
Chapter 5 Parameter estimation 75

Remark 5.2. We are not going to prove the above theorem. But in case you’re
interested, here’s the idea: Instead of maximizing `(θ), we could just as well
maximize `(θ)/n (scaling doesn’t change the maximal point). By the law of large
numbers, it holds
n
1X
`(θ)/n = ln fθ (Xi ) →p Eθ∗ [ln fθ (Xi )],
n i=1

where the expectation on the right is computed under the true model with parameter
θ∗ and density fθ∗ . That is,
Z
Eθ∗ [ln fθ (Xi )] = ln fθ (x)fθ∗ (x)dx,

which is called the cross-entropy between two densities fθ and fθ∗ . Hence, θb is a
value that maximizes cross-entropy asymptotically (as n → ∞). Finally, one can
show that the cross-entropy is maximized by the true parameter value θ∗ .

5.5 Checking for misspecification and model fit

There is one condition in Theorem 5.17 that you should worry about. The
theorem assumes that the model is correctly specified. If it is not, the estimated
model will not converge to the true one. As we have seen from the Corona crash
example, this can have severe consequences. If the misspecification is less extreme,
the MLE may still be useful, however.
Philosophically speaking, it’s unreasonable to assume that the true distribution
really belongs to a specific model class F. There is an apt quote from one of the
greatest figures in 20th century statistics:

All models are wrong, but some are useful. — George E. Box

In that sense, it’s useless to worry about the model being incorrect. But we
should certainly think about whether a statistical model is useful. If what we
observe doesn’t align with the properties of the model, it’s probably not a useful
one.
So how do we check? There’s one simple tool, called quantile-quantile plot
or just QQ-plot. Suppose θb is the estimated parameter and Fθb the associated
distribution function. The QQ-plot is simply a graph with the theoretical quantiles
Fθb−1 (p) on the x-axis and the empirical quantiles Fn−1 (p) on the y-axis. If the
model fit is good, all points should lie on the main diagonal x = y.
A QQ-plot for the Dow Jones data is shown in Fig. 5.2. The QQ-plot for
the normal distribution is indicated by black dots (parameters were estimated
by MOM). The points deviate a lot from the main diagonal. For large losses,
the sample quantiles are much larger than the theoretical quantiles. Hence, the
normal distribuition is a poor model for large losses. The orange triangles are
Chapter 5 Parameter estimation 76

0.2

●
●
●

Sample quantiles
●
0.1
●
●
●●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
0.0 ●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
−0.1 ●

−0.1 0.0 0.1 0.2

Theoretical quantiles

distribution ● Normal Pareto

Figure 5.2: QQ-plots for the normal and Pareto distributions in the Corona crash.
Data are weekly losses on the Dow Jones Index. Since we cut of the
Pareto at ξ = 0.05, only values above this threshold are shown.

the QQ-pairs for the Pareto distribution. They are generally quite close to the
diagonal, so the Pareto model seems to provide a good fit.

5.6 Chi-square fitting

Let’s be clear: use MLE or MOM whenever possible. But there’s one more
method we need to speak about. Early in the last century, astronomers came up
with a method called χ2 fitting. Although antiquated and sub-optimal in many
ways, it is still applied all over the field. Because it’s used so widely, you should
have at least seen it.
The procedure in a nutshell:

1. Compute a histogram of the data. Let ξ1 , . . . , ξK , be the centers of the bins

and b
h(ξk ) the estimated density.

2. Because in general b h(ξk ) 6= f (ξk ), the histogram makes a random estimation

error k = h(ξk ) − f (ξk ) for which we assume k ∼ N (0, σk2 ). The variances σk2
b
may be different for each bin k and can be estimated.

3. Now find the parameter θ that minimizes the χ2 criterion

K
2
X h(ξ
b k ) − f θ (ξk )
χ2 (θ) = 2
.
k=1
σ
b k
Chapter 5 Parameter estimation 77

There are numerous issues with the above procedure: binning causes bias,
it’s unclear how to choose number and location of bins, error variances σk2 need
to be estimated, . . . There are equally many modifications of the above: taking
logarithms of Xi first, taking logarithms of b
h and fθ when computing the criterion,
how to compute σk , etc. — many things that try to fix issues caused by binning.
In the old days, data were often recorded or shared in the form of binned
counts. Then the only option is to go through these chores. Gladly, this is rare
in modern times and we can just use MLE or MOM.
 Uncertainty quantification
6
By now, we have seen many estimators θb of some parameter θ∗ :

• The sample average X̄n , variance Sn2 , correlation Rn as estimators of the

parameters E[X], V[X], and ρ(X, Y ).

• The empirical distribution function Fn (x) and quantile Fn−1 (p) as estimators
of a CDF F (x) and quantile F −1 (p).

• The histogram b
h(x) as an estimator for the PDF/PMF f (x).

• The MOM and maximum-likelihood method as estimators for the parameter

of a parametric statistical model.

All but MOM and MLE are also called nonparametric estimators, because we do
not need to specify a parametric model for them to work.
So if we have computed an estimator θ, b can we say that θ∗ = θ? b Of course,
not! The estimator θb is a random variable, but θ∗ is not. Every time we compute
an estimator θb we will make an estimation error θb − θ∗ 6= 0. We cannot know the
exact error without knowing θ∗ . If the estimator is consistent, we know that it
converges for infinitely many observations. But on finite samples, there there is
some uncertainty how close we are to the truth.
In this chapter, we learn how to quantify this uncertainty probabilistically.

6.1 The central limit theorem

b ∗ is a random variable, so it has a distribution. The main
The estimation error θ−θ
question is therefore what this distribution is. In special cases, the distribution
can be derived exactly. But more commonly, we need to rely on asymptotic
approximations.
Consistency (related to the law of large numbers) tells us that the distribution
converges to a point mass in the limit. But that’s not helpful to quantify
uncertainty. There is another important limit theorem, the central limit theorem
(CLT). We first need a definition for convergence of distributions.
Chapter 6 Uncertainty quantification 79

Definition 6.1 (Convergence in distribution). Let Yn be a sequence of random

variables and Y be another random variable. Denote the CDF of Yn by Fn and
the one of Y by F . Then we say that Yn converges in distribution to Y or

Yn →d Y,

if for all y ∈ R where F is continuous,

Fn (y) → F (y), as n → ∞.

The restriction to continuity points is necessary to allow for non-continuous

distributions. There is a relationship between convergence in probability and
distribution. If Yn →p Y , then also Yn →d Y . The reverse in not true in general.
But if the limiting random variable Y is constant, i.e., P(Y = c) = 1 for some
c ∈ R, then Yn →d c also implies Yn →p c.
For uncertainty quantification, we are really interested in situations where the
limit is genuinely random. The CLT provides just that.

Theorem 6.2 (Central limit theorem, CLT). Let Y1 , . P. . , Yn be iid with mean
E[Y1 ] = µ and variance V[Y1 ] = σ and define Ȳn = n ni=1 Yi . Then
2 1

√
Ȳn − E[Ȳn ] n(Ȳn − µ)
p = →d N (0, 1),
V[Ȳn ] σ

and we say that the sequence Ȳn is asymptotically normal.

Remark 6.1. The statement of the theorem uses the common short notation
(Yn − µ)/σ →d N (0, 1). The long form is “there is a random variable Y ∼ N (0, 1)
such that (Yn − µ)/σ →d Y .” An alternative way to write it is
√
n(Ȳn − µ) →d N (0, σ 2 ).

Remember when we said that (most) averages behave like a Gaussian random
variable? The CLT is the mathematically precise formulation of this fact. The
interpretation is that, for large enough n, the sample average Ȳn behaves approxi-
mately1 like a N (µ, σ 2 /n) random variable. This also explains why the Gaussian
distribution is found everywhere in nature. It is the natural model when many
independent factors contribute to an outcome.
As n → ∞, the variance V[Ȳn ] = σ 2 /n vanishes. Hence, in a probabilistic
√ Ȳn − µ gets closer to 0 (that’s the law of large numbers).
sense, the difference
The scaling with n allows us to obtain a non-trivial
√ limit. You can think of it
this
√ way: multiplying a random variable √ by n√blows up its variance. The rate
n strikes just the right balance: V[ nȲn ] = ( n)2 V[Ȳn ] = σ 2 ∈ (0, ∞).
1
“Approximately behaves like” refers to probability statements: probability statements con-
cerning Ȳn are approximated by probability statements concerning N (µ, σ 2 /n).
Chapter 6 Uncertainty quantification 80

The central limit theorem is quite remarkable. The only assumptions are that
the sequence is iid with finite variance. It is called central because it plays such
a central role in probability and statistics. The name was first used by George
Pólya2 in 1920 (in German, “Zentraler Grenzwertsatz”), but the idea is older
and many other famous mathematicians contributed, including Laplace, Cauchy,
Bessel, Poisson (all part-time astronomers!).
As a side note, let me mention that there are several generalizations of the
CLT. The multivariate CLT states that an average of random vectors behaves
like a multivariate normal random variable. Furthermore, the variables do not
have to be iid. For example, we can allow their distribution to change with n or
for (weak) dependence between observations.
There is a joke about statisticians taking averages all day and, in a sense, this
is true. Many estimators we have seen so far can be expressed as averages (or
functions of averages). We shall see that even when they don’t, they can often be
approximated by a suitable average. The CLT tells us that all these estimators
behave like a Gaussian when properly scaled. How nice is that?

6.2 Asymptotic normality of estimators

So how is this useful for uncertainty quantification? If θb − θ∗ ≈ N (0, σ 2 /n), we
can compute an (approximate) probability that θb is within some distance of θ∗ .
In particular, for any  > 0,
 √ b  √
n(θ − θ) n
P(|θb − θ| < ) = P <
σ σ
 √ √ b √ 
n n(θ − θ) n
=P − < <
σ σ σ
√   √ 
n n
(CLT) ≈ Φ −Φ − .
σ σ

If the variance σ 2 is known, we can actually compute this number. Often it is

unknown, but can be estimated.
As n → ∞, the probability above approaches 1: the more data we have, the
more certain we are that θb √is close to θ∗ . Note that the standard deviation of θb is
approximately se[θ] b = σ/ n. This term is called standard error and often used
as a measure of uncertainty. As n → ∞, the standard error goes to zero, which
reflects our increase in certainty.
The CLT applies directly to the sample average θb = X̄n . This is an estimator
for the parameter θ∗ = E[X]. Let’s revisit some of the other examples from
the beginning. As always, we assume that the data are iid random variables
iid
X1 , . . . , X n ∼ F .

2
You might have been tortured by his ‘urn’ in high school.
Chapter 6 Uncertainty quantification 81

Example 6.3. The empirical distribution function is defined as

n
1X
Fbn (x) = 1(Xi ≤ x), x ∈ R.
n i=1

Convince yourself that

F (x) 1 − F (x)
E[Fbn (x)] = F (x), V[Fbn (x)] = .
n
(Hint: what’s the distribution of 1(Xi ≤ x)?) By the CLT,
√ b
n(F (x) − F (x))
q n
→d N (0, 1).
F (x) 1 − F (x)


The variance σ 2 = F (x) 1 − F (x) is not known, because it involves the unknown


b2 = Fbn (x) 1 − Fbn (x) .
distribution F . However, we can estimate it by σ

Example 6.4. Suppose for simplicity that F is continuous. The histogram for
x ∈ (xk−1 , xk ] is defined as
n
1 X
hn (x) =
b 1(xk−1 < Xi ≤ xk ).
n(xk − xk−1 ) i=1

Setting pk = F (xk ) − F (xk−1 ), we get

pk pk (1 − pk )
E[b
hn (x)] = , V[b
hn (x)] =
xk − xk−1 n(xk − xk−1 )2

and therefore
 
pk pk (1 − pk )
hn (x) ≈ N
b , .
xk − xk−1 n(xk − xk−1 )2

Note that E[b hn (x)] 6= f (x), so the histogram is biased. (One can check that it
is asymptotically unbiased, however.) If we want to compute probabilities like
P(|bhn (x) − f (x)| < ), we would need to estimated not just the variance of b
hn (x),
but also its bias. That’s beyond the scope of this course, but be aware that biased
estimators complicate things.

6.3 Asymptotic normality of the MLE

Because the MLE is so important, it deserves special treatment. We will some-
times write fθ (x) as f (x; θ) to indicate more clearly that f is a function of θ. Our
main result in this section is the asymptotic normality of the MLE.
Chapter 6 Uncertainty quantification 82

iid
Theorem 6.5. Suppose X1 , . . . , Xn ∼ fθ for some fθ∗ ∈ F. Under some
regularity conditions, the MLE θb satisfies
√
n(θb − θ∗ ) →d N 0, I(θ∗ )−1 ,

where
 2 
∂ ln f (X; θ)
I(θ) = Eθ .
∂θ

The function I determining the variance is called the Fisher information. It is

interpreted as the amount of information one observation Xi carries about the
unknown parameter θ∗ . If I(θ) is large (Xi provides a lot of information), the
variance of the MLE will be small. That makes sense: if the data carry more
information, we can be more certain about the estimate.
There is a lot of beautiful theory around the Fisher information that you don’t
need to worry about. For example, one can prove that for any estimator θb of
θ∗ , V[θ]
b ≥ 1/ nI(θ∗ )). Hence, no estimator can have smaller variance than the
MLE. Put differently: The MLE is the most efficient way to extract information
from the data. One useful fact is the alternative representation3
 2 
∂ ln fθ (X)
I(θ) = −Eθ .
(∂θ)2
iid
Example 6.6. Recall from Example 5.16 that for X1 , . . . , Xn ∼ Pareto(ξ ∗ , α∗ )
and known ξ ∗ ,
αξ α
fα (x) = α+1 , for x > ξ.
x
The log-density is

f (x; α) = ln(α) + α ln(ξ) − (α + 1) ln(x).

Thus,

∂f (x; α) 1 ∂ 2 f (x; α) 1
= + ln(ξ) − ln(x), 2
= − 2.
∂α α (∂α) α

b − α∗ ≈ N 0, (α∗ )2 /n .


Hence, I(α) = 1/α2 and the MLE satisfies α

Remark 6.2. Theorem 6.5 also generalizes to multi-dimensional parameters

θ. In that case, the limit is a multivariate normal distribution and the Fisher
information is a matrix.
Deriving Theorem 6.5 is more involved than the previous cases. Except in
special cases, the MLE cannot be expressed as an average. We shall see that we
3
There’s a proof on wikipedia if you don’t believe it.
Chapter 6 Uncertainty quantification 83

can approximate it by an average, however. This technique is quite powerful and

commonplace in mathematical statistics.
Proof of Theorem 6.5. Let’s simplify notation and write ∂θ = ∂/(∂θ). To find the
maximum of the log-likelihood `(θ), we equate its first derivative to zero. Thus,
the MLE θb solves
n
X
∂θ ln f (Xi ; θ)
b = 0.
i=1

A first order Taylor approximation around θ∗ gives

n
X n
X n
X
∗
0= b ≈
∂θ ln f (Xi ; θ) ∂θ ln f (Xi ; θ ) + ∂θ2 ln f (Xi ; θ∗ )(θb − θ∗ ).
i=1 i=1 i=1

Solving the above for θb − θ∗ yields

Pn ∗ 1
Pn ∗
∗ i=1 ∂θ ln f (X i ; θ ) n i=1 ∂θ ln f (Xi ; θ )
θb − θ ≈ − Pn 2 = .
∗ − n1 ni=1 ∂θ2 ln f (Xi ; θ∗ )
P
i=1 ∂θ ln f (Xi ; θ )

By the law of large numbers, the denominator converges to I(θ∗ ) in probability.

Thus,
1
Pn ∗
∗ n i=1 ∂θ ln f (Xi ; θ )
θ−θ ≈
b .
I(θ∗ )

Now apply the central limit theorem to the the right hand side. (This was the
interesting part of the proof, you can skip the following details if you want.)
Because, by the chain rule

∂θ f (Xi ; θ∗ )
∂θ ln f (Xi ; θ∗ ) = ,
f (Xi ; θ∗ )

it holds
∂θ f (x; θ∗ )
  Z
∗
E ∂θ ln f (Xi ; θ ) = f (x; θ∗ )dx
f (x; θ∗ )
Z
= ∂θ f (x; θ∗ )dx
Z
= ∂θ f (x; θ∗ )dx

= ∂θ 1
= 0.
Chapter 6 Uncertainty quantification 84

Further,

∂θ ln f (Xi ; θ∗ )
 
1
V ∂θ ln f (Xi ; θ∗ )
 
V ∗
= ∗ 2
I(θ ) I(θ )
 
1  ∗ 2

  ∗ 2

= E ∂θ ln f (Xi ; θ ) − E ∂θ ln f (Xi ; θ )
I(θ∗ )2
1
2 
E ∂θ ln f (Xi ; θ∗ )

= ∗ 2
I(θ )
1
= .
I(θ∗ )

Then the result follows from the CLT.

6.4 The delta method

Sometimes we are interested in a transformation g(θ∗ ) of a parameter θ∗ rather
than the parameter itself. If we know that θb is asymptotically normal, what does
b For example, one can show that the sample variance S 2 is
that mean for g(θ)? n
asymptotically normal. To preserve
p units, we would like to look at the sample
standard deviation g(Sn2 ) = Sn2 .
b − g(θ∗ ) is
The answer is simple. If g is continuously differentiable, then g(θ)
also asymptotically normal. This follows from the Taylor approximation
b − g(θ∗ ) ≈ g 0 (θ∗ )(θb − θ∗ ).
g(θ)

The derivative g 0 (θ∗ ) tells us how to adjust the asymptotic variance.

√
Theorem 6.7 (Delta method). Suppose n(θb − θ∗ ) →d N (0, σ 2 ) and that g
is continuously differentiable. Then,
√
b − g(θ∗ ) →d N 0, g 0 (θ∗ )2 σ 2 .



n g(θ)

Example
√ 6.8. Let σ 2 = V[X]. For the sample variance Sn2 , one can show
n(Sn2 − σ 2 ) →d N (0, µ4 − σ 4 ), where µp 4
4 = E[(X − µ) ]. Now consider the
√
sample standard deviation Sn = g(Sn2 ) = Sn2 . It hold’s g 0 (x) = 1/(2 x) and
therefore
√ µ4 − σ 4
 
n(Sn − σ) →d N 0, .
4σ 2

Example 6.9. The delta rule is often useful when computing probabilities from
an estimated model. Recall the Corona crash example following Example 5.16.

We computed the MLE α α) = 0.021 × 1 − Fξ,bα (0.17) .4
b and then a probability p(b
4
Let’s treat 0.021 as a fixed number for simplicity. Strictly speaking it’s also a random variable.
Chapter 6 Uncertainty quantification 85

You can check that Fξ,α (x) = 1 − (ξ/x)α for x > ξ. Hence,

p0 (α) = −0.021 × (ξ/0.17)α ln(ξ/0.17),

and
√
α) − p(α∗ ) →d N 0, p0 (α∗ )2 (α∗ )2 .



n p(b

For easier interpretation, we also computed an inverse probability t(b α) =

1/(52 × p(bα)), which we interpret as the expected number of years between such
events. It holds,

p0 (b
α) 0.021 × (ξ/0.17)α ln(ξ/0.17)
t0 (α) = − = ,
52 × p(b α)2 α )2
52 × p(b

and
√
α) − t(α∗ ) →d N 0, t0 (α∗ )2 (α∗ )2 .



n t(b

6.5 Confidence intervals

Confidence intervals are the most common way to communicate uncertainty. We
want to construct an interval (θbl , θbu ) around around the estimated value θb in a
way that we can be confident that it covers the true parameter θ∗ . Our confidence
is quantified by a probability γ, the confidence level. A γ-confidence interval is
an interval that includes the true parameter with probability at least γ.

Definition 6.10 (Confidence interval). An interval (θbl , θbu ) is called a γ-

confidence interval, if

P θ∗ ∈ (θbl , θbu ) ≥ γ.

There is a subtlety: The parameter θ∗ is a fixed number, it is the interval that

is random. So the probability in Definition 6.10 is a statement about the interval,
not about the true parameter. The graph5 in Fig. 6.1 might help to understand
this better. We repeat the same experiment 14 times:

(i) simulate data X1 , . . . , Xn ,

(ii) compute the MLE θ,

b

(iii) construct a confidence interval (θbl , θbu ).

5
Taken from https://seeing-theory.brown.edu, a beautiful introduction to statistics with
interactive graphics. Check it!
Chapter 6 Uncertainty quantification 86

Figure 6.1: Illustration of confidence intervals. The dashed line is the true pa-
rameter θ∗ , intervals are constructed repeatedly from simulated data.

The dashed line indicates the fixed location of the true parameter θ∗ . The dots
are the estimates θ,b the bars indicate the intervals (θbl , θbu ). The estimates and
intervals are random, so they are different for every of the 14 runs. Some of
the intervals cover the true value θ∗ , some don’t. For γ-confidence intervals, we
expect that the long-run proportion6 of intervals covering θ∗ is at least γ.
So how do we construct such intervals? Suppose that an estimator θb is asymp-
totically normal, that is θb ≈ N (θ∗ , se[θ]
b 2 ). The standard error se[θ]
b may not be
known, but estimated by some se[ b θ]
b (see, e.g., Example 6.3). Recall that Φ is
the cdf of the standard normal function. Set γ = 1 − α (α is called significance
level in a related context, but we’ll get to that). Define zα/2 as the corresponding
(1 − α/2)-quantile

zα/2 = Φ−1 (1 − α/2)

and note that, by symmetry of Φ, −zα/2 = Φ−1 (α/2).

Theorem 6.11. If θb ≈ N (θ∗ , se[θ]

b 2 ) and se[
b θ]b →p se[θ],
b the interval


(θbl , θbu ) = θb − zα/2 se[
b θ],
b θb + zα/2 se[
b θ]b

is an approximate (1 − α)-confidence interval.

6
‘Long-run’ means that we repeat the experiment a large number of times
Chapter 6 Uncertainty quantification 87

Proof. It holds

P θ∗ ∈ (θbl , θbu ) = P θb − zα/2 se[ b < θ∗ < θb + zα/2 se[

b θ] b θ]b
θb − θ∗
 
= P −zα/2 < < zα/2
se[
b θ]b
θb − θ∗
 
b θ] →p se[θ])
(se[ b b ≈ P −zα/2 < < zα/2
se[θ]
b
(asymptotic normality) ≈ Φ(zα/2 ) − Φ(−zα/2 )
(definition of zα/2 ) = (1 − α/2) − α/2
= 1 − α.

The choice of γ (or, equivalently, α) is up to the researcher. The most common

choices are γ = 90% and γ = 95%. The corresponding quantiles are

z5% ≈ 1.64, and z2.5% ≈ 1.96.

Example p6.12. Let θ∗ = E[θ∗ ] and θb = X̄n . The CLT states θb ≈ N (θ∗ , V[X]/n),
so se[θ]
b = V[X]/n, which we can approximate by se[ b = Sn /√n. Hence,
b θ]
 
zα/2 Sn zα/2 Sn
X̄n − √ , X̄n + √
n n

is a (1 − α)-confidence interval for the sample average.

Example 6.13. Let’s reconsider our Corona crash example. We computed the
MLE for the Pareto shape as 2.87 following Example 5.16. In Example 6.9, we
have shown that
√
α) − t(α∗ ) →d N 0, t0 (α∗ )2 (α∗ )2 ,



n t(b

where t(α∗ ) is the expected number of years between events. Recall that the MLE
was αb = 2.87, ξ = 0.05, and n = 39. Substituting these values in the expressions
derived in Example 6.9 yields

|t0 (α∗ )|α∗

se[
b θ]b = √ ≈ 13.8.
n

Therefore, a 90%-confidence interval for t(b

α) is

(27 − 1.64 · 13.8, 27 + 1.64 · 13.8) ≈ (4.23, 49.63).

With 90% confidence, we expect Corona-like crashes to happen every 4 to 50 years.

Remark 6.3. Note that the conditions of Theorem 6.11 do not apply to the
histogram because of its bias. The intervals can still be used to guide intuition,
Chapter 6 Uncertainty quantification 88

but the confidence level will not be correct.

6.6 The bootstrap

The above procedure requires three things:

(i) asymptotic normality of an estimator θ,

b

(ii) an expression for the standard error se[θ],

b

(iii) a consistent estimator se[

b θ]b for the standard error.

The first is rarely an issue. The second and third require hard work. For complex
statistical models or estimators, the standard error may not be known, difficult
to derive, or difficult to estimate.
Luckily, Bradley Efron came up with an ingenious idea in 1987. The bootstrap is
one of the most celebrated and widely used techniques for uncertainty quantifica-
tion. Recall that to quantify uncertainty, we need to approximate the distribution
of the random variable θb − θ.7 Alas, we only observe a single realization of this
variable: the estimate computed from the observed data X1 , . . . , Xn .
Suppose for a moment that we can simulate from the true distribution F .
Consider the following bootstrap algorithm:

Step 1. Simulate B ∈ N independent data sets X1,b , . . . , Xn,b from F , for b =

1, . . . , B.

Step 2. For each b, compute the estimator θbb = g(X1,b , . . . , Xn,b ).

Step 3. Define qbα/2 and qb1−α/2 as the α/2 and (1 − α/2) sample quantiles of the
‘observations’ θb1 , . . . , θbB .

Step 4. Define the confidence interval

(θbl , θbu ) = qbα/2 , qb1−α/2 ).

For large B and n, (θbl , θbu ) is an approximate (1 − α)-confidence interval if

θb is unbiased. Why so? Recall that we are interested in the distribution of
the random variable θ. b The algorithm allows us to ‘observe’ B independent
realizations from this distributions. If this feels like we are “pulling ourselves up
by our bootstraps”8 , you now also know where name comes from.
However, we made the assumption that we can simulate from F . But if we
would know F , we wouldn’t need to estimate anything. In practice, we replace
F by a consistent estimate Fb. There are two ways to do that:

7
This distribution is also called sampling distribution of θ.
b
8
https://en.wiktionary.org/wiki/pull_oneself_up_by_one%27s_bootstraps
Chapter 6 Uncertainty quantification 89

• Parametric bootstrap: We postulate a parametric model F = {fβ : β ∈

B} and compute an estimator βb for its parameter. Then we simulate from
the model Fβb in step 1 of the algorithm.

• Nonparametric bootstrap: We approximate F by the empirical distri-

bution function Fn of the data X1 , . . . , Xn . Simulating X1,b , . . . , Xn,b from
Fn is equivalent to drawing n times from the set X = {X1 , . . . , Xn } with
replacement. That is, for each i = 1, . . . , n, draw a random variable J from
the uniform distribution on {1, . . . , n} and set Xi,b = XJ .

The parametric bootstrap fails if the model F is misspecified. The nonparametric

bootstrap essentially always works. The number B should be chosen large enough
for the quantile estimates qbα/2 , qb1−α/2 to be stable. B = 100 is the absolute
minimum, better use 200 or 500.
So why should we care about asymptotic normality at all? The bootstrap is
computationally demanding. For example, numerically maximizing the likelihood
of a complex model with many parameters can take forever, especially on larger
data sets. If computing the estimator θb once is expensive, computing it a large
number of times is infeasible.
 7
Testing
The history of science is full of hypotheses: Pavlov’s dogs, Mendel’s peas, Newton’s
Radiant Prisms, and so forth. You surely heard about a few famous hypotheses
in astronomy: Copernicus’ hypothesis that planets circulate around the sun,
Hubble’s hypothesis of the expanding universe, the hypothesis that our universe
contains dark energy. An intrinsic property of an hypothesis is that, at the time
it is stated, we don’t know whether it is true or not. What we can do is a reality
check: does the data contradict the hypothesis? The statistical term for this
check is hypothesis testing.
This chapter is a bit different than the others. The statistical framework for
hypothesis testing consists of many different concepts and definitions. It’s easy to
get lost in the details and miss the bigger picture. To avoid that, we shall quickly
walk through all concepts with a concrete example at hand (Section 7.1). We
then discuss a number of issues with hypothesis testing in Section 7.2. These two
sections contain the main takeaways from this chapter. The following sections
introduce the concepts more formally and give additional details.
Something else is unusual: we will see only one concrete examples of a statistical
test. There are thousands, each of them different in its own way. I don’t find
it useful to teach you specific tests that you will immediately forget after this
course is over. What’s important is that you understand the core principles and
the problems associated with them.

7.1 Hypothesis testing: a quick walk through

Null and alternative hypotheses
Marius mentioned in his session that star-forming galaxies tend to emit bluer
light than passive ones. He said there are physical reasons to believe this, but
without having seen any data, we should treat it as an hypothesis. I have no clue
about physics and want to disprove him.
To use statistics (or do science), we need to formalize the hypothesis. Let the
random variable X (A) be the blue light (g − r apparent magnitude) emitted by
a star-forming galaxy and X (P ) the blue light emitted by a passive galaxy. Our
hypothesis is that the difference ∆∗ = E[X (A) ] − E[X (P ) ] is negative: on average,
star-forming galaxies emit bluer light.
The hypothesis we want to test is called the null hypothesis or H0 . Every state
of our world where H0 does not hold, is collected in the alternative hypothesis,
Chapter 7 Testing 91

H1 . In our example,

H0 : ∆∗ < 0, H1 : ∆∗ ≥ 0.

Now we want to check whether the data contradicts the hypothesis. Note that I
want to reject H0 to prove that I’m right. This is how statistical tests are usually
set up, but more on that later.

Test statistics
Suppose that we know which galaxies in the data are star-forming and which are
(A) (A) (P ) (P )
not. We have data X1 , . . . , Xn from active galaxies and data X1 , . . . , Xm
from passive galaxies. The true value of ∆∗ is unknown to us, but can be
estimated by
n m
b = X̄ (A) − X̄ (P ) = 1 1 X (P )
X (A)
∆ n m X − X .
n i=1 i m i=1 i

One can use the CLT to show that

b − ∆∗
∆
Tn =
b p →d N (0, 1),
σbA2 /n + σ
bP2 /m

where σbA2 and σ

bP2 are the sample variances of active and passive galaxies. The
random variable Tbn is called test statistic, because we will use it to decide whether
to reject the null hypothesis. Recall that H0 states that ∆∗ is negative. The
larger (more positive) ∆b (or Tbn ), the more evidence the data provide against H0 .

P-values
Note that Tbn is a random variable (because ∆ b is) from which we see only one
realization — the one computed from the data we observed. Let’s denote this
number by t to make the distinction between the random variable and the
realization more clear.
To decide whether or not to reject H0 , we compute the p-value: the probability
of seeing a value of Tbn at least as large as t, if H0 would be true:

p = PH0 (Tbn ≥ t),

where the subscript H0 indicates that this probability is computed assuming

that H0 is true. If the probability is small, we have found evidence against H0 .
Warning: If p is large, we can only conclude that we found no evidence against
H0 , not that we found evidence for it.
H0 spans many possible values of ∆∗ and its good practice to assume the worst
Chapter 7 Testing 92

case, ∆∗ = 0.1 In that case,

∆
b
Tbn = p 2 →d N (0, 1).
σ bP2 /m
bA /n + σ

and

p = P(∆∗ =0) (Tbn > t) = 1 − Φ(t).

The test above is a Wald test. That is, a test constructed from an asymptotically
normal estimator. Since you already know many estimators that are asymp-
totically normal, you should be able to construct Wald tests for other types of
hypothesis as well.

Significance
Ultimately, we want to make a decision: do we reject H0 or not? We do this by
comparing the p-value against a significance level α. Recall that, a small value
of p constitutes evidence against H0 . Hence, we use the rule

• if p < α: reject H0 ,

• if p ≥ α: don’t reject H0 .2

If p < α, we also say that the result is statistically significant at level α. Similar
to the confidence level γ, choosing the significance level α is up to the researcher.
The most common value is 5%, but this depends on the field and type of research.
But what does it actually mean? The value of α controls the probability of a
false positive: rejecting H0 although it is true. We call this a ‘positive’, because
most tests use H0 as the hypothesis of ‘no effect’. If we want to establish an
effect, we actually want to reject H0 . α = 5% means that, if H0 is true, we expect
it to be rejected in 5% of the cases — just due to chance. This is unacceptable
in many physical experiments, where much smaller levels for α are used.
However, the smaller α, the harder it is to reject H0 (or to ‘detect an effect’).
The probability of detecting a real effect (rejecting H0 when it is false) is also
called power of the test. If a test has little power, we will rarely reject H0 , no
matter if it is true or not. That’s why a large p-value should not be interpreted
as evidence for H0 . Generally, the power of a test increases if we have more data
to base our decision on.

Multiple testing
Let’s assume we found t = −3, such that p ≈ 0.999. Unfortunately, I couldn’t find
evidence against Marius’ hypothesis that star-forming galaxies emit bluer light.
1
Here, worst case means that it is harder to find evidence against ∆∗ < 0 than against ∆∗ < c
for any other c ≤ 0.
2
Again, we never “accept”, we only “not reject”.
Chapter 7 Testing 93

That’s a bit embarrassing. Maybe I can at least prove that I’m not a complete
idiot and the difference is small. Let’s test a new hypothesis H0 : ∆∗ < −0.2.
We find p = 0.04 and conclude that I’m not a complete idiot at significance level
α = 5%.
That would be even more embarrassing, because it would mean that I also have
no clue about statistics. By testing two hypotheses, we increase the probability of
a false positive (reject H0 although it is true). It means that, even if we compare
p against 5%, the level of the two tests combined is larger than 5%. This is a
multiple testing problem and for sake of good science, we need to correct for it.
There are two popular ways to do that:
• The Bonferroni correction compares p against α/m, where m is the number
of tests. This correction guarantees that the false positive rate is at most
α. It is generally conservative and safeguards against the worst case.

• The Benjamini-Hochberg (BH) method is a bit more complicated and does

not control the false positive rate (the proportion of false rejections among
all tests). Instead, it controls the false discovery rate: the proportion of
false rejections among all rejections. This is generally less conservative.

7.2 Issues with hypothesis testing

There are several issues with hypothesis testing3 and we should address them
early.

7.2.1 Overuse
There is a tendency to overuse statistical tests. Very often, estimation and
confidence intervals are better tools. It’s a good idea to ask yourself three
questions:
Q1. Do I have a well-defined and well-motivated hypothesis to test for?

Q2. Do I really want to make a yes-or-no decision?

Q3. Do I really care about error probabilities?

If one of the answers is ‘no’, then testing is probably not the right tool. Let me
give a few common examples where tests are misused:
• Exploratory data analysis: There is absolutely no reason to do formal tests
while exploring a data set. In EDA, we would normally answer Q1 and Q3
above with ‘no’. Visualization, estimation, and confidence intervals tell you
everything you need. Yes-or-no decisions like “should I remove this outlier?”
should be based on the scientific context rather than a statistical test.
3
The American Statistical Association even issued a statement on this: https://amstat.
tandfonline.com/doi/full/10.1080/00031305.2016.1154108#.Vt2XIOaE2MN
Chapter 7 Testing 94

• Assessing estimation uncertainty: There is a reason why we didn’t speak

about tests in the previous chapter. If you want to assess or communicate
how variable estimates are, there’s no need to make a yes-or-no decision
(Q2). Use confidence intervals!

• Model selection: Quite often, we have multiple plausible models F1 , . . . , FM

and want to decide which is best. Let F be the unknown true distribution of
the data. A widespread (mal-)practice is to test the hypothesis H0 : F ∈ Fm
for all m = 1, . . . , M and pick the one with the largest p-value. That’s
not what testing is for (see Q1-Q3). If possible, use visualizations like
histograms and QQ-plots to see if the models fit the data or where they
deviate. We shall see formal methods for model selection later in the course.

7.2.2 Statistical significance vs. scientific relevance

Often, we know upfront that the hypothesis is wrong. For example, if H0 is ‘there
is no difference between populations A and B’ or ‘F ∈ F’, it’s hard to believe
that this is exactly right. But a statistically significant violation of H0 may be
scientifically irrelevant. If we have enough data, we can detect even the smallest
deviations from a hypothesis. But would it be relevant if blue light emissions
in active and passive galaxies differ by 10−100 magnitudes? Effect sizes matter
— and whether they are relevant depends on the scientific context. The p-value
alone has little meaning.

7.2.3 Misinterpretation of p-values

The p-value is not the probability that H0 is true (or H1 is false). This is a
widespread mistake, commonly found in articles on public media.
The p-value is a probability, but a weird one: if H0 was true, how unlikely
is it that the test statistic is as large (or small) as the one we observed. The
important part is ‘if H0 was true’. If H0 is not true (and that’s what we want
to show), then the probability bears no real-world meaning. Because the actual
interpretation is so unwieldy, it is not a good a measure to communicate scientific
results to broader audiences.

7.2.4 The replication crisis

You may have heard of the replication crisis in psychology and medicine. Large
consortia of researchers set out to reproduce results of high profile scientific
‘discoveries’. The shocking outcome was that — even with huge sample sizes and
identical study protocols — many findings could not be reproduced. Conservative
estimates today say that around a third of ‘discoveries’ in these fields are in fact
false positives. Many of them were considered well-established and formed entire
lines of research, with hundreds of publications over several decades. Similar
observations were made in other disciplines.
Chapter 7 Testing 95

But with α = 5% or lower, how can it be that the false positive rate is so large?
There are several likely reasons for this, some good and some bad. Among the
good ones is that multiple testing issues are ignored across a huge proportion
of the scientific literature — mainly due to a lack of awareness and insufficient
statistics education. If this reason counts as ‘good’, you get an idea of what
comes next.
Academic journals are less likely to publish research with no significant results.
Because scientists know this, many don’t even try to publish insignificant ones.
We don’t really know how often someone failed to reject a hypothesis, we only
see the significant results. This is known as the file drawer effect.
It gets worse. If people are aware of it or not, uncorrected multiple testing
actually makes it easier to claim ‘scientific discoveries’. If we use α = 5%, we
can test 100 things where there is no effect, but will make significant ‘discoveries’
in 5 of them — just by chance. Scientists acquire fame and secure their job
through ‘discoveries’, so there is an incentive to make as many as possible. As a
consequence, the incentives suggest to not correct for multiple testing.
Much worse. Remember when I was unhappy with the outcome of my test, so
I tested another hypothesis instead? This is known as HARKing or hypothesizing
after results are known and it’s problematic. If the same data is used to form a
hypothesis and to test it, all inferences (like error probabilities) are corrupted.
Unfortunately, this is practice is widespread. Intentions don’t need to be bad.
For example, data can be expensive or even impossible to collect twice (for
example, when testing hypotheses about a certain time period). What one can do
nevertheless, is to clearly communicate how (and when) a hypothesis was formed
and whether this has implications for inference.
In fact, surveys suggest that many researchers torture their data until they
make a ‘discovery’. This can mean to come up with and test new hypotheses
until p < 0.05 for one of the tests. It can also mean to change the data to push
the p-value beyond the significance boundary by, e.g., excluding or including
outliers, control variables, or sub-groups of the data. These practices are known
as p-hacking and are poison to scientific progress.
In the past five years or so, these issues started to attract attention and things
are changing for the better. Luckily, astronomy and physics are fields where
such practices have been less problematic. But they are not immune to these
issues either. Take the above as a cautionary tale. Small violations of the rules
accumulate and corrupt the scientific endeavor. So it’s better to be aware and
avoid corrupting your own field.

7.3 Null and alternative hypotheses

Now it’s time to formalize our walk-through above. A statistical hypothesis is
a statement about an unknown parameter θ∗ ∈ Θ. We separate the parameter
Chapter 7 Testing 96

space Θ into two parts, Θ = Θ0 ∪ Θ1 . A statistical hypothesis has the form

H0 : θ∗ ∈ Θ0 , H1 : θ∗ ∈ Θ1 .

There are two common types of hypotheses involving a specific value θ0 :4

• one-sided hypotheses:
– H0 : θ∗ < θ0 and H1 : θ∗ ≥ θ0 ,
– H0 : θ∗ ≤ θ0 and H1 : θ∗ > θ0 ,
– H0 : θ∗ > θ0 and H1 : θ∗ ≤ θ0 ,
– H0 : θ∗ ≥ θ0 and H1 : θ∗ < θ0 ,

• two-sided hypotheses: H0 : θ∗ = θ0 and H1 : θ∗ 6= θ0 .

Marius’ hypothesis above was one-sided. Two-sided hypotheses are generally

more common. Let’s see a few examples before we continue.

Example 7.1. Consider the hypothesis that, on average, stars in the Milky Way
and Andromeda galaxies have the same mass. If µM W is the expected mass of a
star in the Milky Way and µA the expected mass of an Andromeda star. Then
θ∗ = µM W − µA and

H0 : µM W − µA = 0, H1 : µM W − µA 6= 0.

Example 7.2. Consider the hypothesis that the metallicity of a quasar is inde-
pendent of its age. If they are independent, the theory predicts that they must
be uncorrelated. Denote by ρ the correlation between metallicity and age. Then
θ∗ = ρ and

H0 : ρ = 0, H1 : ρ 6= 0.

Example 7.3. Consider the hypothesis that the luminosity of stars in the Milky
way (in magnitudes) follows a normal distribution. Denote by Φµ,σ2 the corre-
sponding CDF and let F = {Φµ,σ2 : (µ, σ 2 ) ∈ R × (0, ∞)} be the statistical model.
The parameter of interest is the true CDF F . That is, θ∗ = F 5 and

H0 : F ∈ F, H1 : F ∈
/ F.

This is called goodness-of-fit (GoF) testing and sometimes used as plausibility

check. (I’m not a big fan of this.)

4
The ‘side’ refers to the alternative hypothesis.
5
Note that here the parameter θ∗ is not just a number, but an entire function.
Chapter 7 Testing 97

7.4 Test statistics

A statistical test is an evidence-based decision: given the data, should we reject
the null hypothesis? A test statistic is a number that helps us to make this
decision. It is similar to an estimator in the sense that it is a function of the
data: Tbn = t(X1 , . . . , Xn ) for some function t.

Definition 7.4. A statistical test is a decision rule based on a test statistic

Tbn and set R (called rejection region):

• if Tbn ∈ R, reject the null-hypothesis,

• if Tbn ∈
/ R, do not reject the null-hypothesis.

Most commonly, the rejection region takes the form {Tbn > c} (for one-sided tests)
or {|Tbn | > c} (for two-sided tests), where c ∈ R is a critical value. It is more
common to reformulate the decision rule in terms of p-values, to which we’ll get
in a minute.
Note that we never ‘accept’ the null-hypothesis. If we don’t reject it, this can
have several reasons. The main one is that the test statistic is not informative
enough. That does not mean that we found evidence for H0 , only that we couldn’t
find any against it.

7.5 Test errors

Wasserman gives a nice analogy in his book:

Hypothesis testing is like a legal trial. We assume someone is innocent

unless the evidence strongly suggests that he is guilty. Similarly, we
retain H0 unless there is strong evidence to reject H0 .

There are two types of errors we can make: convicting someone innocent and
letting the perpetrator go unpunished. The same is true for statistical tests:

• type I error: rejecting H0 although it is true,

• type II error: retaining H0 although it is false.

I’ve said it before, let me say it again: statisticians are terrible at naming
things.6 A more intuitive terminology comes from medicine. The outcome of a
medical test is termed positive if it indicates disease (as in ‘HIV-positive’) and
negative if not. The type I error corresponds to a false positive: diagnosing a
disease when the patient is healthy. The type II error corresponds to a false
negative: not detecting the disease although the patient is ill. See the table below
for a summary.
6
Confession time: I need to check Wikipedia every time ‘type I/II’ errors are mentioned.
Chapter 7 Testing 98

retain H0 reject H0
H0 true , type I/false positive
H0 false type II/false negative ,
One−sided test Two−sided test

p p
p
● 2 ● ● 2
Tn − Tn Tn

Figure 7.1: Illustration of p-values: The curve is the density of the test statistic
under the null hypothesis. The p-value is the area under the curve
beyond the observed test statistic Tbn .

7.6 Significance and p-values

Just like estimators, test statistics are random variables. Hence, we can make
probabilistic statements about the outcome of a statistical test. This motivates
the convention for setting up the rejection region: choose the critical value c such
that the probability of a false positive does not exceed some target level α. This
level is called significance level or size of the test.
A false positive is a rejection of H0 although it is true. So assuming that H0 is
true, the probability of a false positive is7

max Pθ (Tbn > c) = Pθ0 (Tbn > c) (one-sided) or

θ∈Θ0

max Pθ (|Tbn | > c) = Pθ0 (|Tbn | > c) (two-sided).

θ∈Θ0

This is also called the false positive rate. The subscript θ in Pθ indicates that
the probability is computed under the assumption θ∗ = θ: “If θ∗ was equal to θ,
what is the probability of rejecting H0 ?” In the two-sided case, Θ0 = {θ0 }; in the
one-sided case, θ0 is the worst-case parameter of Θ0 .
Let’s denote FTbn (t) = Pθ0 (Tbn ≤ t) as the CDF of Tbn under H0 . Then

Pθ0 (Tbn > t) = 1 − FTbn (t), Pθ0 (|Tbn | > t) = 1 − FTbn (t) + FTbn (−t).

The p-value is defined as p = 1 − FTbn (Tbn ) and p = 1 − FTbn (|Tbn |) + FTbn (−|Tbn |),
7
The max is actually a sup, but that ship has sailed.
Chapter 7 Testing 99

respectively. This is illustrated in Fig. 7.1. By construction, the decision rule

• if p < α, reject H0 ,
• if p ≥ α, retain H0 ,
gives a false positive rate of at most α. If p < α we speak of a statistically
significant result at level α.
The most widely used level is α = 5%: if H0 is true, we want to reject in less
than 5% of the cases. Physics is a notable exception, where much smaller values
are common. For example, for the Higgs boson, a statistical test of

H0 : a Higgs boson does not exist at 126 GeV mass

was required to be significant at level8 α = Φ(−5) ≈ 3 × 10−6 : If there was no

Higgs boson, the probability of thinking there is one should be at most 3 × 10−6 .

7.7 Power
The probabilities used above can be generalized as follows.

Definition 7.5 (Power). The power function of a statistical test is defined

as

β(θ) = Pθ (Tbn ∈ R).

The value β(θ) is the probability of rejecting the null hypothesis if θ∗ was equal
to θ.
When the null hypothesis is false (θ∗ ∈ Θ1 ), we want a to reject it with high
probability. That is, we want the power β(θ) to be as large as possible for all
θ ∈ Θ1 .

Example 7.6. Recall that in the example from Section 7.1,

∆b − ∆∗
Tbn = p 2 →d N (0, 1),
σ bP2 /m
bA /n + σ
so
 
c − ∆
β(∆) = P∆ (Tbn > c) ≈ 1 − Φ p 2 .
σA /n + σP2 /m

A few observations:
• As the true difference ∆ grows larger (more positive), the power increases.
That is, we are more likely to reject the hypothesis that ∆∗ < 0. This makes
sense: the more positive the true ∆ is, the easier it is to detect.
8
The probability of a ‘five standard deviation event’ or 5σ-event.
Chapter 7 Testing 100

• The larger the critical value c, the stronger the deviation for H0 has to be
for us to reject and, consequently, the less powerful is the test.

• If c − ∆ < 0 and the sample sizes n, m increase, the test becomes more
powerful. For if we have more data, it becomes easier to detect deviations
from the null.

The observations made in this example hold more generally. Large deviations
from the null are easier to detect, and more data helps.
When the power of a test is low, the probability of rejecting H0 is small, no
matter if it is true or not. That’s why Tbn ∈ / R should not be interpreted as
evidence for H0 .

7.8 Multiple testing

(k)
Suppose we are testing multiple hypotheses H0 , k = 1, . . . , m. The family-wise
error rate (FWER) is defined as the probability of having at least one false
positive. Define Ak as the event that test k results in a false positive. Suppose
that all hypotheses are true and that each test has level α, i.e., P(Ak ) = α.
Clearly,
m
[ 
FWER = P Ak ≥ P(A1 ) = α.
k=1

But equality only holds when A1 = · · · = Am . In the worst case, the events
A1 , . . . , Am are all disjoint. Therefore,
m
[  m
X
FWER = P Ak ≤ P(Ak ) = mα.
k=1 k=1

So to ensure FWER ≤ α∗ , we need to take α = α∗ /m. This is called Bonferroni

correction.
The Bonferroni method is quite conservative: first, it protects us against the
worst-case; second, it protects us against making even a single rejection. If the
number of tests m is very large, the Bonferroni method can be prohibitive. For
example, such a situation appears in studies of the cosmic microwave background
(CMB). To measure the CMB, a satellite is looking in the sky. As the CMB
is only the background, it is masked by other objects in the foreground (like
thermal dust and galaxies). To filter out the foreground objects, a statistical test
is performed with H0 : “there is no foreground object in the way”. This is done
hundreds or thousands of times: once for every tiny part of the sky. In this case,
the Bonferroni method would post such a high hurdle that almost no foreground
could be detected.
In such situations, it is more reasonable to control a less restrictive measure.
The false discovery rate (FDR) is defined as the expected proportion of false
Chapter 7 Testing 101

rejections among all the rejections. A key difference is that we don’t assume that
all hypotheses are true. Let m be the number of tests, R be the total number of
rejections, and R0 be the number of false positives. Then FDR = E[R0 /R]. The
Benjamini-Hochberg (BH) procedure allows to control the FDR. To ensure
that FDR ≤ α:

1. Let P(1) , . . . , P(m) be the p-values of the tests in ascending order.

2. Find the largest k such that P(k) ≤ kα/m. Denote it by k ∗ .

(i) (i)
3. Reject H0 , for all hypotheses where Pi ≤ P(k∗ ) ; retain H0 otherwise.

7.9 Some classes of tests

To conclude this chapter, we shall briefly discuss some common classes of tests.
There’s no need to memorize any of this; it’s enough to have heard the names.
When you really need to do a test in your research (or read about someone else’s)
and have understood the concepts above, you’ll easily be able to find what you
need on the internet.9

• t-test: A test for the mean, i.e., H0 : E[X] = µ0 or H0 : E[X] = E[Y ]. If

the data are exactly Gaussian (so never), the test statistic has a ‘Student
t-distribution’.

• Wald tests: Tests derived from asymptotic normality of an estimator. The

example in Section 7.1 was such a test.

• One- and two-sample tests: A one-sample test is about a property of a

single population, e.g., H0 : E[X] = µ0 . A two-sample test is about the
similarly of two populations, e.g., H0 : E[X] = E[Y ].

• χ2 −tests: all tests where the test statistic follows a χ2 distribution: X ∼

iid
χ2 (k) if X = kj=1 Yj2 with Yk ∼ N (0, 1). Among them are some tests for
P
independence, goodness-of-fit, and many more.

• Likelihood-ratio tests: Tests for nested models, like H0 : X ∼ N (0, 1)

vs H1 : X ∼ N (µ, 1) with µ 6= 0. The test statistic is the difference of
log-likelihoods and typically follows a χ2 distribution.

• Rank-based tests: Construct test statistics from ranking the data. Among
them are tests for equality of distributions and independence.

• Permutation tests: Similar as rank-based, but based on random permuta-

tions of observations instead.

9
See, for example, https://en.wikipedia.org/wiki/Category:Statistical_tests.
 Regression models
8
Broadly speaking, Regression models are statistical models for conditional distri-
butions. The goal is usually to explain some target quantity (Y ) with the help
of others (X). The models can be used to formalize scientific theories and make
predictions. Outside of statistics the term regression has become out of fashion.
But most methods trading under the names machine learning and artificial in-
telligence today are fundamentally regression models. We touched on regression
models briefly in Chapter 4 and Example 5.15 and will expand on them a bit
more in this chapter.

8.1 Terminology
A regression model involves two types of variables:

• a response variable Y ∈ R (also called independent variable in social sciences

or label in machine learning).

• a vector of covariates X ∈ Rp (also called dependent variables, predictors,

or features).

Generally speaking, a regression model is a model for some aspect of the

conditional distribution FY |X . Mostly, interest is in the conditional expectation
E[Y | X]: given the information provided by X, what is our best guess for the
value of Y ? This is also called mean regression. Less frequently, the conditional
distribution FY |X itself (distribution regression) or conditional quantiles FY−1|X
(quantile regression) are considered.
Also here, we distinguish between parametric and nonparametric models. As
always, parametric models are characterized by a finite-dimensional parameter.
This is quite a strong assumption, but facilitates building and interpreting models.
Nonparametric models make almost no assumptions, which makes them harder to
estimate and interpret. We shall therefore focus on parametric regression models
and only briefly discuss nonparametric ones towards the end.
Chapter 8 Regression models 103

8.2 The linear regression model

8.2.1 Model formulation
The linear model is both the simplest and most common type of regression model.
It takes the form

Y = β > X + , (8.1)

where

• β ∈ Rp are model parameters, also called regression coefficients;

•  is a noise or error term and assumed to satisfy E[ | X] = 0.

An equivalent formulation of model (8.1) is E[Y | X] = β > X: we assume that

the conditional expectation E[Y | X] is linear in X.

Remark 8.1. The model assumes a linear relationship between the response and
the predictors. Note that we could take, for example, X3 = X22 , so that non-linear
relationships can be represented as well. We will speak more about this later.

The first element of X is usually set to X1 = 1 and the corresponding coefficient

β1 called intercept. All other elements of X are proper random variables. In this
case, the model can be written equivalent as
p
X
Y = β1 + βk Xk + .
k=2

In Example 5.15, we separated the intercept from the remaining predictors, but
the current formulation will be more convenient.

Example 8.1. Let V and B be the visual and blue band magnitudes of a star. A
linear regression model for the color-magnitude diagram is

V = β1 + β2 × (B-V) + ,

where B-V is the color index. Fig. 8.1 shows an example of a linear regression
model for the color-magnitude diagram of selected stars from the Hipparcos catalog.
Each point represents a star, the straight line is the function β1 + β2 × (B-V).
We see that, on average, the data exhibit an (almost) linear relationship on: the
bluer the star, the brighter it tends to be. Of course, not every star falls on the
line β1 + β2 × (B-V). The vertical distance to the line is the error term . For
some stars it is positive, for some negative; for some larger, for some smaller.

The regression coefficients in Fig. 8.1 were not chosen arbitrarily, but estimated
from the data. That’s our next topic.
Chapter 8 Regression models 104

15

Visual band magnitude

10

0
0 1 2
B−V color index (mag)

Figure 8.1: Linear regression model for the color-magnitude diagram of 2655 stars
from the Hipparcos catalog.

8.2.2 Parameter estimation

In Example 5.15, we have derived a way to estimate the regression coefficients. By
assuming  ∼ N (0, σ 2 ) for some σ 2 > 0, the MLE is equivalent to the minimizer
of the least-squares criterion:
n
X
βb = arg min (Yi − β > Xi )2 .
β
i=1

It turns out that this criterion also works if  is not Gaussian (but then βb is no
longer the MLE). The intuition is that (Yi − β > Xi )2 is a measure for prediction
error. The smaller it is (on average), the better the model is at explaining Yi
from Xi . The true parameter β ∗ is the one that explains Yi best in the sense
that the expected error E[(Y − β > X)2 ] is minimal.
To find an explicit expression for β,
b we equate the derivative of the criterion
to zero:
n
1X
(Yi − βb> Xi )Xi> = 0
n i=1
n n
1X > 1 X b>
⇔ Y i Xi = β Xi Xi>
n i=1 n i=1
 X n −1 X n
1 > 1
⇔ β=
b Xi X i Y i Xi . (8.2)
n i=1 n i=1

Note that Xi Xi> is a p × p matrix, so that the solution involves matrix inversion.
The estimator βb above is also called the ordinary least squares (OLS) estimator.
Chapter 8 Regression models 105

Theorem 8.2. Define β ∗ = arg minβ E[(Y − β > X)2 ]. Then the OLS (8.2) is
consistent for β ∗ : βb →p β ∗ .

Proof. Recall that β ∗ = arg minβ E[(Y − β > X)2 ]. Using the same arguments as
in (8.2), we can show that

β ∗ = E[XX > ]−1 E[Y X].

Then the claim follows from the law of large numbers:

n n
1X 1X
Xi Xi> →p E[XX > ], Yi Xi →p E[Y | X].
n i=1 n i=1

Remark 8.2. Note that Theorem 8.2 does not assume that the model (8.1) is
correctly specified. The OLS converges to the best linear predictor β ∗ (the one
minimizing E[(Y − β > X)2 ]) in any case. However, if (8.1) does not hold, βb> X
does not converge to E[Y | X].
Let use look at the OLS estimator in a bit more detail in the simple case where
X = (1, X2 ). One can show that the formula simplifies to

Sn,Y
βb1 = Ȳn − βb2 X̄2,n , βb2 = Rn ,
Sn,X2

where Rn is the sample correlation of (Y, X2 ) and Sn,Y , Sn,X2 are the sample
standard deviations of Y and X2 , respectively. First note that the correlation
is unit-free, while the sample standard deviations have the same units as the
variables they’re computed from.
Now let’s interpret the coefficients above:
• The intercept βb1 is (literally) the average value of Yi after the average effect
of Xi,2 has been removed. It has the same units as Yi . It’s interpretation
is sometimes meaningful and sometimes not. Essentially it is the expected
value of Yi if Xi,2 = 0.

• The coefficient βb2 is proportional to the correlation between Yi and Xi,2 ,

but adjusted according to the scales of the variables. The unit of βb2 is the
ratio of units of Yi and Xi,2 . It holds βb2 = 0 if and only if Rn = 0.1 If
βb2 6= 0, the interpretation is as follows: if Xi,2 is increased by one unit, then
Yi is expected to increase by βb2 units.

Example 8.3. Consider again the color-magnitude diagram in Fig. 8.1. The
straight line in the graph is in fact the OLS estimate which gives

βb1 = 4.70, βb2 = 4.59.

1
This should remind you of the fact that the correlation measures linear dependence.
Chapter 8 Regression models 106

The coefficient βb1 is in the same units as V , i.e., magnitudes. It tells us that a star
with (B-V) = 0 is expected to have a V -band magnitude of 4.7. The coefficient
βb2 is unit-free, because V and B-V have the same units. It tells us that, for an
increase of 1 mag in B-V, we expect to see an increase of 4.59 mag in V .

8.2.3 Confidence intervals and significant covariates

If you look closely to Fig. 8.1, you will see that there is a gray shaded area around
the regression line. This area is a 95%-confidence region for the line. It is very
narrow, because there are so many data (n) compared to the number of covariates
(p). To construct these intervals, you can use the bootstrap or a closed-form
expression derived from asymptotic normality.

Theorem 8.4. Define β ∗ = arg minβ E[(Y − β > X)2 ]. Then the OLS βb
satisfies for all j = 1, . . . , p,
√ −1/2
nΣ b (βb − β ∗ ) →d N (0, Ip×p ),

where Ip×p is the p × p identity matrix and

n  X n −1
b= 1 1
X
Σ (Yi − βb> Xi )2 × >
Xi Xi .
n i=1 n i=1

The results follows from the multivariate CLT, but we won’t bother with it. Note
b is a p × p matrix, not a single number. The standard error for βbj is
that Σ
computed as the jth diagonal element of Σ b 1/2 /√n.
Theorem 8.4 can be used for a Wald test for the effect of individual covariates:

H0 : βj∗ = 0, H1 : βj∗ 6= 0.

If we reject H0 , we say that covariate j has a significant effect. Beware: if we

test multiple covariates, we need to correct for multiple testing!

8.2.4 Fitted values and residuals

While Yi and Xi are observed, the error term i in the equation

Yi = β > Xi + i

is not. Given an estimate β,

b our model predictions for Yi would be

Ybi = βb> Xi ,

which are called the fitted values. The regression residuals are defined as

i = Yi − Ybi = Yi − βb> Xi .
b
 Chapter 8 Regression models 107

They residuals approximate the error term i and are quite useful to check for
model fit and misspecification.
The residual sum of squares (RSS) is defined as
n
X n
X
RSS = 2
 =
b (Yi − βb> Xi )2 ,
i=1 i=1

and measures the quality of the fit. We already saw it pop up in the asymptotic
variance in Theorem 8.4. If the RSS is small, it means that our model predictions
are close to the observed values. However, the RSS depends crucially on the
variance of . If this variance is large, the RSS will be large, too. (Why?) A
standardized version, called R-squared, is
Pn 2 2
2  Sn,b
R = 1 − Pn i=1 −
b
2
= 1 2
.
(Y
i=1 i − Ȳ n ) Sn,Y

and measures the proportion of Y ’s variability that is explained by the model.

R2 = 0 means that the model has no explanatory power; R2 = 1 means that all
observations can be explained perfectly (b
i = 0 for all i).
2
An issue with R is that we can always improve by adding more covariates.
This is fixed by the adjusted R2
n−1
R̄2 = 1 − (1 − R2 ) ,
n−p−1
which penalizes large p.

8.2.5 Implementation in Python

You will probably never need to implement any of the methodology yourself;
any reasonable statistics software does this for you. In Python, you may use
the statsmodels library. Let me shamelessly copy & paste an excerpt from the
documentation:
1 # Load modules and data
2 In [1]: import numpy as np
3 In [2]: import statsmodels . api as sm
4 In [3]: spector_data = sm . datasets . spector . load ( as_pandas = False )
5 In [4]: spector_data . exog = sm . add_constant ( spector_data . exog , prepend = False )
6
7 # Fit and summarize OLS model
8 In [5]: mod = sm . OLS ( spector_data . endog , spector_data . exog )
9 In [6]: res = mod . fit ()
10 In [7]: print ( res . summary () )
11 OLS Regression Results
12 ==============================================================================
13 Dep . Variable : y R - squared : 0.416
14 Model : OLS Adj . R - squared : 0.353
15 Method : Least Squares F - statistic : 6.646
16 Date : Fri , 21 Feb 2020 Prob (F - statistic ) : 0.00157
17 Time : 13:59:19 Log - Likelihood : -12.978
18 No . Observations : 32 AIC : 33.96
19 Df Residuals : 28 BIC : 39.82
 Chapter 8 Regression models 108

20 Df Model : 3
21 Covariance Type : nonrobust
22 ==============================================================================
23 coef std err t P >| t | [0.025 0.975]
24 ------------------------------------------------------------------------------
25 x1 0.4639 0.162 2.864 0.008 0.132 0.796
26 x2 0.0105 0.019 0.539 0.594 -0.029 0.050
27 x3 0.3786 0.139 2.720 0.011 0.093 0.664
28 const -1.4980 0.524 -2.859 0.008 -2.571 -0.425
29 ==============================================================================
30 Omnibus : 0.176 Durbin - Watson : 2.346
31 Prob ( Omnibus ) : 0.916 Jarque - Bera ( JB ) : 0.167
32 Skew : 0.141 Prob ( JB ) : 0.920
33 Kurtosis : 2.786 Cond . No . 176.
34 ==============================================================================

The first three instructions import the libraries and some data set. The fourth
instruction adds an intercept to the covariates (as the last element, because
prepend=false). The fifth instruction specifies which variables are response and
which are the covariates (Y = spector_data.endog, X = (spector_data.exog,
1)). The sixth instruction, computes the OLS and the seventh instruction prints
a summary of the fitted model.
There’s more information in the output than you will normally need and more
than what’s covered here. So let me just point you to the important bits.

• The (adjusted) R2 is given in lines 13–14 on the right.

• Below you’ll find the log-likelihood (assuming Gaussian errors) and the
model selection criteria AIC and BIC, which we’ll cover later.

• In the table below (lines 23–28) you see everything related to parameter
estimates. x1, x2, x3 are the names of the (random) covariates, const
refers to the intercept that we added to the model.

• The first column (coef) contains the estimated parameters βbk followed by
the standard error.

• The fourth column (P>|t|) is the p-value for H0 : βk∗ = 0 — not corrected
for multiple testing!

• The last two columns are the lower and upper bounds of a 95% confidence
interval. If you want another confidence level, you can compute your own
from the standard errors in the second column.

8.2.6 Heteroscedasticity
In the model formulation (8.1), we made no assumption about the variance of
. To derive the OLS criterion from the normal distribution, we assumed that
this variance is constant. Heteroscedasticity (another terrible name) refers to
situations where the variance depends on X, i.e., V[ | X] is not constant.
Consider for example the model in Fig. 8.1. In some B-V-regions the residuals b
i
tend to be larger (in absolute terms). This is a common phenomenon, especially
Chapter 8 Regression models 109

when bi is a measurement error. For example, measurements of redshift are

generally less accurate for distant objects, which translates to a larger error
variance.
While the OLS is still consistent under heteroscedasticity, it is not very efficient.
Observations with large V[i | Xi ] are overweighted. This can be fixed if we know
the error variance σi2 = V[i | Xi ] for each observation or have an estimator σ bi2 .
Then, we should use the weighted least squares criterion
n
X (Yi − β > Xi )2
βb = arg min .
β
i=1
bi2
σ

8.3 Generalized linear models

8.3.1 The idea
The linear model Eq. (8.1) made no assumptions at all about the distribution
of Y or . It works best when  is at least approximately normal. If Y has a
different distribution, predictions from the model can be nonsensical, though.
Take for example Y ∼ Bernoulli(p) and the model Y = X2 + . For large values
of X2 , the model would predict values of Y that are much larger than 1, which
does not make sense. Similarly, if the distribution is such that Y > 0, the model
might nevertheless predict negative values.
When we have a good guess for the (conditional) distribution of Y , a generalized
version of the linear model is more appropriate. If  ∼ N (0, σ 2 ), yet another
formulation of (8.1) is that the conditional distribution FY |X is normal with
parameter µ = β > X and variance σ 2 . In short,

Y | X ∼ N (β > X, σ 2 ).

From a more abstract point of view, we assume that FY |X belongs to a parametric

family where the parameter µ is a linear function of X.
So why not just replace the normal distribution by another parametric family?
That’s the idea behind the class of generalized linear models (GLMs). There is
a minor caveat though. Take for example Y | X ∼ Bernoulli(p) with p = β > X.
We still have the issue that β > X may fall outside of (0, 1). This is easily fixed
by inserting a link function g : R → P, where P is the set of allowed parameter
values. In the classical linear regression model, we have the identity link, g(x) = x.

8.3.2 Popular GLMs

This idea can be used with all the parametric families we’ve seen so far. Let’s
consider a few examples:

• Logistic regression: Y | X ∼ Bernoulli(p) with p = g(β > X) and the logistic

 Chapter 8 Regression models 110

link
ex
g(x) = .
1 + ex
Instead of regression, we often call this a classification model, because it
models the probability of class-membership, e.g., radio-loud vs radio-quite,
star-forming or not, etc.

• Binomial regression: Y | X ∼ Binomial(p, n) with fixed n, p = g(β > X)

and the logistic link.

• Poisson regression: Y | X ∼ Poisson(λ) with λ = g(β > X) and g(x) = ex .

This is called log-link, which refers to the inverse g −1 (x) = ln(x).

• Exponential regression: Y | X ∼ Exp(λ) with λ = g(β > X) and log-link.

• Gamma regression: We assume Y | X ∼ Gamma(α, ν) with ν not depend-

ing on X and α = g(β > X)/ν. The default link function in most software
is the inverse link g(x) = 1/(β > X), but that doesn’t really enforce the
right parameter values. I recommend to use the log-link here too.

This is only a small sample from an extremely rich class of models. For example,
one can play with other link functions or let both parameters of the Gamma
family vary with X.

8.3.3 Parameter estimation

The MLE can be used to estimate the parameters. Let

F = fg(β> X),η : β ∈ Rp , η ∈ E


be the statistical model for the conditional distribution of Y given X. Here, η

refers to all parameters in the model that are not a function of X. Then the
log-likelihood is
n
X
`n (β, θ) = ln fg(β> Xi ),η (Yi ),
i=1

and all results from Section 5.4 apply.

8.3.4 Implementation in Python

The GLMs mentioned above are just easily implemented as the linear model.
Again from the statsmodels documentation:
1 # Load modules and data
2 In [1]: import statsmodels . api as sm
3 In [2]: data = sm . datasets . scotland . load ( as_pandas = False )
4 In [3]: data . exog = sm . add_constant ( data . exog )
 Chapter 8 Regression models 111

5
6 # Instantiate a gamma family model with the default link function .
7 In [4]: gamma_model = sm . GLM ( data . endog , data . exog , family = sm . families . Gamma () )
8 In [5]: gamma_results = gamma_model . fit ()
9 In [6]: print ( gamma_results . summary () )
10 Generalized Linear Model Regression Results
11 ==============================================================================
12 Dep . Variable : y No . Observations : 32
13 Model : GLM Df Residuals : 24
14 Model Family : Gamma Df Model : 7
15 Link Function : inverse_power Scale : 0.0035843
16 Method : IRLS Log - Likelihood : -83.017
17 Date : Fri , 21 Feb 2020 Deviance : 0.087389
18 Time : 13:59:13 Pearson chi2 : 0.0860
19 No . Iterations : 6
20 Covariance Type : nonrobust
21 ==============================================================================
22 coef std err z P >| z | [0.025 0.975]
23 ------------------------------------------------------------------------------
24 const -0.0178 0.011 -1.548 0.122 -0.040 0.005
25 x1 4.962 e -05 1.62 e -05 3.060 0.002 1.78 e -05 8.14 e -05
26 x2 0.0020 0.001 3.824 0.000 0.001 0.003
27 x3 -7.181 e -05 2.71 e -05 -2.648 0.008 -0.000 -1.87 e -05
28 x4 0.0001 4.06 e -05 2.757 0.006 3.23 e -05 0.000
29 x5 -1.468 e -07 1.24 e -07 -1.187 0.235 -3.89 e -07 9.56 e -08
30 x6 -0.0005 0.000 -2.159 0.031 -0.001 -4.78 e -05
31 x7 -2.427 e -06 7.46 e -07 -3.253 0.001 -3.89 e -06 -9.65 e -07
32 ==============================================================================

There are only minor differences to the example we’ve seen above. The code
above sets up a Gamma regression model (fourth instruction) instead of a linear
model. The model summary contains a bit less information, but the important
parts are still there. The estimate νb of the fixed parameter is given as Scale in
line 15, right column.

8.4 Non-linear models

Consider again the color-magnitude diagram in Figure 8.1. The linear model
looks like an OK approximation, but we also see some systematic deviations. For
very low and large values of the B-V-index, most points fall below the regression
line. For medium values of the index, most points fall above the regression line.
So it seems that the relationship is not exactly linear.
One might argue that nothing’s ever truly linear. It then depends on the
scientific goals whether it’s worthwhile to capture nonlinearities explicitly. In the
following we learn how this can be done within the framework of (generalized)
linear models.
In non-linear models, we assume that the true relationship is characterized by
a function h(X). For example, our model may take the form

Y = h(X) + , or Y | X ∼ Fg(h(X)),η .

We wish to estimate the function without making restrictive assumptions (like

linearity). For simplicity, we shall mostly restrict ourselves to the simpler model
on the left. Everything transfers naturally to the generalized non-linear model
on the right.
Chapter 8 Regression models 112

q=2 q = 10
12.5
12

10.0
Visual band magnitude

Visual band magnitude

8
7.5

5.0
4

2.5

0
0.0
0 1 2 0 1 2
B−V color index (mag) B−V color index (mag)

Figure 8.2: Non-linear regression model for the color-magnitude diagram of 2655
stars from the Hipparcos catalog using polynomial expansions of order
2 (left) and 10 (right).

8.4.1 Polynomial expansions

Earlier in this chapter, I already mentioned a simple way to model non-linear
relationships. For sake of simplicity, we shall assume that there is only a single
covariate X. Now define Z = (1, X, X 2 , . . . , X q ) and use this as the new vector
of covariates. For example, the linear model (8.1) becomes

Y = β > Z +  = β1 + β2 X + · · · + βq+1 X q + . (8.3)

Instead of assuming a linear relationship between Y and X, we now assume

that the relationship is a polynomial of (up to) order q. The intuition behind
this is Taylor’s theorem. If the function h is sufficiently smooth, it can be well
approximated by a polynomial of some order. This also works for expansions in
more than one variable.
Similarly, we may replace an initial covariates X in a GLM with an extended
vector Z. Since we merely replaced the covariate vector by another, the OLS/MLE
methods and related theory still work when assuming a polynomial relationship.
Fig. 8.2 shows the OLS fit of the extended linear model with a quadratic (q = 2)
and tenth-order (q = 10) polynomial (right). Clearly, our models now predict a
non-linear relationship between the V-band magnitude and color index. We see
less systematic deviations, but we also see that the confidence intervals become
wider as we increase q. This is a general rule: the more complex we make the
model, the more uncertainty we have in parameter estimates.
We also see that the q = 10 curve behaves rather erratically, especially in
areas where there are little observations. The reason is that we have too much
flexibility in the model, so that we can trace the points at the right boundary
almost perfectly. Of course, we should not trust the model with q = 10 unless
there are good physical reasons to believe in such erratic behavior. There’s always
Chapter 8 Regression models 113

1.2
1.00

0.9
0.9
0.75

0.6 0.6
s

s
0.50

0.3 0.3
0.25

0.00 0.0 0.0

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
x x x

Figure 8.3: Examples of spline functions of degrees 0 (left), 1 (middle), and 3

(right).

a sweet spot for q, but unfortunately it’s hard to know where it is in advance.
Later, we’ll discuss methods on how to find this sweet spot.

8.4.2 Spline expansions

Polynomial expansions are simple, but have known issues. One is that the
function h must be very smooth for Taylor’s theorem to apply. A second is that
polynomial expansions converge rather slowly to the function they’re supposed
to approximate. A smarter way to approximate h are spline expansions.

Spline functions
Splines are functions that are composed piece-wise from polynomials.

Definition 8.5. Let [a, b] be some interval. A spline of degree q is a

function s defined on a grid of knots a = ξ0 < ξ1 < . . . < ξm = b, such that

s(x) = Pk (x), for x ∈ [ξk−1 , ξk ),

where Pk , k = 1, . . . , m, are polynomials of order q. Additionally, we require

that the (q − 1)th derivative is continuous.

Less formally, a spline is a function that ‘stitches’ together m different polyno-

mials defined on sub-intervals.
Fig. 8.3 shows spline functions of degree q = 0 (left), q = 1 (middle), and q = 3
(right). In all graphs, the interval [0, 1] is split into 10 sub-intervals, indicated
by vertical dashed lines. On each of these intervals, the function is simply a
polynomial of degree q. On the left (q = 0), the function is constant on every
interval; in the middle (q = 1), it is linear on every interval; on the right (q = 3)
it is cubic on every interval. Cubic splines are used most commonly and should
be the default choice.
Chapter 8 Regression models 114

q=1 q=3
1.00

0.75
sk(x)

0.50

0.25

0.00
0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
x

Figure 8.4: Spline basis functions on 5 knots for degrees 0 (left) and 3 (right).

Spline basis
So let’s get back to the statistical problem, estimating the unknown function h
in (8.3). If we assume that h is a spline on m knots, we now need to estimate
10 polynomials of degree q, giving m × q coefficients. That doesn’t sound very
convenient. But hold on: we also have this little side constraint of continuous
derivatives. Quite remarkably, this seemingly innocent assumptions reduces our
degrees of freedom a lot. Even more remarkably, any spline can be represented
conveniently as a linear combination of just m + q + 1 known basis functions Bj :
q+m
X
s(x) = βj Bj (x),
j=0

where

Bj (x) = xj , j = 0, . . . , q,
Bq+1+j (x) = max{0, (x − ξj )}q , j = 0, . . . , m − 1,

and βj are unknown coefficients (to be estimated). Other forms of the basis
exist, but the number of functions stays the same. You can create such a basis in
Python using statsmodels.gam.smooth basis.BSplines. Such basis functions
Bj for q = 1, 3 and m = 4 are shown in Fig. 8.4, where each color corresponds to
a different j. A spline function is simply a linear combination of these functions.
For a single covariate X ∈ [a, b], we can therefore assume
q+m
X
h(X) ≈ s(X) = βj Bj (X),
j=0
Chapter 8 Regression models 115

m=1 m=6

10
10
Visual band magnitude

Visual band magnitude

5 5

0 0
0 1 2 0 1 2
B−V color index (mag) B−V color index (mag)

Figure 8.5: Spline regression model for the color-magnitude diagram of 2655 stars
from the Hipparcos catalog using cubic splines (q = 3).

or
q+m
X
Y ≈ βj Bj,q (X) + 
j=0

which is just a linear model

Y ≈ β>Z + 

on the transformed covariates

Z = (B0 (X), . . . , Bq+m (X)) ∈ Rq+m+1 .

Since the basis functions Bj are known, the coefficients β can be estimated
easily with OLS or MLE. An example is shown in Fig. 8.5, where we fit cubic
splines with m = 1, 6 to the Hipparcos data. We observe that splines tend to
be more stable than the polynomials in the prevous section. The spline with
m = 6 has 10 parameters, but produces a very reasonable model (in contrast to
the polynomial of 10th order.)

The bias-variance trade-off

Although the usual OLS/MLE results apply, we’re not really interested in how
well the coefficients are estimated, but rather how well the function h is estimated.
Define
q+m
X
h(x) =
b βbj Bj (x)
j=0

as our estimated function.

Chapter 8 Regression models 116

One can prove the following:

Theorem 8.6. Suppose that |ξk − ξk−1 | = 1/m for all k = 1, . . . , m. Under
some regularity conditions
 
−(q+1) m
E[b
h(x)] = h(x) + O(m ), V[b
h(x)] = O , for all x ∈ [a, b].
n

Remark 8.3. The O-symbol is to be read as ‘is of the same order as’. For-
mally, for two sequences an , bn , an = O(bn ) means limn→∞ |an /bn | < ∞. More
intuitively, it says that if bn → 0, then also an → 0 at least as fast.

Let’s take this apart.

• We assume that the distance between any two subsequent knots ξk−1 and
ξk is the same. That is, the spline is defined on intervals of the same length.
(This is only to simplify result, it’s by no means necessary in practice.)

• The ‘regularity conditions’ are very mild. The main assumption is that
the true function h is a few times continuously differentiable. This just
excludes cases where h goes completely wild.

• The bias behaves like m−(q+1) , which is decreasing in m. If m is fixed, then

the estimator is usually biased. But if we take m → ∞, the bias vanishes.
Moreover, the bias vanishes faster for larger q. We conclude that we prefer
large m to make the bias small.

• The variance is of order m/n. If m is fixed, the variance goes to zero as

n → ∞. However, the variance is increasing in m, so we prefer small m to
keep the variance small.

The last two bullets describe a ubiquitous phenomenon in function estimation (as
opposed to parameter estimation). We have a tuning parameter (here m) that
controls a trade-off between bias and variance. If we decrease the variance, we
increase the bias; if we decrease the bias, we increase the variance.
Gladly, large sample sizes n reduce the variance. We can therefore afford larger
values of m when there’s a lot of data. Again there’s a sweet spot that balances
bias variance in an optimal way. One can show that the mean squared error is
optimal if m increases at the order n1/(2q+3) , but that’s not helpful in practice.
We’ll see how to solve this shortly.

Remark 8.4. One can mathematically prove that the bias-variance trade-off is
unavoidable when estimating regression or density functions. For example, the
same phenomenon appears for the histogram, where more bins decrease bias, but
increase variance.
Chapter 8 Regression models 117

Remark 8.5. Just so you know: there is another popular way to control the
bias-variance tradeoff for splines. Here, we take a large number of knots m, but
put a penalty on the magnitude of coefficients βj . This is called a penalized spline
and the strength of the penalty is controlled by a parameter α ≥ 0. For α = 0
there is no penalty, and for α = ∞, all coefficients are βj = 0.

Generalized Additive Models (GAMs)

So far, we only considered the case of a single covariate X. Let’s briefly discuss
two methods to handle multi-dimensional X ∈ Rp . The first, is to assume that
p
X
h(X) = hk (Xk ), (8.4)
k=1

where each hk is a spline function. This is called an additive model: we assume

that the the multi-dimensional function h can be decomposed into a sum of
one-dimensional functions hk . The generalized additive model (GAM) takes a
parametric model {Fθ,η : θ ∈ Θ, η ∈ E} and postulates

Y | X ∼ Fg(h(X)),η ,

where h is as in (8.4) and g is an appropriate link. Such models are still quite
easy to estimate and interpret, because the functions hk can be treated sepa-
rately. Don’t worry about the details of estimation, these models have great
implementations (e.g., statsmodels.gam).

Tensor product splines

The additivity assumption (8.4) can also be relaxed. A p-dimensional version of
the spline is the tensor product spline:
m+q m+q
X X
s(x) = ··· βj1 ,...,jp Bj1 ,q (x1 ) · · · Bjp ,q (xp ),
j1 =0 jp =0

where Bjp ,q are the basis functions from before. Tensor product splines can
approximate all continuous, p-dimensional functions with arbitrary accuracy.
However, the model has a (m + q + 1)p parameters to estimate and is much harder
to interpret. As a rule of thumb, tensor product splines are only useful when
p ≤ 3.

More on non-linear models

Non-linear and nonparametric regression is a huge and very active field of research.
We’ve only seen a small glimpse of it, so I want to mention a few popular
alternatives:
Chapter 8 Regression models 118

• Kernel methods estimate non-linear functions by (roughly) taking clever

weighted averages.

• Support vector methods use different types of expansions of the covariates

with appropriate penalties on the parameters.

• Neural networks essentially nest multiple GLMs into each other.

The principles we learned in this course also apply to these models, but require
more advanced mathematics. If you want to learn more about splines or GAMs,
there is an excellent book by Simon Wood “Generalized Additive Models: An
Introduction with R”.

8.5 Model selection

By now you’ve learned about many different models: linear models, GLMs, non-
linear models. For each of these models, you can additionally decide whether to
include or exclude some of the covariates (X1 , . . . , Xp ). So how to pick the final
model?
In any case, you should use your best judgement to rule out certain models.
For example, if Y is binary, a Gamma regression model doesn’t make sense. If a
covariate Xj is binary, it doesn’t make sense to expand it with polynomials or
splines. If you know from the physics of your problem that Xj influences Y , you
should include it in the model. If you have only 20 data points, don’t expand
covariates to the 1000th order.
Usually, your judgement only gets you so far and you are left with a number
of sensible models. Model selection refers to statistical procedures that help
you make a final choice with a theoretical foundation. To set up the stage, we
denote the candidate models M1 (θ1 ), . . . , MK (θK ), where θk is the collection
of all parameters that characterize model Mk . Everything that follows will be
phrased in this abstract context, so let’s consider a few concrete examples before
we continue.
Γ
Example 8.7. Suppose you want to choose between a Gamma-GLM Fg(β > X),ν
N
and a Gaussian GLM Fg(β> X),σ2 (which is just usual the linear model). Then

Γ N
M1 (β1 , ν) : Fg(β > X),ν , M2 (β2 , σ 2 ) : Fg(β > X),σ 2 .
1 2

Example 8.8. Consider the linear regression model Y = β1> X + . Suppose you
want to use only one covariate, but you don’t know which. Define

Mk (βk ) : Y = βk,0 + βk,1 Xk + , k = 1, . . . , p.

The extension to subsets of X of size larger than one is straightforward.

Chapter 8 Regression models 119

Example 8.9. Consider the spline GLM Fg(sm (X)),η , where m is the number of
knots the spline function sm is defined on. To choose the tuning parameter m,
define

Mk (βk ) : Fg(sk (X)),η , k = 1, . . . , K,

where βk are the spline basis coefficients we need to estimate.

Of course, all the examples above can be combined: you might want to choose
between different types of GAMs, the covariates to include, and the smoothing
parameter at the same time.
The two most popular criteria for model selection are Akaike’s information
criterion (AIC) and the Bayesian information criterion (BIC). Both are based
on the likelihood of a model. For model Mk , let

• θbk be the MLE of the model parameters θk ,

• `k (θbk ) be the corresponding log-likelihood,

• pk the number of parameters of the model,

• n the sample sized used for estimation.

Then

AIC(Mk ) = −2`k (θbk ) + 2pk ,

BIC(Mk ) = −2`k (θbk ) + ln(n)pk .

Clearly, a good model has a large likelihood, so we want AIC or BIC to be as

small (negative) as possible. The second term in the criteria is a penalty for
model complexity. The intuition is that the more parameters we add to our
model, the better is the likelihood we can achieve. But we don’t want to include
any unnecessary parameters and the penalties take care of that. The best model
according to AIC/BIC is then the one that minimizes AIC/BIC. If n > 8, the
BIC penalizes the number of parameters more strongly than AIC. Hence, the
BIC tends to select less complex models.
Both criteria can be formally shown to select the “best” model, but they differ
√ they see as best. Without going into too much detail: as n → ∞ and
in what
pk ≤ n,

• AIC selects the best predictive model among a number of possibly misspec-
ified models.

• BIC selects the true model (with minimal number of necessary parameters)
if it is included in the candidate set.

So as a general rule of thumb: if the main goal is prediction, use AIC; if the main
goal is identification of the truth, use BIC.
Chapter 8 Regression models 120

The two criteria can be used to select arbitrary statistical models with a
likelihood, not just regression models. Furthermore, for the linear model, one
commonly replaces `k by the residual sum of squares, which is equivalent to
assuming Gaussian errors.

Remark 8.6. There’s one caveat when model parameters are not estimated by
plain maximum-likelihood (like penalized splines). Then there is something called
effective number of parameters or effective degrees of freedom that needs to be
substituted for pk in the formulas above. Software usually takes care of that for
you.
 Missing data
9
Missing data refers to situations where some of the objects or quantities that we
measure are not or only partially observed. Missing data can be a problem if
the observed sample gives a biased view on the whole population. Quite often,
however, missingness can be accounted for by careful statistical modeling. This
chapter gives an overview over different types of missingness and methods to
address them.

9.1 Setup and notation

Assume that we analyze data drawn from a population with distribution FY
and density fY . Denote by Y1 , . . . , Yn the sample that we would have observed
if there were no missing data. For each of these observations, denote Ii =
1(Yi is fully observed). We shall clarify later what we mean by partially observed
data. For the moment you may assume that Ii = 0 means that we don’t see the
Yi at all.

9.2 Types of missing data

Missing completely at random (MCAR)
MCAR refers to situations where Ii is independent from Yi . This means that
whether or not we observe Yi has nothing to do with the actual value of Yi .

Example 9.1. Suppose an instrument is measuring short bursts of light occuring

at random times. Each light burst takes around 0.1s, but the instrument is only
measuring at a frequency of 1Hz. All bursts between measurements are missing.

Example 9.2. There’s a large data base of nearby stellar objects and the survey
is known to be complete. To get a sense of the data, you extract a random subset
of the observations. All objects not in this subset are missing.

This turns out to be the (rare) best case scenario. If data is MCAR, the usual
statistical procedures remain valid.
As an instructive example, suppose we want to estimate the mean Pn E[Y ]. The
1
complete-data estimator would be just the sample mean Ȳ = n i=1 Yi . Let’s
Chapter 9 Missing data 122

assume only m = ni=1 Ii < n of the data are actually observed. If we apply the
P
sample mean to the incomplete data set, we get
Pn 1
Pn
Yi Ii Yi Ii E[Yi Ii ]
Ỹ = Pni=1
= 1 Pi=1
n
n →p , (9.1)
i=1 Ii n i=1 Ii E[Ii ]

where the last step follows from applying the law of large numbers to the
numerator and denominator separately. Because Yi and Ii are independent,
E[Yi Ii ] = E[Yi ]E[Ii ] and, thus, Ỹ →p E[Y ].
The same holds true for essentially all statistical methods, including tests and
regression models. If we believe data is MCAR, there’s no reason to worry about
it any further. Unfortunately, that’s rarely the case.

Missing not and random (MNAR)

MNAR data is the worst case. It refers to situations where Yi and Ii are not
independent: the value of Yi influences whether or not we observe it.

Example 9.3. Suppose a survey is measuring stellar mass, but cannot detect
masses smaller than 1/100 of the sun’s mass. Less massive stars are missing
from the survey and massive stars are overrepresented.

Example 9.4. Suppose we measure Beryllium abundances in stars. Due to

technical limitations the instrument can only measure abundances up to three
times the sun’s abundance. The observed data will then be truncated at this value.
This is an example of partially observed data.

MNAR is problematic because it leads to a biased sample. Consider again

the sample mean based on incomplete observations. The computation in (9.1)
remains valid. But because Yi and Ii are dependent, we cannot simplify E[Yi Ii ] =
E[Yi ]E[Ii ]. Consequently, we may (and usually will) have

E[Yi Ii ]
6= E[Yi ].
E[Ii ]

For example, when large Yi are less likely to be observed, we will underestimate
the true mean.
MNAR is called non-ignorable because we have to do something about it to
obtain valid inferences. More precisely, we have to come up with a model for
mechanism leading to missing data. In general, this mechanism is not identifiable,
meaning that it cannot be estimated from the observed data. To make (approxi-
mately) valid statistical inferences nevertheless, we need to come up with a model
for the missingness mechanism. Optimally, we know a thing or two about how
data are collected and can use domain knowledge to model the mechanism. If
that’s not the case, the best we can do is making educated guesses and be very
careful in drawing conclusions from the results.
Chapter 9 Missing data 123

Missing at random (MAR)

MAR is somewhere in between MCAR and MNAR and assumes having complete
observations of additional covariates X1 , . . . , Xn (the ‘complete’ is important).
MAR means that the probability of Yi being missing depends only on the fully
observed covariates. More formally, data is MAR if Yi is independent of Ii given
Xi : for all y ∈ R, i ∈ {0, 1}, x ∈ Rp ,

P(Yi ≤ y, Ii = i | X = x) = P(Yi ≤ y | X = x)P(Ii = i | X = x).

Example 9.5. Suppose a study estimates exoplanet masses from optical photom-
etry data. If the planet doesn’t emit light for accurate photometric measurements,
it’s mass is marked as missing. Here, only the photometric measurements deter-
mine missingness in mass. Thus, conditionally on the photometric outcome, the
value of the mass and whether we observe it are independent.

MAR is less problematic than MNAR, because the missingness mechanism is

identifiable. More concretely, we can estimate a logistic regression model with
response Ii and covariates Yi . This model tells us everything we need to know
about the probabilities of an instances Yi being missing or not. As we shall see,
this information can be used to correct statistical procedures for missingness.

9.3 Censoring and truncation

Censoring and truncation are mechanisms leading to partially observed data.
Partially observed means: even though the actual value of Yi is not observed, we
instead see another value Ỹi that carries some information about Yi .
We call an observation Yi right-truncated, if all we see is Ỹi = min{Yi , c} for some
fixed value of c ∈ R. It is called left-truncated if we see Ỹi = max{Yi , c}. It is called
doubly truncated if Ỹi = max{c1 , min{Yi , c2 }}. Truncation appears frequently due
to technical limitations of measurement instruments. It is unproblematic if we
know the truncation value c. Then, the missingness mechanisms is known and
statistical methods can be adjusted rather easily. Truncation appears frequently
due to technical limitations of measurement instruments, see Example 9.4. It
is unproblematic if we know the truncation value c. Then, the missingness
mechanisms is entirely known and statistical methods can be adjusted rather
easily.
Censoring is similar to truncation, but with random truncation point Ci . For
example, Yi is right-censored if we observe Ỹi = min{Yi , Ci } and similarly for left-
and doubly censored data. It’s also possible to correct for this type of missingness
if we additionally observe a censoring indicator Ii = 1(Yi ≤ Ci ).

Example 9.6. Censoring occurs most commonly when Yi is a time of some event
of interest; for example the time until light is reflected back to earth. We can wait
only a finite amount of time until this happens. If the light has not been reflected
Chapter 9 Missing data 124

at that time, all we know is that (i) it has not yet been reflected back (Ii = 0),
(ii) that the actual time Yi must be larger than Ỹi = time between emission from
earth and the end of the study.

9.4 Correcting for missingness

Dealing with missing data has a long history in statistics and there are many
techniques to account for missing data. They can be broadly classified into three
categories:

1. Likelihood methods: explicit modifications of the likelihood function, often

involving integration.

2. Imputation: Missing observations Yi are substituted with ‘plausible’ values Yi∗

generated from some model.

3. Inverse probability weighting (IPW): Complete observations are upweighted

by the inverse probability of them being observed. (That’s what you did in
the assignment.)

We shall only consider the third category here, because it is both easy to implement
and very general.
Let’s start with a general setup. Suppose we have a model for the probability
π(Yi , Xi ) = P(Ii = 1 | Yi , Xi ). Here, Ii = 1(Yi is fully observed) such that Ii = 0
if Yi is not or only partially observed. If no covariates Xi are available, they can
be omitted in the formulas, i.e., π(Yi , Xi ) = π(Yi ).
The idea is as follows: first, we throw away all incomplete observations (includ-
ing partially observed ones). Now for all y, x with π(y, x) < 1, observations with
(Yi , Xi ) = (Y, x) are underrepresented in the remaining data; a complete data
set would contain 1/π(y, x) times more of such observations. We correct for this
by up-weighting the complete observations by this factor. This assumes that the
quantity we compute is a sum or average. But as I’ve said earlier in the semester:
almost everything in statistics is an average or well approximated by one.
To make this more concrete, reconsider the example of estimating the mean
E[Yi ] from incomplete (MNAR or MAR) data. The IPW version of the sample
mean is
n
IP W 1 X Yi Ii
Ȳ = .
n i=1 π(Yi , Xi )

The Ii in the numerator is responsible for “throwing away all incomplete data”.
The π(Yi , Xi ) in the numerator is up-weighting the complete cases. By the law
of large numbers, we have
 
IP W Yi Ii
Ȳ →p E .
π(Yi , Xi )
Chapter 9 Missing data 125

Using the Tower rule, we get

    
Yi Ii Yi Ii
E =E E | Y i , Xi
π(Yi , Xi ) π(Yi , Xi )
 
Yi  
=E E Ii | Yi , Xi
π(Yi , Xi )
 
Yi
=E π(Yi , Xi )
π(Yi , Xi )
= E[Yi ].

The second equality is due to the fact that Yi , Xi are fixed numbers if we condition
on Yi , Xi . Hence, the IPW version of the sample mean is a consistent estimator.
The same arguments apply whenever we rely on estimating one or more ex-
pectations of the form E[g(Yi )] for some function g(Yi ). That covers almost
everything we learned in this course, including empirical CDFs, histograms, sam-
ple quantiles, sample variances, maximum-likelihood estimators, etc. In practice,
the probabilities π(Yi , Xi ) are rarely known. If we observe the indicator Ii , we
can estimate them with a regression model. If the indicator Ii is unobserved, we
have to use domain knowledge and EDA to postulate a plausible model.

9.5 Takeaways
1. Missing data problems are common and it is important to think about them
carefully. What type of missingness do you face? What is the reason/mecha-
nism for incomplete observations?

2. There is a large arsenal of well-established methods to account for missing

data. As soon you have identified what type you are facing, search for a
method that’s suitable for your specific problem. IPW will work in most cases,
but sometimes it’s worth investigating other options.
 10
Bayesian inference
This chapter gives a very brief introduction to Bayesian inference. Bayesianism
is an entire school of statistics and deserves its on course. However, Bayesian
methods have become quite popular in astronomy, so it’s worthwhile understand-
ing the basics. The goal of this chapter is to become familiar with the core ideas
and concepts. You can learn more about Bayesian methods in the MSc course
Modern Astrostatistics.

10.1 Frequentism vs. Bayesianism

The statistical methods and related results we’ve seen so far all operate under the
frequentist paradigm. They focus on long-run frequencies of events. For example,
consistency requires long-run frequencies of “being away from the true parameter”
to vanish. Confidence intervals are set such “they cover the true parameter” with
prescribed frequency. Statistical procedures are then designed to adhere to these
targets. You may notice that I used the term ‘frequency’ instead of ‘probability’
in the last sentences. And that’s the clue:

For a frequentist, probabilities are the long-run limit of frequencies.

We already faced an important consequence in Section 5.4.1 and Section 6.5:

because the true parameters of our probability model are fixed numbers, we
cannot make meaningful probabilistic statements about them. They attain
their true values with frequency/probability 100% and any other value with
frequency/probability 0%.
The Bayesian paradigm is fundamentally different.

For a Bayesian, probabilities express degree of belief.

An immediate consequence is that a Bayesian can make meaningful probability

statements about unknown parameters — even before having seen any data. If I
ask you how likely it is that I woke up before 8 AM today, you may answer 70%.
That’s a Bayesian probability: Either I woke up before 8 AM (100% frequency)
or I didn’t (0% frequency), but 70% is nevertheless a valid description of your
belief. You think it’s roughly twice as likely that I woke before 8 than after 8.
Because you haven’t seen any data on my morning routine, the 70% is called
a prior probability (as in ‘belief prior to seeing any data’). Bayesian statistical
Chapter 10 Bayesian inference 127

procedures are then designed to update our belief optimally after seeing some
data.
As you can see, frequentism vs. Bayesianism is a matter of philosophy. There
has been quite some dispute over the right or wrong over the last decades and
some hold strong opinions. Nowadays, the majority of statisticians take a rather
neutral stance and use whatever is most convenient in a given situation.

10.2 Bayesian model setup

As usual, suppose we want to make inferences about an unknown parameter θ
from repeated observations of a random variable X. Bayesian procedures all
follow the same recipe.

1. Choose a prior probability density π(θ) that expresses our beliefs about the
unknown parameter θ.

2. Choose a statistical model f (x | θ) = f (x; θ) that reflects our beliefs about

the behavior of X if the true parameter would be θ.

3. Calculate the posterior density f (θ | X1 , . . . , Xn ), which reflects our updated

belief after having observed data X1 , . . . , Xn .

Note that step 1 only makes sense if we view the unknown parameter as a random
variable Θ. In that view, the true parameter θ is the realization of this random
variable in our universe.1 Accordingly we write the statistical model in step 2 as
f (x | θ). It is our model for the data conditional on the event Θ = θ. In practice,
this model is formulated just as in the frequentist paradigm.

Example 10.1. Suppose we want to make inferences about the parameter p ∈

(0, 1) of a Bernoulli distribution. In the Bayesian paradigm, we view this param-
eter as a random variable P . If we take an agnostic view on the distribution
of P , we would specify a flat prior, where all values in (0, 1) are equally likely.
That is, our prior is the uniform distribution, π(p) = 1 for p ∈ (0, 1). Condi-
tional on P = p, we belief that our data X1 , . . . , Xn are iid and have Bernoulli(p)
distribution. Using Bayesian conventions, the entire model is written succinctly
as

X1 , . . . , Xn ∼ Bernoulli(p), p ∼ Uniform(0, 1).

10.3 Bayesian updating

The big question is how we come up with the posterior in step 3. And that’s
where the name Bayesian comes from. Suppose for the moment that both X and
1
Most Bayesians do not distinguish random variable Θ and realization θ in notation and just
write θ for both.
Chapter 10 Bayesian inference 128

and Θ are discrete. Then Bayes theorem (Theorem 2.25) and the law of total
probability (Theorem 2.26) give

P(X = x | Θ = θ)P(Θ = θ)
f (θ | x) = P(Θ = θ | X = x) =
P(X = x)
P(X = x | Θ = θ)P(Θ = θ)
=P
θ P(X = x | Θ = θ)P(Θ = θ)
P(X = x | Θ = θ)π(θ)
=P
θ P(X = x | Θ = θ)π(θ)

If X and Θ are continuous, the analogous statement is

f (x | θ)π(θ)
f (θ | x) = R .
f (x | θ)π(θ)dθ

The above rule gives the optimal update of our belief π(θ) having seen a single
observation X = x. If we see multiple iid observations X1 , . . . , Xn , we replace
the likelihood of a single observation f (x | θ) by the joint likelihood
n
Y
Ln (θ) = f (X1 , . . . , Xn | θ) = f (Xi | θ).
i=1

This gives,

Ln (θ)π(θ)
f (θ | X1 , . . . , Xn ) = R .
Ln (θ)π(θ)dθ
R
The denominator Ln (θ)π(θ)dθ is called the marginal likelihood of the data. It
is a normalizing constant not depending on θ and usually irrelevant for inference.
Thus the main take away is

f (θ | X1 , . . . , Xn ) ∝ Ln (θ)π(θ)

(posterior is proportional to likelihood times prior). This posterior reflects our

updated belief — coming from prior belief π(θ) and having seen data X1 , . . . , Xn .
In the vast majority of cases, the posterior density is complex and we use
simulation for inference (more on that later). It is instructive to study at least
one case where the posterior has a simple form.

Example 10.2. We continue with the model from Example 10.1:

X1 , . . . , Xn ∼ Bernoulli(p), p ∼ Uniform(0, 1).

Using our main finding, we get

Pn Pn
Xi
f (p | X1 , . . . , Xn ) ∝ Ln (p)π(p) = Ln (p) = p i=1 (1 − p)n− i=1 Xi
.
Chapter 10 Bayesian inference 129

4 type
density

prior
posterior (20 obs)
posterior (50 obs)
2

0
0.00 0.25 0.50 0.75 1.00
p

Figure 10.1: Prior and posterior densities in Example 10.2

A seasoned statistician
Pn would immediately
Pn realize that this is proportional to the
density of a Beta( i=1 Xi + 1, n − i=1 Xi + 1) random variable (‘proportional’
only because we threw away the normalizing constant). Hence, our posterior belief
about the unknown parameter p is expressed as
n
X n
X 
p | X1 , . . . , Xn ∼ Beta Xi + 1, n − Xi + 1 .
i=1 i=1

NowPnsuppose that in reality p = 0.3 and we observe n = 20 observations such

that i=1 Xi = 6. Fig. 10.1 shows the prior, π(p) = 1, as a solid line and
the posterior as dotted line. Intuitively, we saw 6/20 = 0.3 observations with
Xi = 1 so our updated beliefs should reflect that. Indeed, we see that the posterior
concentrates onPna region near p = 0.3. If we get 30 additional data points (so
n = 50) with i=1 Xi = 15, we can strengthen our posterior beliefs and this is
reflected by a higher degree of concentration (dashed line).

10.4 Bayesian inference

Bayesian methods for parameter estimation and uncertainty quantification are
all based on the posterior density f (θ | X1 , . . . , Xn ) ∝ Ln (θ)π(θ). The standard
estimator for the unknown θ is the posterior mean
Z
θ̄n = E[Θ | X1 , . . . , Xn ] = θf (θ | X1 , . . . , Xn )dθ.
Chapter 10 Bayesian inference 130

To quantify uncertainty, we can construct a Bayesian credible interval C such

that
Z
P(θ ∈ C | X1 , . . . , Xn ) = f (θ | X1 , . . . , Xn )dθ = 1 − α.
C

The set C is called credible interval instead of confidence interval to emphasize the
difference in paradigm. The above is a probability statement about the unknown
parameter θ (which would be meaningless under the frequentist paradigm): Given
the data we have, our belief that θ ∈ C is described by the probability 1 − α.
If the posterior density has a simple form, posterior means and credible sets
are easy to extract.

Example 10.3. Let’s continue where we left off in (10.2):

n
X n
X 
p | X1 , . . . , Xn ∼ Beta Xi + 1, n − Xi + 1 .
i=1 i=1

The mean of a Beta(α, β) is α/(α + β), so the posterior mean is

Pn  
i=1 Xi + 1 n n 1
p̄ = = X̄n + 1 − .
n+2 n+2 n+2 2

Hence, the posterior mean is a weighted average between the sample mean X̄n
(which is the frequentist MLE) and the prior mean, 1/2. As n → ∞, the weight
for the sample mean, n/(n + 2), tends to one. This is a general phenomenon:
for large samples, Bayesian and frequentist estimates are very similar. On small
samples, the prior contribution matters however.

10.5 The choice of prior

The prior plays an important role in Bayesian methods. First and foremost, it
makes them inherently subjective. Militant Bayesians see this is as the main
advantage, militant frequentists as the main disadvantage. In any case, the choice
of prior can be important, and choosing a bad prior can set us up for failure.

10.5.1 Bad priors

To illustrate how bad priors can go wrong, we’ll go back to the origins of Bayes’
theorem. Bayes’ motivation for his framework was to model how rational people
should make decisions under uncertainty. One such decision is to vote for our
political leaders. Suppose there are only two options, candidates A and B. People
vote for the candidate that they believe to be most suitable for the role. Let’s
assume that all people are rational and update their beliefs according to Bayes’
theorem. In reality, candidate A is a huge idiot and everything he does reflects
that. What happens if some people have the prior belief P(A is best) = 1?
Chapter 10 Bayesian inference 131

10.0

7.5
type
density

prior
5.0
posterior (20 obs)
posterior (50 obs)

2.5

0.0
0.00 0.25 0.50 0.75 1.00
p

Figure 10.2: The effect of a biased prior on the posterior densities in Example 10.2

Well, no matter the evidence, if they apply Bayes theorem to update their
beliefs, they will end up with P(A is best | data) = 1. (You can do the calculation
yourself.) They will vote for the idiot, no matter how outrages his actions. This
is a very extreme example, but a weaker phenomenon occurs if a voter’s prior
is strongly biased towards one candidate. The stronger the bias, the more
contradictory evidence the voter must see before changing their opinion.
So what does that mean for Bayesian procedures? First, if we assign prior
probability 0 to some region C of the parameter space, the posterior probability
for this region will inevitably be 0. Thus, we should make sure that the prior
density assigns positive mass to all possible outcomes. Second, if our prior heavily
favors a certain region, this bias will only gradually fade out from the posterior.
This is illustrated in Fig. 10.2, where we chose a prior that is heavily biased
towards large values of p. Given the same data as in Fig. 10.1, the posteriors
preserve the bias for large values of p and only gradually move towards the true
value p = 0.3.

10.5.2 Improper and non-informative priors

To avoid unreasonable bias in the posterior, we may be tempted to assign all values
equal prior probability. This is what we’ve done in Example 10.1 by choosing a
flat prior, π(θ) ∝ const. If the parameter rangeR for θ is unbounded,
R flat priors
cannot be proper probability densities (because π(θ)dθ = const.dθ = ∞ = 6 1).
In that case, we say that the prior is improper. Improper priors are usually
unproblematic, the posterior will be proper density nevertheless.
A more subtle question is whether flat priors are really as agnostic as they seem.
Spoiler: they are not. The key insight is that flat priors are not transformation
invariant. Suppose that g is some nonlinear function. If we reparametrize our
Chapter 10 Bayesian inference 132

model by g(θ) instead of θ, the flat prior for θ turns into a non-flat prior for
g(θ) (this follows from the transformation-of-densities theorem, Theorem 2.58).
A better choice for non-informative priors is Jeffrey’s prior

π(θ) = I(θ)1/2 ,

where I(θ) is the Fisher information from Section 6.3. This prior can indeed be
shown to be transformation invariant, but is often hard to compute.

10.5.3 Conjugate priors

Conjugate priors are a particularly convenient choice. A prior is called conjugate
(relative to the statistical model f (x | θ)), if both the prior π(θ) as well as the
posterior f (θ | X1 , . . . , Xn ) belong to the same parametric family. An example is
the Beta distribution in Example 10.1. If we choose p ∼ Beta(α, β) as our prior,
one can show that the posterior is
n
X n
X 
p | X1 , . . . , Xn ∼ Beta Xi + α, n − Xi + β .
i=1 i=1

(The flat prior π(θ) ≡ 1 is a special case with α = β = 1.) Conjugate priors are
nice because they allow to do everything in closed form. However, there is only a
handful of rather simple statistical models for which conjugate priors are known.

10.5.4 The good news: a frequentist perspective

We may ask whether Bayesian methods are reasonable in terms of their frequen-
tists properties.2 In summary, the answer is ‘yes’ as long the prior isn’t plain
stupid. The only condition we need is that the prior assigns positive mass to the
true value θ. Then posterior means are consistent and asymptotically normal
and (1 − α)-credible intervals cover the true parameter with probability 1 − α.
And even if our prior is biased, the bias will disappear at the
√ order 1/n and is
therefore asymptotically negligible (compared to the usual 1/ n contribution of
the variance).

10.6 Bayesian inference via simulation

For non-conjugate models, the default strategy for Bayesian inference relies on
simulation from the posterior. As mentioned earlier, posterior densities are often
complex and only known up to the normalizing constant. So how can we simulate
from them? The solution is a simulation technique called Markov Chain Monte
Carlo or MCMC for short. Let me say in advance that you rarely have to

2
We need to switch to the frequentists paradigm to objectively assess the quality of estimators.
The Bayesian view is always subjective through the prior.
Chapter 10 Bayesian inference 133

implement this yourself, there are excellent libraries (like emcee and PyStan for
Python). So we’ll just quickly brush over the main idea.
The goal is to simulate from the posterior density f (θ | X1 , . . . , Xn ) ∝
Ln (θ)π(θ), optimally without knowing the normalizing constant. We do this
by constructing a stationary Markov chain Θ1 , . . . , ΘT . The sequence of random
variables Θ1 , . . . , ΘT is called stationary if all Θt have the same distribution. In
our case, we want this distribution to be the posterior. However, random variables
in a Markov Chain are not independent. They only need to satisfy the Markov
property
f (Θt | Θt−1 , . . . , Θ1 ) = f (Θt | Θt−1 ).
Hence, the distribution of Θt depends on the past realizations Θ1 , . . . , Θt−1 , but
only through the most recent element Θt−1 .
The dependence in a stationary Markov chain is weak enough for the law of
large numbers to hold. So if we are able to simulate such a stationary Markov
chain Θ1 , . . . , ΘT , the posterior mean can be approximated by
T
1X
θb = Θt
T t=1

and a (1 − α) credible interval can be computed from corresponding α/2 and

(1−α/2) sample quantiles. The quality of these approximations depends primarily
on the length of the chain (T ) and the strength of dependence between consecutive
elements. It is customary to throw away a good portion (maybe 5-10%) of the
first elements of the chain (the so-called burn-in period ), because most simulation
algorithms take a while to stabilize.
So how do we simulate from the desired Markov chain? This has been and still
is a very active field of research, so let’s only have a quick look at the simplest
method of all, the Metropolis-Hastings algorithm.

1. Pick some density q(· | Θt−1 ) that we can easily simulate from, called proposal
distribution. For example we may take q(· | Θt−1 ) ∼ N (Θt−1 , σ 2 ) if θ ∈ R or a
suitable Beta density if θ ∈ (0, 1).

2. Set Θ0 to an arbitrary value.

3. For t = 1, . . . , T :
(i) Simulate a proposal value Θt ∼ q(· | Θt−1 ).
(ii) Compute
Ln (Θt )π(Θt )q(Θt−1 | Θt )
R= .
Ln (Θt−1 )π(Θt−1 )q(Θt | Θt−1 )

(iii) Simulate U ∼ Uniform(0, 1). If R > U , set Θt = Θt−1 .

R
Note that we do not require the normalizing constant Ln (θ)π(θ) anywhere in
the algorithm. One can show that the algorithm indeed produces the desired
Chapter 10 Bayesian inference 134

Markov chain if q assigns positive probability to all values in the parameter range
of θ. How much dependence there is between θt and θt−1 depends on two factors:
how much the proposal density q(· | Θt−1 ) concentrates around Θt−1 and how
often we set Θt = Θt−1 in step 3(iii). The dependence will be weaker, the closer
q(· | θt−1 ) is to the posterior density f (θ | X1 , . . . , Xn ).

10.7 Concluding remarks

You should now be familiar with the core ideas behind Bayesian inference. There
is also a Bayesian variant of hypothesis testing. Here, so-called Bayes factors
play a similar role to frequentist p-values. But Bayes factors actually do quantify
the probability of a certain hypothesis given the data (and our prior belief). That
was impossible in the frequentist paradigm.
The Bayesian paradigm has many advantages and equally many disadvantages
over the frequentist one. For some people, the main advantage is that we can
incorporate prior information in a principled manner. Sometimes this is important,
but more often than not specifying the prior is treated as ancillary task. Some
find the Bayesian philosophy more intuitive. Being able to make probability
statements about unknown parameters and hypotheses is a nice consequence.
There is another advantage that I think is the driving force behind its popularity:
it is user friendly. No matter how complex a model is, we always do the same:
write down the likelihood, specify priors for all parameters, and throw them into
some Python function. The function returns samples from the posterior density
and we get parameter estimates and uncertainty quantification for free (more or
less).
All these points have a flipside. First and foremost, we absolutely must specify
a prior. For complex models with many parameters, coming up with a justifiable
choice is hard. More importantly, priors make our inferences inherently subjec-
tive. While we can make probability statements about unknown parameters and
hypotheses, someone else holding different priors would end up with different
answers. Which one is more valid scientifically? Hard to say. Furthermore,
throwing every model into an MCMC algorithm is convenient but computation-
ally demanding. Especially for very large samples, this can be annoying or even
prohibitive.
In the end, both philosophies have their merits. Empirically, well-designed
Bayesian and frequentists methods give almost the same answers (there’s even
mathematical theory to back that up). In my opinion, modern statisticians should
have both in their toolbox and decide on a case-by-case basis what suits the
problem better.
Lastly, a word of caution. There is a lot of nuance to the theory that was
left out intentionally. Before you go all Bayesian with your data, make sure to
consult other textbooks3 .

3
I heard ‘Statistical rethinking’ by Richard McElreath is nice for applications.