Statistical Learning Intro
Statistical Learning Intro
Lecture 1.
Czech Technical University in Prague
Course format
2/10
Teachers: Jan Drchal, Boris Flach, Vojtech Franc and Daniel Bonilla
Format: 1 lecture & 1 tutorial per week (6 credits), tutorials of two types
seminars: discussing solutions of theoretical assignments (published a week before the
Code either as source file or as python notebook. The code must be executable.
Grading: 40% homeworks + 60% written exam = 100% (+ bonus points)
Prerequisites:
probability theory and statistics (A0B01PSI)
optimisation (AE4B33OPT)
computational biology (genes, proteins) and biological imaging & medical diagnosis
computer vision,
You will gain skills to construct learning systems for typical applications by successfully
combining appropriate models and learning methods.
Characters of the play
4/10
object features x ∈ X are observable; x can be:
i.e. `(y, h(x)) is the loss for predicting y 0 = h(x) when y is the true state
Typical concepts:
regression: Y = f (X) + , where f is unknown and is a random error,
classification: p(x, y) = p(y)p(x | y), where p(y) is the prior class probability and
XX
R(h) = E[`(Y, h(X))] = p(x, y)`(y, h(x))
x∈X y∈Y
m
1 X i
R(h) ≈ RS m (h) = `(y , h(xi))
m i=1
Q: how strong can they deviate from each other? (see next lectures)
P |RS m (h) − R(h)| > ≤??
Statistical machine learning
7/10
Choosing an optimal inference rule h(x)
If p(x, y) is known:
The corresponding best possible inference rule is the Bayes inference rule
X
∗
h (x) = arg min p(y | x)`(y, y 0)
y 0 ∈Y y∈Y
But p(x, y) is not known and we can only collect examples drawn from it. We need:
Learning algorithms that use training data and prior assumptions/knowledge about the task
Learning types
8/10
Training data:
m
i i
if T = (x , y ) ∈ X × Y | i = 1, . . . , m ⇒ supervised learning
m
i
if T = x ∈ X | i = 1, . . . , m ⇒ unsupervised learning
m1 S m2
m
if T = Tl Tu , with labelled training data Tlm1 and unlabelled training data Tum2
⇒ semi-supervised learning
1 X
RT (h) = `(y, h(x))
|T |
(x,y)∈T
P |R(h∗T ) − R(h∗)| > ≤??
Learning types
9/10
Generative learning: assume that the true p.d. p(x, y) is in some parametrised family
2. set h∗T = hθT∗ , where hθ denotes the Bayes inference rule for the p.d. pθ .
Q: How strong can θT∗ deviate from θ∗? How does this deviation depend on PΘ?
discr. gener.
superv. yes yes
semi-sup. (yes) yes
unsuperv. no yes
In this course:
discriminative: Support Vector Machines, Deep Neural Networks
How would you estimate the optimal θ∗ by using the MLE? What is the Bayes inference
rule for pθ∗ ?