Lecture 1: Introduction to Neural Networks and Deep Learning

Deep Learning @ UvA

UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 1
Prerequisites
o Calculus, Linear Algebra
◦ Derivatives
◦ Matrix operations
o Probability and Statistics
o Advanced programming
o Time and patience

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 2- 2
OPTIMIZATIONS
Course overview
o Idea: Go in depth in theory & get hands-on practical experience
o What will you learn?
◦ How to train Deep Learning models
◦ Neural Networks for Computer Vision
◦ Neural Networks for Language
◦ Unsupervised and Bayesian Deep Learning
◦ Deep Reinforcement Learning
o All material uploaded on the course website
o Book on Neural Networks and Deep Learning from Y. Bengio

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 3- 3
OPTIMIZATIONS
Course Logistics
o Course: Theory (4 hours per week) + Labs (4 hours per week)
o Rooms are not always the same, check Datanose
o Final grade = 50% from lab assignments + 50% from final exam
◦ Exam moved to Dec 20, 13.00-15.00, check Datanose

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 4- 4
OPTIMIZATIONS
Practicals
o 6 lab assignments=5 practical assignments + 1 presentation
◦ Equally weighed
o Practical assignments done individually
◦ Python + Tensorflow
o Presentation in groups of 3
◦ Pick your team & paper by Nov 30
◦ Present end of December, schedule announced after Nov 30
◦ 7 min per presentation, 3 min for questions, we will give you template
◦ Graded: 50% presentation skills (group), 50% per student Q&A (individual)

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 5- 5
OPTIMIZATIONS
Who we are and how to reach us
o Efstratios Gavves, TAs: Kirill Gavrilyuk, Berkay Kicanaoglu, Patrick Putzky
◦ uva.deeplearning@gmail.com

o Course website: http://uvadlc.github.io

◦ Lecture sides & notes, practicals
o Virtual classroom
◦ Piazza: www.piazza.com/university_of_amsterdam/spring2016/uvadlc/home
◦ Datanose: https://datanose.nl/#course[55904]

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 6- 6
OPTIMIZATIONS
Lecture Overview
o Deep Learning in
◦ Computer Vision
◦ Natural Language Processing (NLP)
◦ Speech
◦ Robotics and AI
◦ Music and the arts!
o A brief history of Neural Networks and Deep Learning
o Basics of Neural Networks

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 7- 7
OPTIMIZATIONS
 Deep Learning in
Computer Vision

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 8
Object and activity recognition

Click to go to the video in Youtube

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 9- 9
OPTIMIZATIONS
Object detection and segmentation

Click to go to the video in Youtube

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 10
OPTIMIZATIONS - 10
Image captioning and Q&A
Click to go to the video in Youtube Click to go to the website

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 11
OPTIMIZATIONS - 11
Why should we be impressed?
o Vision is ultra challenging!
◦ For a small 256x256 resolution and for 256 pixel values
◦ a total 2524,288 of possible images
◦ In comparison there are about 1024 stars in the universe
o Visual object variations
◦ Different viewpoints, scales, deformations, occlusions
o Semantic object variations
◦ Intra-class variation
◦ Inter-class overlaps

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 12
OPTIMIZATIONS - 12
 Deep Learning in
Robotics

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 13
Self-driving cars

Click to go to the video in Youtube

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 14
OPTIMIZATIONS - 14
Drones and robots

Click to go to the video in Youtube

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 15
OPTIMIZATIONS - 15
Game AI
Click to go to the video in Youtube

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 16
OPTIMIZATIONS - 16
Why should we be impressed?
o Typically robotics are considered in controlled environments
◦ Laboratory settings, Predictable positions, Standardized tasks (like in factory robots)
o What about real life situations?
◦ Environments constantly change, new tasks need to be learnt without guidance,
unexpected factors must be dealt with
o Game AI
1048
◦ At least 10 possible GO games. Where do we even start?

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 17
OPTIMIZATIONS - 17
 Deep Learning in
NLP and Speech

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 18
Word and sentence representations

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 19
OPTIMIZATIONS - 19
Speech recognition and Machine translation

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 20
OPTIMIZATIONS - 20
Why should we be impressed?
o NLP is an extremely complex task
◦ synonymy (“chair”, “stool” or “beautiful”, “handsome”)
◦ ambiguity (“I made her duck”, “Cut to the chase”)
o NLP is very high dimensional
◦ assuming 150K english words, we need to learn 150K classifiers
◦ with quite sparse data for most of them

o Beating NLP feels the closest to achieving true AI

◦ although true AI probably goes beyond NLP, Vision, Robotics, … alone

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 21
OPTIMIZATIONS - 21
 Deep Learning in
the arts

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 22
Imitating famous painters

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 23
OPTIMIZATIONS - 23
Or dreaming …

Click to go to the video in Youtube

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 24
OPTIMIZATIONS - 24
Handwriting

Click to go to the website

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 25
OPTIMIZATIONS - 25
Why should we be impressed?
o Music, painting, etc. are tasks that are uniquely human
◦ Difficult to model
◦ Even more difficult to evaluate (if not impossible)
o If machines can generate novel pieces even remotely resembling art, they
must have understood something about “beauty”, “harmony”, etc.
o Have they really learned to generate new art, however?
◦ Or do they just fool us with their tricks?

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 26
OPTIMIZATIONS - 26
 Charles W.
Wightman Frank
Rosenblatt
A brief history of
Neural Networks &
Deep Learning

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 27
First appearance (roughly)

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 28
OPTIMIZATIONS - 28
Perceptrons
o Rosenblatt proposed a machine for binary classifications
o Main idea
◦ One weight 𝑤𝑖 per input 𝑥𝑖
◦ Multiply weights with respective inputs and add bias 𝑥0 =+1
◦ If result larger than threshold return 1, otherwise 0

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 29
OPTIMIZATIONS - 29
Training a perceptron
o Rosenblatt’s innovation was mainly the learning algorithm for perceptrons
o Learning algorithm
◦ Initialize weights randomly
◦ Take one sample 𝑥𝑖 and predict 𝑦𝑖
◦ For erroneous predictions update weights
◦ If the output was 𝑦ෝ𝑖 = 0 and 𝑦𝑖 = 1, increase weights
◦ If the output was 𝑦ෝ𝑖 = 1 and 𝑦𝑖 = 0, decrease weights
◦ Repeat until no errors are made

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 30
OPTIMIZATIONS - 30
From a perceptron to a neural network
o One perceptron = one decision
o What about multiple decisions?
◦ E.g. digit classification
o Stack as many outputs as the
possible outcomes into a layer
◦ Neural network
o Use one layer as input to the next layer
◦ Multi-layer perceptron (MLP)

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 31
OPTIMIZATIONS - 31
XOR & Multi-layer Perceptrons
Output
o However, the exclusive or (XOR) cannot
be solved by perceptrons
◦ [Minsky and Papert, “Perceptrons”, 1969] 𝑤1 𝑤2

◦ 0 𝑤1 + 0𝑤2 <𝜃 →0<𝜃

◦ 0 𝑤1 + 1𝑤2 >𝜃 → 𝑤2 > 𝜃
◦ 1 𝑤1 + 0𝑤2 >𝜃 → 𝑤1 > 𝜃 Input 1 Input 2
◦ 1 𝑤1 + 1𝑤2 <𝜃 → 𝑤1 + 𝑤2 < 𝜃
Input 1 Input 2 Output
1 1 0
Inconsistent!!
1 0 1
0 1 1
0 0 0

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 32
OPTIMIZATIONS - 32
Minsky & Multi-layer perceptrons
o Interestingly, Minksy never said XOR cannot be
solved by neural networks
◦ Only that XOR cannot be solved with 1 layer perceptrons 𝑦𝑖 = {0, 1}

o Multi-layer perceptrons can solve XOR

◦ 9 years earlier Minsky built such a multi-layer perceptron
o However, how to train a multi-layer perceptron?
o Rosenblatt’s algorithm not applicable, as it
expects to know the desired target
◦ For hidden layers we cannot know the desired target

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 33
OPTIMIZATIONS - 33
Minsky & Multi-layer perceptrons
𝑎𝑖 =? ? ?
o Interestingly, Minksy never said XOR cannot be
solved by neural networks
◦ Only that XOR cannot be solved with 1 layer perceptrons 𝑦𝑖 = {0, 1}

o Multi-layer perceptrons can solve XOR

◦ 9 years earlier Minsky built such a multi-layer perceptron
o However, how to train a multi-layer perceptron?
o Rosenblatt’s algorithm not applicable, as it
expects to know the desired target
◦ For hidden layers we cannot know the desired target

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 34
OPTIMIZATIONS - 34
The “AI winter” despite notable successes

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 35
OPTIMIZATIONS - 35
The first “AI winter”
o What everybody thought: “If a perceptron cannot even solve XOR, why bother?
◦ Also, the exaggeration did not help (walking, talking robots were promised in the 60s)
o As results were never delivered, further funding was slushed, neural networks
were damned and AI in general got discredited
o “The AI winter is coming”
o Still, a few people persisted
o Significant discoveries were made, that laid down the road for today’s
achievements

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 36
OPTIMIZATIONS - 36
Backpropagation
o Learning multi-layer perceptrons now possible
◦ XOR and more complicated functions can be solved
o Efficient algorithm
◦ Process hundreds of example without a sweat
◦ Allowed for complicated neural network architectures
o Backpropagation still is the backbone of neural network training today
o Digit recognition in cheques (OCR) solved before the 2000

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 37
OPTIMIZATIONS - 37
Recurrent networks
o Traditional networks are “too plain”
◦ Static Input  Processing  Static Output
o What about dynamic input
◦ Temporal data, Language, Sequences
o Memory is needed to “remember” state changes
◦ Recurrent feedback connections
o What kind of memory
◦ Long, Short?
◦ Both! Long-short term memory networks (LSTM), Schmidhuber 1997

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 38
OPTIMIZATIONS - 38
The second “AI winter”
o Until 1998 some nice algorithms and methods were proposed
◦ Backpropagation
◦ Recurrent Long-Short Term Memory Networks
◦ OCR with Convolutional Neural Networks
o However, at the same time Kernel Machines (SVM etc.) suddenly become very
popular
◦ Similar accuracies in the same tasks
◦ Neural networks could not improve beyond a few layers
◦ Kernel Machines included much fewer heuristics & nice proofs on generalization
o As a result, once again the AI community turns away from Neural Networks

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 39
OPTIMIZATIONS - 39
The thaw of the “AI winter”

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 40
OPTIMIZATIONS - 40
Neural Network and Deep Learning problems
o Lack of processing power
◦ No GPUs at the time
o Lack of data
◦ No big, annotated datasets at the time
o Overfitting
◦ Because of the above, models could not generalize all that well
o Vanishing gradient
◦ While learning with NN, you need to multiply several numbers 𝑎1 ∙ 𝑎2 ∙ ⋯ ∙ 𝑎𝑛 .
◦ If all are equal to 0.1, for 𝑛 = 10 the result is 0.0000000001, too small for any learning

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 41
OPTIMIZATIONS - 41
Despite Backpropagation …
o Experimentally, training multi-layer perceptrons was not that useful
◦ Accuracy didn’t improve with more layers
o The inevitable question
◦ Are 1-2 hidden layers the best neural networks can do?
◦ Or is it that the learning algorithm is not really mature yet

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 42
OPTIMIZATIONS - 42
Deep Learning arrives
o Layer-by-layer training
◦ The training of each layer individually is an
easier undertaking
o Training multi-layered neural networks
became easier
o Per-layer trained parameters initialize
further training using contrastive Training layer 1
divergence

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 43
OPTIMIZATIONS - 43
Deep Learning arrives
o Layer-by-layer training
◦ The training of each layer individually is an
easier undertaking
o Training multi-layered neural networks
became easier Training layer 2

o Per-layer trained parameters initialize

further training using contrastive
divergence

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 44
OPTIMIZATIONS - 44
Deep Learning arrives
o Layer-by-layer training
◦ The training of each layer individually is an
easier undertaking Training layer 3

o Training multi-layered neural networks

became easier
o Per-layer trained parameters initialize
further training using contrastive
divergence

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 45
OPTIMIZATIONS - 45
Deep Learning Renaissance

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 46
OPTIMIZATIONS - 46
More data, more …
o In 2009 the Imagenet dataset was published [Deng et al., 2009]
◦ Collected images for each term of Wordnet (100,000 classes)
◦ Tree of concepts organized hierarchically
◦ “Ambulance”, “Dalmatian dog”, “Egyptian cat”, …
◦ About 16 million images annotated by humans
o Imagenet Large Scale Visual Recognition Challenge (ILSVRC)
◦ 1 million images
◦ 1,000 classes
◦ Top-5 and top-1 error measured

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 47
OPTIMIZATIONS - 47
Alexnet
o In 2013 Krizhevsky, Sutskever and Hinton re-implemented [Krizhevsky2013] a
convolutional neural network [LeCun1998]
◦ Trained on Imagenet, Two GPUs were used for the implementation
o Further theoretical improvements
◦ Rectified Linear Units (ReLU) instead of sigmoid or tanh
◦ Dropout
◦ Data augmentation
o In the 2013 Imagenet Workshop a legendary turmoil
◦ Blasted competitors by an impressive 16% top-5 error, Second best around 26%
◦ Most didn’t even think of NN as remotely competitive
o At the same time similar results in the speech recognition community
◦ One of G. Hinton students collaboration with Microsoft Research, improving state-of-the-art
by an impressive amount after years of incremental improvements [Hinton2012]

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 48
OPTIMIZATIONS - 48
Alexnet architecture

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 49
OPTIMIZATIONS - 49
Deep Learning Golden Era

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 50
OPTIMIZATIONS - 50
The today
o Deep Learning is almost everywhere
◦ Object classification
◦ Object detection, segmentation, pose estimation
◦ Image captioning, question answering
◦ Machine translation
◦ Speech recognition
◦ Robotics
o Some strongholds
◦ Action classification, action detection
◦ Object retrieval
◦ Object tracking

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 51
OPTIMIZATIONS - 51
The ILSVC Challenge over the last three years
CNN based, non-CNN based

Figures taken from Y. LeCun’s CVPR 2015 plenary talk

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 52
OPTIMIZATIONS - 52
2015 ILSVRC Challenge Alexnet, 2012

o Microsoft Research Asia won the competition with a 2014

legendary 150-layered network
o Almost superhuman accuracy: 3.5% error
◦ In 2016 <3% error
o In comparison in 2014 GoogLeNet had 22 layers

2015

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 53
OPTIMIZATIONS - 53
 Datasets of everything (captions,
So, why now? question-answering, …),
reinforcement learning, ???
1. Better hardware
??? Imagenet: 1,000 classes
from real images,
Object recognition with CNN 1,000,000 images

OCR with CNN Bank cheques

Backpropagation

Perceptron Parity, negation problems

Mark I Perceptron
Potentiometers
implement perceptron
2. Bigger data weights

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 55
OPTIMIZATIONS - 55
So, why now? (2)
1. Better hardware
2. Bigger data
3. Better regularization methods, such as dropout
4. Better optimization methods, such as Adam, batch normalization

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 56
OPTIMIZATIONS - 56
 Deep Learning:
The What and Why

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 57
Long story short
o A family of parametric, non-linear and hierarchical representation learning
functions, which are massively optimized with stochastic gradient descent
to encode domain knowledge, i.e. domain invariances, stationarity.
o 𝑎𝐿 𝑥; 𝜃1,…,L = ℎ𝐿 (ℎ𝐿−1 … ℎ1 𝑥, θ1 , θ𝐿−1 , θ𝐿 )
◦ 𝑥:input, θ𝑙 : parameters for layer l, 𝑎𝑙 = ℎ𝑙 (𝑥, θ𝑙 ): (non-)linear function
o Given training corpus {𝑋, 𝑌} find optimal parameters

θ∗ ← arg min𝜃 ෍ ℓ(𝑦, 𝑎𝐿 𝑥; 𝜃1,…,L )

(𝑥,𝑦)⊆(𝑋,𝑌)

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 58
OPTIMIZATIONS - 58
Learning Representations & Features
o Traditional pattern recognition

Hand-crafted Separate Trainable “Lemur”

Feature Extractor Classifier

o End-to-end learning  Features are also learned from data

Trainable “Lemur”
Trainable Classifier
Feature Extractor

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 59
OPTIMIZATIONS - 59
Non-separability of linear machines
o 𝑋 = 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∈ ℛ 𝑑

linear separability
Probability of
o Given the 𝑛 points there are in
total 2𝑛 dichotomies
o Only about 𝑑 are linearly
separable P=N #samples

o With 𝑛 > 𝑑 the probability 𝑋 is

linearly separable converges to 0
very fast
o The chances that a dichotomy is
linearly separable is very small

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 60
OPTIMIZATIONS - 60
Non-linearizing linear machines
o Most data distributions and tasks are non-linear
o A linear assumption is often convenient, but not necessarily truthful
o Problem: How to get non-linear machines without too much effort?
o Solution: Make features non-linear
o What is a good non-linear feature?
◦ Non-linear kernels, e.g., polynomial, RBF, etc
◦ Explicit design of features (SIFT, HOG)?

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 61
OPTIMIZATIONS - 61
Good features
o Invariant
◦ But not too invariant
o Repeatable
◦ But not bursty
o Discriminative
◦ But not too class-specific
o Robust
◦ But sensitive enough

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 62
OPTIMIZATIONS - 62
How to get good features?
o High-dimensional data (e.g. faces) lie in lower dimensional manifolds
◦ Goal: discover these lower dimensional manifolds
◦ These manifolds are most probably highly non-linear
o Hypothesis (1): Compute the coordinates of the input (e.g. a face image)
to this non-linear manifold  data become separable
o Hypothesis (2): Semantically similar things lie closer together than
semantically dissimilar things

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 63
OPTIMIZATIONS - 63
Feature manifold example
o Raw data live in huge dimensionalities
o Semantically meaningful raw data prefer lower dimensional manifolds
◦ Which still live in the same huge dimensionalities
o Can we discover this manifold to embed our data on?
Dimension 1

Dimension 2

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 64
OPTIMIZATIONS - 64
The digits manifolds
o There are good features and bad features, good manifold representations
and bad manifold representations
o 28 pixels x 28 pixels = 784 dimensions
PCA manifold
t-SNE manifold
(Two eigenvectors)

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 65
OPTIMIZATIONS - 65
End-to-end learning of feature hierarchies
o A pipeline of successive modules
o Each module’s output is the input for the next module
o Modules produce features of higher and higher abstractions
◦ Initial modules capture low-level features (e.g. edges or corners)
◦ Middle modules capture mid-level features (e.g. circles, squares, textures)
◦ Last modules capture high level, class specific features (e.g. face detector)
o Preferably, input as raw as possible
◦ Pixels for computer vision, words for NLP

Initial Middle Last

“Lemur”
modules modules modules

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 66
OPTIMIZATIONS - 66
Why learn the features?
o Manually designed features
◦ Often take a lot of time to come up with and implement
◦ Often take a lot of time to validate
◦ Often they are incomplete, as one cannot know if they
are optimal for the task
o Learned features
◦ Are easy to adapt
◦ Very compact and specific to the task at hand
◦ Given a basic architecture in mind, it is relatively easy and
fast to optimize
o Time spent for designing features now spent for
designing architectures

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 67
OPTIMIZATIONS - 67
Types of learning
Is this a dog or a cat?
o Supervised learning
◦ (Convolutional) neural networks

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 68
OPTIMIZATIONS - 68
Types of learning
Reconstruct this image
o Supervised learning
◦ (Convolutional) neural networks
o Unsupervised learning
◦ Autoencoders, layer-by-layer training

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 69
OPTIMIZATIONS - 69
Types of learning
o Supervised learning
◦ (Convolutional) neural networks
o Unsupervised learning
◦ Autoencoders, layer-by-layer training
o Self-supervised learning
◦ A mix of supervised and unsupervised learning

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 70
OPTIMIZATIONS - 70
Types of learning
o Supervised learning
◦ (Convolutional) neural networks
o Unsupervised learning
◦ Autoencoders, layer-by-layer training
o Self-supervised learning
◦ A mix of supervised and unsupervised learning
o Reinforcement learning
◦ Learn from noisy, delayed rewards from your environment
◦ Perform actions in your environment, so as to make decisions what data to collect

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 71
OPTIMIZATIONS - 71
Deep architectures
o Feedforward
◦ (Convolutional) neural networks
o Feedback
◦ Deconvolutional networks
o Bi-directional
◦ Deep Boltzmann Machines, stacked autoencoders
o Sequence based
◦ RNNs, LSTMs

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 72
OPTIMIZATIONS - 72
Convolutional networks in a nutshell
Dog or Cat?
Output layers

Is this a dog or a cat?

Hidden layers

Input layer

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 73
OPTIMIZATIONS - 73
Recurrent networks in a nutshell
Output: “To” “be” “or” “not” …

Output t

Previous Previous Previous …

memory memory memory
Input t
LSTM LSTM LSTM LSTM LSTM

Memory from
previous state
Current input Current input Current input Current input
t=0 t=1 t=2 T=3 …

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 74
OPTIMIZATIONS - 74
Deconvolutional networks

Convolutional network Deconvolutional network

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 75
OPTIMIZATIONS - 75
Autoencoders in a nutshell

Encoding Decoding

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 76
OPTIMIZATIONS - 76
 Philosophy of
the course

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 77
The bad news 
o We only have 2 months = 14 lectures
o Lots of material to cover
o Hence, lots of assignments that you have to implement
◦ Basic neural networks, learning Tensorflow, learning to program on a server, advanced
optimization techniques, convolutional neural networks, recurrent neural networks,
unsupervised learning
o This course is hard

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 78
OPTIMIZATIONS - 78
The good news 
o We are here to help
◦ Kirill, Berkay and Patrick have done some excellent work and we are all ready here to
help you with the course
o We have agreed with SURF SARA to give you access to the Dutch
Supercomputer Cartesius with a bunch of (very) expensive GPUs
◦ You should have no problem with resources
◦ You get to know how it is to do real programming on a server
o You’ll get to know some of the hottest stuff in AI today

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 79
OPTIMIZATIONS - 79
The good news 
o You’ll get to know some of the hottest stuff in AI today
◦ in academia

NIPS CVPR

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 80
OPTIMIZATIONS - 80
The good news 
o You will get to know some of the hottest stuff in AI today
◦ in academia & in industry

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 81
OPTIMIZATIONS - 81
The even better news 
o In the end of the course we will give a few MSc Thesis Projects in
collaboration with Qualcomm/QUVA Lab
o Students will become interns in the QUVA lab and get paid during thesis
o Requirements
◦ Work hard enough and be motivated
◦ Have top performance in the class
◦ And interested in working with us
o Come and find me after the course finishes

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 82
OPTIMIZATIONS - 82
Code of conduct
o We encourage you to help each other
◦ 3 students with highest participation in Q&A in Piazza get +1 grade
◦ Your grade depends on what you do, not what others do
o We encourage you to actively participate, give feedback etc
◦ It’s only the first real run of the course after all
o However, we do not tolerate blind copy
◦ Not from each other
◦ Not from the internet
◦ We have (deep) ways to check that

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 83
OPTIMIZATIONS - 83
 First lab
assignment

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 84
Deep Learning Framework
o Tensorflow [https://www.tensorflow.org/]
o Relies on Python
o Very good documentation
o Lots of code
◦ You can get inspired but not copy, we have ways to check that

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 85
OPTIMIZATIONS - 85
Content & Goal
o 1 hour presentation from SURF SARA on how to use their facilities
o Multi-layer perceptrons
o Solve a neural network in pen and paper
o Basic hyper-parameter tuning
o Your first neural network classifier

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 86
OPTIMIZATIONS - 86
Some practical information
o We organized 2 sessions for you, so that you can all comfortably follow
the presentation from SURF SARA
o 11.00-13.00
◦ SP B0.201
◦ Names: A-M
o 13.00-15.00
◦ SP D1.115
◦ Names N-Z

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 87
OPTIMIZATIONS - 87
 o A brief history of neural networks and deep
learning

Summary o What is deep learning and why is it happening

now?
o What types of deep learning exist?
o Demos and tasks where deep learning is
currently the preferred choice of models

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 88
Reading material & references
o http://www.deeplearningbook.org/
◦ Chapter 1: Introduction, p.1-p.28

UVADEEP
UVA DEEPLEARNING
LEARNINGCOURSE
COURSE– –EFSTRATIOS
EFSTRATIOSGAVVES
GAVVES INTRODUCTION
DEEPER TO DEEP
INTO LEARNING
DEEP AND
LEARNING NEURAL
AND NETWORKS- 89
OPTIMIZATIONS - 89
 o Learn how to describe neural networks as a
pipeline of layers and modules

Next lecture o Learn how to build your own modules

o Learn how to optimize modular neural networks
efficiently in theory and in practice
o And some theory

UVA DEEP LEARNING COURSE

EFSTRATIOS GAVVES
INTRODUCTION TO DEEP LEARNING AND NEURAL NETWORKS - 90