This document summarizes the first lecture of the course DD2437 - Artificial Neural Networks and Deep Architectures. It provides an overview of the course, including a brief history of ANNs, the course outline and learning activities, and an introduction to fundamental concepts like nodes, activation functions, learning rules, network architecture and data in ANNs.
This document summarizes the first lecture of the course DD2437 - Artificial Neural Networks and Deep Architectures. It provides an overview of the course, including a brief history of ANNs, the course outline and learning activities, and an introduction to fundamental concepts like nodes, activation functions, learning rules, network architecture and data in ANNs.
This document summarizes the first lecture of the course DD2437 - Artificial Neural Networks and Deep Architectures. It provides an overview of the course, including a brief history of ANNs, the course outline and learning activities, and an introduction to fundamental concepts like nodes, activation functions, learning rules, network architecture and data in ANNs.
This document summarizes the first lecture of the course DD2437 - Artificial Neural Networks and Deep Architectures. It provides an overview of the course, including a brief history of ANNs, the course outline and learning activities, and an introduction to fundamental concepts like nodes, activation functions, learning rules, network architecture and data in ANNs.
• Introduction to ANNs Fundamental characteristics Brain inspirations Current developments – theory and applications
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • McCulloch and Pitts • Donald Hebb “The Organization of Behaviour”
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • McCulloch and Pitts • Donald Hebb “The Organization of Behaviour” • ADALINE for binary patterns by Widrow and Hoff
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • McCulloch and Pitts • Donald Hebb “The Organization of Behaviour” • ADALINE for binary patterns by Widrow and Hoff • Rosenblatt’s perceptron
After Rosenblatt (1958)
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • McCulloch and Pitts • Donald Hebb “The Organization of Behaviour” • ADALINE for binary patterns by Widrow and Hoff • Rosenblatt’s perceptron • Marvin Minsky and Seymour Papert's criticism (1969) • first AI winter: fears and outrageous claims led to lower interest and poor funding
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • Hopfield’s impact
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • Hopfield’s impact • Renewed enthusiasm in the 1980s and 1990s (backpropagation, 1986)
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Brief history of ANNs • Hopfield’s impact • Renewed enthusiasm in the 1980s and 1990s (backpropagation, 1986) • interest swifts towards more mathematically rigorous statistical learning theory • another revival of interest in deep neural networks
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
A wide scope of applications • Pattern recognition: classification and clustering • General interpolation problems (generalisation) • Data representations, coding and compression • Signal processing • Time series prediction • System identification • Decision support (e.g. medical or industrial diagnostics) • Memory storage, modelling • Optimisation, combinatorial problems • etc……. KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Course outline • Learning activities 11 lectures + 1 summary and Q&A lecture + 1 reserve slot (Oct 10) 10 “lab” sessions (evaluation and support) 4 lab assignments (in groups of 3 students)
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Course outline • Learning activities 11 lectures + 1 summary and Q&A lecture + 1 reserve slot (Oct 10) 10 “lab” sessions (evaluation and support) 4 lab assignments (in groups of 3 students)
• Assessment, grading 4 mandatory lab assignments with bonus point deadlines (check in Canvas: under Lab Review as well as Content, assessment) lab review (redovisning) and a short report to submit in Canvas (P/F) written exam: A‐F
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Lab review • be prepared for short 10‐min demonstration of the lab results (in groups) and 2‐3 min questions, we do not tend to inspect your code • time is short so planning and preparation are critical – focus on main points, key questions – what is the lab about? what are the assumptions? share your insights and formulate clear conclusions what did you learn? are there any open questions left?
• you are in charge of the lab review but we are likely to ask questions • please ensure the entire group is involved (time sharing) • please demonstrate in the form of a report with clear & clean figures KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Lab report • a concise document to be uploaded to Canvas as a pdf file before your lab demonstration (bring the report with you) • a typical layout involves (for each lab there is a template) clear header (lab title, course signature, list of authors) aim/scope of the lab assignment, assumptions, tools used main results/findings supported with figures or tables (with captions, and always referred to in the main text) along with short commentary, your interpretations reflections, open questions and conclusions (to the point) no code needed and please do not add any appendices – there is an upper limit for the number of pages, e.g. 6 for lab 1.
• follow instructions on Canvas KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Course content Lectures L1: Course introduction and fundamental concepts
L2: From perceptron learning rule to backpropagation in feedforward networks – supervised learning
L3: Generalization, regularization, model selection and validation
L4: Practical aspects of ANN approaches to pattern recognition problems
L5: Radial basis function networks and introduction to unsupervised learning
L6: Self‐organising maps
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Course content Lectures L7: Temporal processing with ANNs: feedforward vs recurrent network architectures
L8: Hopfield networks and introduction to stochastic networks
L9: Deep learning fundamentals: general philosophy and a review of deep architectures
L10: Representation learning and deep generative models
L11: Deep neural networks: practicalities, challenges and current trends
L12: Course summary, old exam questions, Q&A
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Fundamental characteristics • nodes, units • activation function • learning rule • topology, network architecture • data
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Inspirations from biology
Multitude of neuron types in the brain
electrically excitable cells for processing and transmitting information (ca. 100 billion in the human brain)
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Inspirations from biology
electrically excitable cells for processing and transmitting information (ca. 100 billion in the human brain)
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Inspirations from biology – action potential (firing)
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Inspirations from biology – action potential (firing)
• commonly referred to as “spike” (nerve impulse) • threshold phenomenon accounts for “all‐or‐ nothing” paradigm • spike (action potential of fixed amplitude) travels along the axon
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Inspirations from biology – action potential (firing)
• commonly referred to as “spike” (nerve impulse) • threshold phenomenon accounts for “all‐or‐ nothing” paradigm • spike (action potential of fixed amplitude) travels along the axon • when spike reaches synaptic terminal, it contributes to post‐synaptic potentials
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Inspirations from biology – action potential (firing) 0 Level of depolarization needed to trigger an action potential ‐ 70
Depolarization (Excitatory) 0 Level needed to trigger an action potential
‐ 70
Hyperpolarization (Inhibitory) KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Biological neurons and networks – computational aspects • numerous simple units (type of integrators) • binary communication – spike (“all‐or‐nothing”) • but analog transmission in synapse (connection weight) • many inputs, interconnect • summation (integration) of inputs
KTH DD2437 L1 Course intro and fundamental concepts • A historical note • Course outline • Introduction to ANNs
Biomimetic nature of ANNs Biological neurons and networks – computational aspects • local processing, learning (based on local information) • synaptic memory (memory stored in weights) • synaptic plasticity – weight adaptation according to learning rules (largely phenomenological) • weights can be positive and negative (but Dale’s law) • fast nature of parallel processing • tolerance to errors/noise in data, weights etc.
KTH DD2437 L1 Course intro and fundamental concepts
