Recurrent Neural Networks have shown to be very powerful models as they can propagate context over several time steps. Due to this they can be applied effectively for addressing several problems in Natural Language Processing, such as Language Modelling, Tagging problems, Speech Recognition etc. In this presentation we introduce the basic RNN model and discuss the vanishing gradient problem. We describe LSTM (Long Short Term Memory) and Gated Recurrent Units (GRU). We also discuss Bidirectional RNN with an example. RNN architectures can be considered as deep learning systems where the number of time steps can be considered as the depth of the network. It is also possible to build the RNN with multiple hidden layers, each having recurrent connections from the previous time steps that represent the abstraction both in time and space.
RNN AND LSTM
This document provides an overview of RNNs and LSTMs:
1. RNNs can process sequential data like time series data using internal hidden states.
2. LSTMs are a type of RNN that use memory cells to store information for long periods of time.
3. LSTMs have input, forget, and output gates that control information flow into and out of the memory cell.
Recurrent Neural Network
ACRRL
Applied Control & Robotics Research Laboratory of Shiraz University
Department of Power and Control Engineering, Shiraz University, Fars, Iran.
Mohammad Sabouri
https://sites.google.com/view/acrrl/
Recurrent neural networks (RNNs) are a type of artificial neural network that can process sequential data of varying lengths. Unlike traditional neural networks, RNNs maintain an internal state that allows them to exhibit dynamic temporal behavior. RNNs take the output from the previous step and feed it as input to the current step, making the network dependent on information from earlier steps. This makes RNNs well-suited for applications like text generation, machine translation, image captioning, and more. RNNs can remember information for long periods of time but are difficult to train due to issues like vanishing gradients.
1. Recurrent neural networks can model sequential data like time series by incorporating hidden state that has internal dynamics. This allows the model to store information for long periods of time.
2. Two key types of recurrent networks are linear dynamical systems and hidden Markov models. Long short-term memory networks were developed to address the problem of exploding or vanishing gradients in training traditional recurrent networks.
3. Recurrent networks can learn tasks like binary addition by recognizing patterns in the inputs over time rather than relying on fixed architectures like feedforward networks. They have been successfully applied to handwriting recognition.
Brief introduction on attention mechanism and its application in neural machine translation, especially in transformer, where attention was used to remove RNNs completely from NMT.
This document discusses recurrent neural networks (RNNs) and their applications. It begins by explaining that RNNs can process input sequences of arbitrary lengths, unlike other neural networks. It then provides examples of RNN applications, such as predicting time series data, autonomous driving, natural language processing, and music generation. The document goes on to describe the fundamental concepts of RNNs, including recurrent neurons, memory cells, and different types of RNN architectures for processing input/output sequences. It concludes by demonstrating how to implement basic RNNs using TensorFlow's static_rnn function.
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...Simplilearn
This Deep Learning Presentation will help you in understanding what is Deep learning, why do we need Deep learning, applications of Deep Learning along with a detailed explanation on Neural Networks and how these Neural Networks work. Deep learning is inspired by the integral function of the human brain specific to artificial neural networks. These networks, which represent the decision-making process of the brain, use complex algorithms that process data in a non-linear way, learning in an unsupervised manner to make choices based on the input. This Deep Learning tutorial is ideal for professionals with beginners to intermediate levels of experience. Now, let us dive deep into this topic and understand what Deep learning actually is.
Below topics are explained in this Deep Learning Presentation:
1. What is Deep Learning?
2. Why do we need Deep Learning?
3. Applications of Deep Learning
4. What is Neural Network?
5. Activation Functions
6. Working of Neural Network
Simplilearn’s Deep Learning course will transform you into an expert in deep learning techniques using TensorFlow, the open-source software library designed to conduct machine learning & deep neural network research. With our deep learning course, you’ll master deep learning and TensorFlow concepts, learn to implement algorithms, build artificial neural networks and traverse layers of data abstraction to understand the power of data and prepare you for your new role as deep learning scientist.
Why Deep Learning?
It is one of the most popular software platforms used for deep learning and contains powerful tools to help you build and implement artificial neural networks.
Advancements in deep learning are being seen in smartphone applications, creating efficiencies in the power grid, driving advancements in healthcare, improving agricultural yields, and helping us find solutions to climate change. With this Tensorflow course, you’ll build expertise in deep learning models, learn to operate TensorFlow to manage neural networks and interpret the results.
You can gain in-depth knowledge of Deep Learning by taking our Deep Learning certification training course. With Simplilearn’s Deep Learning course, you will prepare for a career as a Deep Learning engineer as you master concepts and techniques including supervised and unsupervised learning, mathematical and heuristic aspects, and hands-on modeling to develop algorithms.
There is booming demand for skilled deep learning engineers across a wide range of industries, making this deep learning course with TensorFlow training well-suited for professionals at the intermediate to advanced level of experience. We recommend this deep learning online course particularly for the following professionals:
1. Software engineers
2. Data scientists
3. Data analysts
4. Statisticians with an interest in deep learning
The document discusses recurrent neural networks (RNNs) and long short-term memory (LSTM) networks. It provides details on the architecture of RNNs including forward and back propagation. LSTMs are described as a type of RNN that can learn long-term dependencies using forget, input and output gates to control the cell state. Examples of applications for RNNs and LSTMs include language modeling, machine translation, speech recognition, and generating image descriptions.
Basics covered regarding Natural Language Processing, How ANN transformed to RNN, Architectures of vanila RNN, LSTM and GRU and few preprocessing techniques
Recurrent Neural Networks are popular Deep Learning models that have shown great promise to achieve state-of-the-art results in many tasks like Computer Vision, NLP, Finance and much more. Although being models proposed several years ago, RNN have gained popularity recently. In this talk, we will review how these models evolved over the years, dissection of RNN, current applications and its future.
In machine learning, a convolutional neural network is a class of deep, feed-forward artificial neural networks that have successfully been applied fpr analyzing visual imagery.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.
Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Part 2 of the Deep Learning Fundamentals Series, this session discusses Tuning Training (including hyperparameters, overfitting/underfitting), Training Algorithms (including different learning rates, backpropagation), Optimization (including stochastic gradient descent, momentum, Nesterov Accelerated Gradient, RMSprop, Adaptive algorithms - Adam, Adadelta, etc.), and a primer on Convolutional Neural Networks. The demos included in these slides are running on Keras with TensorFlow backend on Databricks.
This Edureka Recurrent Neural Networks tutorial will help you in understanding why we need Recurrent Neural Networks (RNN) and what exactly it is. It also explains few issues with training a Recurrent Neural Network and how to overcome those challenges using LSTMs. The last section includes a use-case of LSTM to predict the next word using a sample short story
Below are the topics covered in this tutorial:
1. Why Not Feedforward Networks?
2. What Are Recurrent Neural Networks?
3. Training A Recurrent Neural Network
4. Issues With Recurrent Neural Networks - Vanishing And Exploding Gradient
5. Long Short-Term Memory Networks (LSTMs)
6. LSTM Use-Case
It’s long ago, approx. 30 years, since AI was not only a topic for Science-Fiction writers, but also a major research field surrounded with huge hopes and investments. But the over-inflated expectations ended in a subsequent crash and followed by a period of absent funding and interest – the so-called AI winter. However, the last 3 years changed everything – again. Deep learning, a machine learning technique inspired by the human brain, successfully crushed one benchmark after another and tech companies, like Google, Facebook and Microsoft, started to invest billions in AI research. “The pace of progress in artificial general intelligence is incredible fast” (Elon Musk – CEO Tesla & SpaceX) leading to an AI that “would be either the best or the worst thing ever to happen to humanity” (Stephen Hawking – Physicist).
What sparked this new Hype? How is Deep Learning different from previous approaches? Are the advancing AI technologies really a threat for humanity? Let’s look behind the curtain and unravel the reality. This talk will explore why Sundar Pichai (CEO Google) recently announced that “machine learning is a core transformative way by which Google is rethinking everything they are doing” and explain why "Deep Learning is probably one of the most exciting things that is happening in the computer industry” (Jen-Hsun Huang – CEO NVIDIA).
Either a new AI “winter is coming” (Ned Stark – House Stark) or this new wave of innovation might turn out as the “last invention humans ever need to make” (Nick Bostrom – AI Philosoph). Or maybe it’s just another great technology helping humans to achieve more.
Introduction to Transformers for NLP - Olga PetrovaAlexey Grigorev
Olga Petrova gives an introduction to transformers for natural language processing (NLP). She begins with an overview of representing words using tokenization, word embeddings, and one-hot encodings. Recurrent neural networks (RNNs) are discussed as they are important for modeling sequential data like text, but they struggle with long-term dependencies. Attention mechanisms were developed to address this by allowing the model to focus on relevant parts of the input. Transformers use self-attention and have achieved state-of-the-art results in many NLP tasks. Bidirectional Encoder Representations from Transformers (BERT) provides contextualized word embeddings trained on large corpora.
A comprehensive tutorial on Convolutional Neural Networks (CNN) which talks about the motivation behind CNNs and Deep Learning in general, followed by a description of the various components involved in a typical CNN layer. It explains the theory involved with the different variants used in practice and also, gives a big picture of the whole network by putting everything together.
Next, there's a discussion of the various state-of-the-art frameworks being used to implement CNNs to tackle real-world classification and regression problems.
Finally, the implementation of the CNNs is demonstrated by implementing the paper 'Age ang Gender Classification Using Convolutional Neural Networks' by Hassner (2015).
The document discusses Long Short Term Memory (LSTM) networks, which are a type of recurrent neural network capable of learning long-term dependencies. It explains that unlike standard RNNs, LSTMs use forget, input, and output gates to control the flow of information into and out of the cell state, allowing them to better capture long-range temporal dependencies in sequential data like text, audio, and time-series data. The document provides details on how LSTM gates work and how LSTMs can be used for applications involving sequential data like machine translation and question answering.
Artificial Intelligence, Machine Learning and Deep LearningSujit Pal
Slides for talk Abhishek Sharma and I gave at the Gennovation tech talks (https://gennovationtalks.com/) at Genesis. The talk was part of outreach for the Deep Learning Enthusiasts meetup group at San Francisco. My part of the talk is covered from slides 19-34.
Deep learning (also known as deep structured learning or hierarchical learning) is the application of artificial neural networks (ANNs) to learning tasks that contain more than one hidden layer. Deep learning is part of a broader family of machine learning methods based on learning data representations, as opposed to task-specific algorithms. Learning can be supervised, partially supervised or unsupervised.
Hardware Acceleration for Machine LearningCastLabKAIST
This document provides an overview of a lecture on hardware acceleration for machine learning. The lecture will cover deep neural network models like convolutional neural networks and recurrent neural networks. It will also discuss various hardware accelerators developed for machine learning, including those designed for mobile/edge and cloud computing environments. The instructor's background and the agenda topics are also outlined.
[Paper Reading] Attention is All You NeedDaiki Tanaka
The document summarizes the "Attention Is All You Need" paper, which introduced the Transformer model for natural language processing. The Transformer uses attention mechanisms rather than recurrent or convolutional layers, allowing for more parallelization. It achieved state-of-the-art results in machine translation tasks using techniques like multi-head attention, positional encoding, and beam search decoding. The paper demonstrated the Transformer's ability to draw global dependencies between input and output with constant computational complexity.
Neural machine translation by jointly learning to align and translate.pptxssuser2624f71
The document discusses machine translation techniques including rule-based machine translation (RBMT), statistical machine translation (SMT), and neural machine translation (NMT). It then focuses on neural network approaches, explaining recurrent neural networks (RNNs) and variants like long short-term memory (LSTM) and gated recurrent units (GRU). Finally, it presents a new methodology called RNNsearch that uses an attention mechanism to overcome limitations of fixed-length encodings in encoder-decoder NMT models, showing improved translation performance especially on longer sentences.
This presentation is Part 2 of my September Lisp NYC presentation on Reinforcement Learning and Artificial Neural Nets. We will continue from where we left off by covering Convolutional Neural Nets (CNN) and Recurrent Neural Nets (RNN) in depth.
Time permitting I also plan on having a few slides on each of the following topics:
1. Generative Adversarial Networks (GANs)
2. Differentiable Neural Computers (DNCs)
3. Deep Reinforcement Learning (DRL)
Some code examples will be provided in Clojure.
After a very brief recap of Part 1 (ANN & RL), we will jump right into CNN and their appropriateness for image recognition. We will start by covering the convolution operator. We will then explain feature maps and pooling operations and then explain the LeNet 5 architecture. The MNIST data will be used to illustrate a fully functioning CNN.
Next we cover Recurrent Neural Nets in depth and describe how they have been used in Natural Language Processing. We will explain why gated networks and LSTM are used in practice.
Please note that some exposure or familiarity with Gradient Descent and Backpropagation will be assumed. These are covered in the first part of the talk for which both video and slides are available online.
A lot of material will be drawn from the new Deep Learning book by Goodfellow & Bengio as well as Michael Nielsen's online book on Neural Networks and Deep Learning as well several other online resources.
Bio
Pierre de Lacaze has over 20 years industry experience with AI and Lisp based technologies. He holds a Bachelor of Science in Applied Mathematics and a Master’s Degree in Computer Science.
https://www.linkedin.com/in/pierre-de-lacaze-b11026b/
The document provides an overview of deep learning concepts and techniques for natural language processing tasks. It includes the following:
1. A schedule for a deep learning workshop covering fundamentals of deep learning for machine translation, word embeddings, neural language models, and neural machine translation.
2. Descriptions of neural networks, activation functions, backpropagation, and word embeddings.
3. Details about feedforward neural network language models, recurrent neural network language models, and how they are applied to tasks like language modeling and machine translation.
4. An explanation of attention-based encoder-decoder models for neural machine translation.
Introduction to computer vision with Convoluted Neural NetworksMarcinJedyk
Introduction to computer vision with Convoluted Neural Networks - going over history of CNNs, describing basic concepts such as convolution and discussing applications of computer vision and image recognition technologies
This document provides an introduction to deep learning. It discusses key concepts such as neural network layers that process input tensors, common layer types like convolutional and recurrent layers, and how networks are trained using stochastic gradient descent. Examples of deep learning applications that have achieved near-human level performance are also presented, such as image classification and speech recognition. The document then focuses on convolutional neural networks, covering concepts like convolution operations, spatial hierarchies, and max pooling. It concludes with a demonstration of digit and X-ray image classification using Keras and techniques for dealing with overfitting like dropout and data augmentation.
This document provides an introduction to computer vision with convoluted neural networks. It discusses what computer vision aims to address, provides a brief overview of neural networks and their basic building blocks. It then covers the history and evolution of convolutional neural networks, how and why they work on digital images, their limitations, and applications like object detection. Examples are provided of early CNNs from the 1980s and 1990s and recent advancements through the 2010s that improved accuracy, including deeper networks, inception modules, residual connections, and efforts to increase performance like MobileNets. Training deep CNNs requires large datasets and may take weeks, but pre-trained networks can be fine-tuned for new tasks.
This document provides an overview of convolutional neural networks (CNNs) and describes a research study that used a two-dimensional heterogeneous CNN (2D-hetero CNN) for mobile health analytics. The study developed a 2D-hetero CNN model to assess fall risk using motion sensor data from 5 sensor locations on participants. The model extracts low-level local features using convolutional layers and integrates them into high-level global features to classify fall risk. The 2D-hetero CNN was evaluated against feature-based approaches and other CNN architectures and performed ablation analysis.
This document provides an overview and agenda for a Deep Learning with MXNet workshop. It begins with background on deep learning basics like biological and artificial neurons. It then introduces Apache MXNet and discusses its key features like scalability, efficiency, and programming models. The remainder of the document provides hands-on examples for attendees to train their first neural network using MXNet, including linear regression, MNIST digit classification using a multilayer perceptron, and convolutional neural networks.
Deep Learning Sample Class (Jon Lederman)Jon Lederman
Deep learning uses neural networks that can learn their own features from data. The document discusses the history and limitations of early neural networks like perceptrons that used hand-engineered features. Modern deep learning overcomes these limitations by using hierarchical neural networks that can learn increasingly complex features from raw data through backpropagation and gradient descent. Deep learning networks represent features using tensors, or multidimensional arrays, that are learned from data through training examples.
Artificial Neural Networks have been very successfully used in several machine learning applications. They are often the building blocks when building deep learning systems. We discuss the hypothesis, training with backpropagation, update methods, regularization techniques.
Generative Adversarial Networks : Basic architecture and variantsananth
In this presentation we review the fundamentals behind GANs and look at different variants. We quickly review the theory such as the cost functions, training procedure, challenges and go on to look at variants such as CycleGAN, SAGAN etc.
Convolutional Neural Networks : Popular Architecturesananth
In this presentation we look at some of the popular architectures, such as ResNet, that have been successfully used for a variety of applications. Starting from the AlexNet and VGG that showed that the deep learning architectures can deliver unprecedented accuracies for Image classification and localization tasks, we review other recent architectures such as ResNet, GoogleNet (Inception) and the more recent SENet that have won ImageNet competitions.
In this presentation we discuss the convolution operation, the architecture of a convolution neural network, different layers such as pooling etc. This presentation draws heavily from A Karpathy's Stanford Course CS 231n
Artificial Intelligence Course: Linear models ananth
In this presentation we present the linear models: Regression and Classification. We illustrate with several examples. Concepts such as underfitting (Bias) and overfitting (Variance) are presented. Linear models can be used as stand alone classifiers for simple cases and they are essential building blocks as a part of larger deep learning networks
Naive Bayes Classifier is a machine learning technique that is exceedingly useful to address several classification problems. It is often used as a baseline classifier to benchmark results. It is also used as a standalone classifier for tasks such as spam filtering where the naive assumption (conditional independence) made by the classifier seem reasonable. In this presentation we discuss the mathematical basis for the Naive Bayes and illustrate with examples
Mathematical Background for Artificial Intelligenceananth
Mathematical background is essential for understanding and developing AI and Machine Learning applications. In this presentation we give a brief tutorial that encompasses basic probability theory, distributions, mixture models, anomaly detection, graphical representations such as Bayesian Networks, etc.
This presentation discusses the state space problem formulation and different search techniques to solve these. Techniques such as Breadth First, Depth First, Uniform Cost and A star algorithms are covered with examples. We also discuss where such techniques are useful and the limitations.
This is the first lecture of the AI course offered by me at PES University, Bangalore. In this presentation we discuss the different definitions of AI, the notion of Intelligent Agents, distinguish an AI program from a complex program such as those that solve complex calculus problems (see the integration example) and look at the role of Machine Learning and Deep Learning in the context of AI. We also go over the course scope and logistics.
In this presentation we discuss several concepts that include Word Representation using SVD as well as neural networks based techniques. In addition we also cover core concepts such as cosine similarity, atomic and distributed representations.
Deep Learning techniques have enabled exciting novel applications. Recent advances hold lot of promise for speech based applications that include synthesis and recognition. This slideset is a brief overview that presents a few architectures that are the state of the art in contemporary speech research. These slides are brief because most concepts/details were covered using the blackboard in a classroom setting. These slides are meant to supplement the lecture.
Overview of TensorFlow For Natural Language Processingananth
TensorFlow open sourced recently by Google is one of the key frameworks that support development of deep learning architectures. In this slideset, part 1, we get started with a few basic primitives of TensorFlow. We will also discuss when and when not to use TensorFlow.
Convolutional neural networks (CNNs) are better suited than traditional neural networks for processing image data due to properties of images. CNNs apply filters with local receptive fields and shared weights across the input, allowing them to detect features regardless of position. A CNN architecture consists of convolutional layers that apply filters, and pooling layers for downsampling. This reduces parameters and allows the network to learn representations of the input with minimal feature engineering.
This presentation discusses decision trees as a machine learning technique. This introduces the problem with several examples: cricket player selection, medical C-Section diagnosis and Mobile Phone price predictor. It discusses the ID3 algorithm and discusses how the decision tree is induced. The definition and use of the concepts such as Entropy, Information Gain are discussed.
This presentation is a part of ML Course and this deals with some of the basic concepts such as different types of learning, definitions of classification and regression, decision surfaces etc. This slide set also outlines the Perceptron Learning algorithm as a starter to other complex models to follow in the rest of the course.
This is the first lecture on Applied Machine Learning. The course focuses on the emerging and modern aspects of this subject such as Deep Learning, Recurrent and Recursive Neural Networks (RNN), Long Short Term Memory (LSTM), Convolution Neural Networks (CNN), Hidden Markov Models (HMM). It deals with several application areas such as Natural Language Processing, Image Understanding etc. This presentation provides the landscape.
In this presentation we discuss the hypothesis of MaxEnt models, describe the role of feature functions and their applications to Natural Language Processing (NLP). The training of the classifier is discussed in a later presentation.
In this presentation we describe the formulation of the HMM model as consisting of states that are hidden that generate the observables. We introduce the 3 basic problems: Finding the probability of a sequence of observation given the model, the decoding problem of finding the hidden states given the observations and the model and the training problem of determining the model parameters that generate the given observations. We discuss the Forward, Backward, Viterbi and Forward-Backward algorithms.
Discusses the concept of Language Models in Natural Language Processing. The n-gram models, markov chains are discussed. Smoothing techniques such as add-1 smoothing, interpolation and discounting methods are addressed.
This document provides an introduction to natural language processing and word representation techniques. It discusses how words can take on different meanings based on context and how words may be related in some dimensions but not others. It also outlines criteria for a good word representation system, such as capturing different semantic interpretations of words and enabling similarity comparisons. The document then reviews different representation approaches like discrete, co-occurrence matrices, and word2vec, noting issues with earlier approaches and how word2vec uses skip-gram models and sliding windows to learn word vectors in a low-dimensional space.
Securiport Gambia is a civil aviation and intelligent immigration solutions provider founded in 2001. The company was created to address security needs unique to today’s age of advanced technology and security threats. Securiport Gambia partners with governments, coming alongside their border security to create and implement the right solutions.
Flame emission spectroscopy is an instrument used to determine concentration of metal ions in sample. Flame provide energy for excitation atoms introduced into flame. It involve components like sample delivery system, burner, sample, mirror, slits, monochromator, filter, detector (photomultiplier tube and photo tube detector). There are many interference involved during analysis of sample like spectral interference, ionisation interference, chemical interference ect. It can be used for both quantitative and qualitative study, determine lead in petrol, determine alkali and alkaline earth metal, determine fertilizer requirement for soil.
Increase Quality with User Access Policies - July 2024Peter Caitens
⭐️ Increase Quality with User Access Policies ⭐️, presented by Peter Caitens and Adam Best of Salesforce. View the slides from this session to hear all about “User Access Policies” and how they can help you onboard users faster with greater quality.
TrustArc Webinar - Innovating with TRUSTe Responsible AI CertificationTrustArc
In a landmark year marked by significant AI advancements, it’s vital to prioritize transparency, accountability, and respect for privacy rights with your AI innovation.
Learn how to navigate the shifting AI landscape with our innovative solution TRUSTe Responsible AI Certification, the first AI certification designed for data protection and privacy. Crafted by a team with 10,000+ privacy certifications issued, this framework integrated industry standards and laws for responsible AI governance.
This webinar will review:
- How compliance can play a role in the development and deployment of AI systems
- How to model trust and transparency across products and services
- How to save time and work smarter in understanding regulatory obligations, including AI
- How to operationalize and deploy AI governance best practices in your organization
Project Delivery Methodology on a page with activities, deliverablesCLIVE MINCHIN
I've not found a 1 pager like this anywhere so I created it based on my experiences. This 1 pager details a waterfall style project methodology with defined phases, activities, deliverables, assumptions. There's nothing in here that conflicts with commonsense.
The Challenge of Interpretability in Generative AI Models.pdfSara Kroft
Navigating the intricacies of generative AI models reveals a pressing challenge: interpretability. Our blog delves into the complexities of understanding how these advanced models make decisions, shedding light on the mechanisms behind their outputs. Explore the latest research, practical implications, and ethical considerations, as we unravel the opaque processes that drive generative AI. Join us in this insightful journey to demystify the black box of artificial intelligence.
Dive into the complexities of generative AI with our blog on interpretability. Find out why making AI models understandable is key to trust and ethical use and discover current efforts to tackle this big challenge.
Connecting Attitudes and Social Influences with Designs for Usable Security a...Cori Faklaris
Many system designs for cybersecurity and privacy have failed to account for individual and social circumstances, leading people to use workarounds such as password reuse or account sharing that can lead to vulnerabilities. To address the problem, researchers are building new understandings of how individuals’ attitudes and behaviors are influenced by the people around them and by their relationship needs, so that designers can take these into account. In this talk, I will first share my research to connect people’s security attitudes and social influences with their security and privacy behaviors. As part of this, I will present the Security and Privacy Acceptance Framework (SPAF), which identifies Awareness, Motivation, and Ability as necessary for strengthening people’s acceptance of security and privacy practices. I then will present results from my project to trace where social influences can help overcome obstacles to adoption such as negative attitudes or inability to troubleshoot a password manager. I will conclude by discussing my current work to apply these insights to mitigating phishing in SMS text messages (“smishing”).
IVE 2024 Short Course - Lecture 2 - Fundamentals of PerceptionMark Billinghurst
Lecture 2 from the IVE 2024 Short Course on the Psychology of XR. This lecture covers some of the Fundamentals of Percetion and Psychology that relate to XR.
The lecture was given by Mark Billinghurst on July 15th 2024 at the University of South Australia.
Project management Course in Australia.pptxdeathreaper9
Project Management Course
Over the past few decades, organisations have discovered something incredible: the principles that lead to great success on large projects can be applied to projects of any size to achieve extraordinary success. As a result, many employees are expected to be familiar with project management techniques and how they apply them to projects.
https://projectmanagementcoursesonline.au/
How CXAI Toolkit uses RAG for Intelligent Q&AZilliz
Manasi will be talking about RAG and how CXAI Toolkit uses RAG for Intelligent Q&A. She will go over what sets CXAI Toolkit's Intelligent Q&A apart from other Q&A systems, and how our trusted AI layer keeps customer data safe. She will also share some current challenges being faced by the team.
Airports, banks, stock exchanges, and countless other critical operations got thrown into chaos!
In an unprecedented event, a recent CrowdStrike update had caused a global IT meltdown, leading to widespread Blue Screen of Death (BSOD) errors, and crippling 8.5 million Microsoft Windows systems.
What triggered this massive disruption? How did Microsoft step in to provide a lifeline? And what are the next steps for recovery?
Swipe to uncover the full story, including expert insights and recovery steps for those affected.
2. Objective
• Overview of Neural Networks
• Recurrent Neural Networks (RNN)
• Bidirectional Recurrent Neural Networks (BRNN)
• Differences between Recursive and Recurrent Neural Networks
• Challenges in implementing RNN: Vanishing Gradient Problem
• Gated Recurrent Units (GRUs)
• Long Short Term Memory (LSTM)
• Applications
3. References (Abridged list)
• Machine Learning, T Mitchell
• MOOC Courses offered by Prof Andrew
Ng, Prof Yaser Mustafa, Geoff Hinton
• CMU Videos Prof T Mitchell
• Alex Graves: Supervised Sequence
Labelling with Recurrent Neural Networks
• Andrej Karpathy’s Blogs
• Stanford course CS224d: Socher
• Recurrent Neural Networks based
Language Models: Mikolov etal
• Annotating Expressions of Opinions and
Emotions in Language: Wiebe etal
R Socher
4. Human Cognition
• Most Common human cognitive tasks (such as
understanding and speaking natural language,
recognizing objects etc) are highly non linear.
• Human cognition tasks are often hierarchical
• Though our brain receives low level sensory inputs as
impulses, we process the input as a whole, recognizing
several patterns as opposed to looking at micro level data
• Humans learn continuously often unaided or
unsupervised
• For the same object or pattern that we recognize we see
them in different perspectives.
• E.g. We may know “home” and “residence” as two separate
words but we can interpret them in different contexts. Home
maker versus residence address.
What does it take to build an autonomous car that can
drive itself in Bangalore traffic?
5. Quick recap of last lecture
• ML attempts to approximate real world
applications by mathematical models
• The underlying process behind the given real
world application (that we are trying to
model) is called the unknown target function
• Linear models approximate the real world
using a linear function.
• Most of the real world applications are non-
linear and are hierarchical
• Artificial Neural networks (ANN) are non
linear models and are effective for certain
class of applications.
• Each hidden layer represents a particular level
of abstraction
• ANNs are commonly trained using
backpropagation algorithms
• The model parameters are tunable knobs that
determine the output of the machine and
signify the degrees of freedom
• More the parameters, more easily we can fit
the training data but impacts the
generalization. Regularization keeps the
model parameters under check
• Traditional ANNs with a large number of
hidden layers are hard to train: Problems of
local minima and vanishing/exploding
gradients
• Deep learning techniques are breakthroughs
that enable realization of deep architectures
• Recurrent Neural Networks (RNN), Recursive
Neural Networks and Convolutional Neural
Networks are specializations of the ANN
architecture to handle different nature of
problems.
• For instance RNNs are effective for predicting time
series problems
• For a brief 5 slide refresher on DNNs see:
http://www.slideshare.net/ananth/deep-
learningprimer-7june2014
6. Non linear Models: Neural Networks
• Motivation
• A large number of classification tasks involve inherently highly non linear target functions – for example,
face recognition
• Though we can transform the input vector in to a non linear form and perform classification with linear
models, the model becomes very complex quickly.
• For example:
• Consider a 10 dimensional input vector that needs to be transformed in to a polynomial with degree 3. O(n3)
• Consider the problem of looking at the image of a building and identifying it (say: 100 by 100 pixels)
• Over fitting problems are common when we train more complex models
• Illustration (on black board)
• Boolean functions AND, OR can be effectively modelled by Linear Models
• A single logistic regression unit can’t model more complex Boolean functions such as XOR
• Cascading logistic regression units can classify complex Boolean target functions effectively
• It is shown that with 2 layers of logistic regression units, one can model many complex Boolean
expressions effectively
7. Neural Networks (Fig: courtesy R Socher)
Neural Networks can be built for different
input, output types.
- Outputs can be:
- Linear, single output (Linear)
- Linear, multiple outputs (Linear)
- Single output binary (Logistic)
- Multi output binary (Logistic)
- 1 of k Multinomial output (Softmax)
- Inputs can be:
- A scalar number
- Vector of Real numbers
- Vector of Binary
Goal of training: Given the training data (inputs, targets) and the
architecture, determine the model parameters.
Model Parameters for a 3 layer network:
- Weight matrix from input layer to the hidden (Wjk)
- Weight matrix from hidden layer to the output (Wkj)
- Bias terms for hidden layer
- Bias terms for output layer
Our strategy will be:
- Compute the error at the output
- Determine the contribution of each parameter to the error by
taking the differential of error wrt the parameter
- Update the parameter commensurate with the error it contributed.
8. Design Choices
• When building a neural network, the designer would choose the
following hyper parameters and non linearities based on the
application characteristics:
• Number of hidden layers
• Number of hidden units in each layer
• Learning rate
• Regularization coefft
• Number of outputs
• Type of output (linear, logistic, softmax)
• Choice of Non linearity at the output layer and hidden layer (See next slide)
• Input representation and dimensionality
10. Objective Functions and gradients (derivation of gradient on the board)
• Linear – Mean squared error
• 𝐸 𝑤 =
1
2𝑁 1
𝑁
(𝑡 𝑛 − 𝑦𝑛)2
• Logistic with binary classifications: Cross Entropy Error
• Logistic with k outputs: k > 2: Cross Entropy Error
• Softmax: 1 of K multinomial classification: Cross Entropy Error, minimize NLL
• In all the above cases we can show that the gradient is: (yk - tk) where yk is
the predicted output for the output unit k and tk is the corresponding target
11. High Level Backpropagation Algorithm
• Apply the input vector to the network and forward propagate. This
will yield the activations for hidden layer(s) and the output layer
• 𝑛𝑒𝑡𝑗 = 𝑖 𝑤𝑗𝑖 𝑧𝑖,
• 𝑧𝑗 = ℎ(𝑛𝑒𝑡𝑗) where h is your choice of non linearity. Usually it is sigmoid or
tanh. Rectified Linear Unit (RelU) is also used.
• Evaluate the error 𝛿 𝑘 for all the output units
𝛿 𝑘 = 𝑜 𝑘 − 𝑡 𝑘 where 𝑜 𝑘 is the output produced by the model and 𝑡 𝑘 is the
target provided in the training dataset
• Backpropagate the 𝛿’s to obtain 𝛿𝑗 for each hidden unit j
𝛿𝑗 = ℎ′
(𝑧𝑗) 𝑘 𝑤 𝑘𝑗 𝛿 𝑘
• Evaluate the required derivatives
𝜕𝐸
𝜕𝑊𝑗𝑖
= 𝛿𝑗 𝑧𝑖
13. RNN – Some toy applications to evaluate the system
• Often times some toy applications, even if they are contrived, serve
the following purposes:
• Test the correctness of the implementation of the model
• Compare the performance of the new model with respect to the old ones
• Example applications for verifying the performance of RNN:
• Arithmetic progression (will be demo’d now)
• Process an input of the form: an bj and return true if n = j
• Count the number of words in a sequence ignoring the words that are
enclosed in parenthesis
• Perform XOR of bits of a sequence up to a time step t
16. Training Algorithm (Fig: Xiodong He etal, Microsoft Research)
• Different training procedures
exist, we will use Back Propagation
Through Time (BPTT)
• Similar to standard
backpropagation, BPTT involves
using chain rule repeatedly and
bakpropagating the deltas
• However one key subtlety is that,
for RNNs, the cost function
depends on the activation of
hidden layer not only through its
influence on output layer but also
through its influence on hidden
layer of the next time step
17. A sketch of implementation – Forward pass
Forward Propagation – Key steps
for t from 1 to T
1. Compute hidden activations of time t with current input and hidden activations for (t-1)
2. For all j in the output units compute the netj (dot product of WS with ht )
3. Apply the softmax function on the netj and get the probability distribution for time t
18. A sketch of implementation – Backpropagation
Backpropagation for RNN – Key steps
for t from T down to 1
1. compute the delta at the
output (dy)
2. Compute Δwji where w is the
(softmax) weight matrix WS
3. Determine the bias terms
4. Backpropagate and compute
delta for hidden layer (dhraw)
5. Compute the updates to
weight matrix Whh and Whx
6. Perform BPTT by computing
the error to be propagated to
the previous layer (dbnext).
19. Applications
• Language Model (Mikolov etal)
• Input at a time t is the corresponding word vector
• Output is the predicted next word
• Language translation
• Slot filling (see next slide)
• Character LM (Andrej Karpathy)
• Image captioning and description
• Speech recognition
• Question Answering Systems (We are doing a special topic project on this)
• Semantic Role Labeling (We are doing a special topic project on this)
• NER (demo done last week!)
• And many more sequence based applications
20. Semantic Slot Filling Application Example
Many problems in
Information extraction
require generating a data
structure from a natural
language input
One possible way to cast
this problem is to treat this
as a slot filling exercise.
This can be viewed as a
sequential tagging
problem and use an RNN
for tagging
21. Building an NER with RNN
• The traditional MEMM or CRF based NER design techniques require
domain expertise when designing the feature vector
• RNN based NER’s don’t need feature engineering and with some
minimum text preprocessing (such as removing infrequent words),
one can build an NER that provides comparable performance
• Steps:
• Preprocess the words: tokenization and some simple task dependent
preprocessing as needed
• Get word vectors (this helps reducing the dimensionality)
• Form the training dataset
• Train the NER
• Predict
22. Encoder Decoder Design
• Example: Machine Translation
• Use 2 RNN’s, one for encoding and the other decoding
• The activations of the final stage of the encoder is fed to the decoder
• This is useful when the output sequence is of variable length and if
the entire input sequence can be processed before generating the
output
24. Clipping
• Key Idea: Avoid the vanishing/exploding gradient problem by looking
at a threshold and clip the gradient to that threshold.
• While this is a simple workaround to address the issue, it is crude and
might hamper the performance
• Better solutions: LSTMs and its variants like GRUs (topic of next class!)
25. Bidirectional RNNs
• Key idea:
• Output at a step t not only depends on the past steps (t-1…t1) but also
depends on future steps (t+1, …T).
• The forward pass abstracts and summarizes the context in the forward
direction while the backward pass does the same from the reverse direction
• Examples: Fill in the blanks below
• I want ______ buy a good book _______ NLP
• I want ______ Mercedes
• Let’s illustrate bidirectional RNNs with an application example from:
Opinion Mining with Deep Recurrent Nets by Irsoy and Cardie 2014
26. Problem Statement: Ref Irsoy and Cardie 2014
• Given a sentence, classify each
word in to one of the tags: {O,
B-ESE, I-ESE, B-DSE, I-DSE}
• Definitions
• Direct Subjective Expressions
(DSE): explicit mentions of private
states or speech events
expressing private states
• Expressive Subjective Expressions
(ESE): Expressions that indicate
sentiment, emotion, etc., without
explicitly conveying them.
27. Bidirectional RNN Model
• Input: A sequence of words. At each
time step t a single token
(represented by its word vector) is
input to the RNN. (Black dots)
• Output: At each time step t one of
the possible tags from the tagset is
output by the RNN (Red dots)
• Memory: This is the hidden unit that
is computed from current word and
the past hidden values. It
summarizes the sentence up to that
time. (Orange dots)
29. Deep Bidirectional RNNs
• RNNs are deep networks with depth in
time.
• When unfolded, they are multi layer feed
forward neural networks, where there
are as many hidden layers as input
tokens.
• However, this doesn’t represent the
hierarchical processing of data across
time units as we still use same U, V, W
• A stacked deep learner supports
hierarchical computations, where each
hidden layer corresponds to a degree of
abstraction.
• Stacking a simple RNN on top of others
has the potential to perform hierarchical
computations moving over the time axis
30. Training the BRNN (ref: Alex Graves: Supervised Sequence Labelling with
Recurrent Neural Networks)
Forward Pass
for t = 1 to T do
Forward pass for the forward hidden layer, storing activations at each time step
for t = T to 1 do
Forward pass for the backward hidden layer, storing activations at each time step
for all t, in any order do
Forward pass for the output layer, using the stored activations from both hidden layers
Backward Pass
for all t, in any order do
Backward pass for the output layer, storing terms at each time step
for t = T to 1 do
BPTT backward pass for the forward hidden layer, using the stored terms from the output layer
for t = 1 to T do
BPTT backward pass for the backward hidden layer, using the stored terms from the output layer
31. Long Short Term Memory (LSTM): Motivation 1 of 2
• Consider the cases below, where a customer is interested in iPhone 6s plus and he needs to
gift it to his father on his birthday on Oct 2. He goes through a review that reads as below:
• Review 1: Apple has unveiled the iPhone 6s and iPhone 6s Plus - described by CEO Tim Cook as the "most
advanced phones ever" - at a special event in San Francisco on Wednesday. Pre-orders for the new iPhone
models begin this Saturday and they have a launch date (start shipping) in twelve countries on September
25. The price for the iPhone 6s and iPhone 6s Plus remain unchanged compared to their predecessors:
$649 for the 16GB iPhone 6s, $749 for the 64GB iPhone 6s and 16GB iPhone 6s Plus, $849 for 128GB
iPhone 6s and 64GB iPhone 6s Plus, and $949 for the 128GB iPhone 6s Plus (all US prices). There's no
word yet on India price or launch date
• How would we design a RNN that advises him: Buy/No Buy?
• Suppose the customer doesn’t have the time constraint as above but has a price constraint,
where his budget is around Rs 50K, what would be our decision?
• Suppose there is another review article that reads as below:
• Review 2: Priced at INR 75K for the low end model, Apple iPhone boasts of an ultra slim device with an
awesome camera. Apple’s CEO while showcasing the device at San Francisco, announced its availability on
12 countries including India. This is the best phone that one can flaunt if he can afford it!
32. LSTM Motivation 2 of 2
• Observations from the case studies:
• A product review has many sentences and the pieces of information that we may be interested for making our
buying decision is found at various places in the text.
• Certain aspects are “must have” for us that can’t be compromised. For instance if a customer needs an item
within a few days, he can’t wait for it indefinitely. Similarly if he has a budget constraint, he can’t buy the item
even if it is the best fit for his other requirements.
• If we find a sentence that implies that a must have feature can’t be met, rest of the sentences don’t
contribute to the buying decision
• Hence the context plays a vital role in the classification decision.
• In a large text (say a 5 page product review) with over 100 sentences, just the first sentence alone may
contribute to the decision.
• While an RNN can carry the context, there are 2 limitations:
• Due to the vanishing gradient problem, RNN’s effectiveness is limited when it needs to go back deep in to the
context.
• There is no finer control over which part of the context needs to be carried forward and how much of the past needs
to be “forgotten”
• LSTM is proposed as a solution to address this issue
33. The five key Architectural Elements of LSTM
• Input Gate
• Forget Gate
• Cell
• Output Gate
• Hidden state output
34. Effect of LSTM on sensitivity (Ref: Graves)
• In a simple RNN with sigmoid or tanh
neuron units, the later output nodes of
the network are less sensitive to the
input at time t = 1. This happens due to
the vanishing gradient problem
• An LSTM allows the preservation of
gradients. The memory cell remembers
the first input as long as the forget gate
is open and the input gate is closed.
• The output gate provides finer control
to switch the output layer on or off
without altering the cell contents.
35. Implementing an LSTM: Notes for practitioners
• Some points to take in to account while
choosing an LSTM architecture:
• LSTM has many variants compared to the
architecture proposed in the paper by Sepp
Hochreiter and Schmidhuber
• The LSTM initially didn’t have forget gate, it
was later added.
• Most of the current implementations are
based on the 3 gate LSTM model (input,
forget, output).
• Some variants adopt a simpler version. E.g.
peephole connections may be omitted
• Training is a bit complex compared to
feedforward ANN
• Many training techniques are reported. For
BPTT see Alex Graves’s thesis
• See Theano for Python DL library
• LSTMs can be stacked vertically to create a
deep LSTM network