Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
16 views

Deep Learning Tutorial 3

The document discusses different types of activation functions used in artificial neural networks including linear, non-linear, threshold, sigmoid, tanh, ReLU, LeakyReLU, and softmax functions.

Uploaded by

Atal Bihari
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
16 views

Deep Learning Tutorial 3

The document discusses different types of activation functions used in artificial neural networks including linear, non-linear, threshold, sigmoid, tanh, ReLU, LeakyReLU, and softmax functions.

Uploaded by

Atal Bihari
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 12

Deep Learning from

Scratch
Theory + Practical
FAHAD HUSSAIN
MCS, MSCS, DAE(CIT)

Computer Science Instructor of well known international Center


Also, Machine Learning and Deep learning Practitioner

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Artificial
Normalize/Standardize
Neural Network
𝒎

∑ 𝒘𝒊 𝒙𝒊
𝒊= 𝒙

Applying Activation
Function

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
What is an Activation Function?
Activation functions are an extremely important feature of the artificial neural networks.
They basically decide whether a neuron should be activated or not. Whether the
information that the neuron is receiving is relevant for the given information or should it
be ignored.

The activation function is the non linear transformation that we do over the input
signal. This transformed output is then seen to the next layer of neurons as input.

• Linear Activation Function


• Non Linear Activation Function
For further assistance, code and slide https://fahadhussaincs.blogspot.com/
YouTube Channel : https://www.youtube.com/fahadhussaintutorial
What is an Activation Function?
Linear Function
The function is a line or linear. Therefore, the output of
the functions will not be confined between any range

Non Linear Function


1.Threshold
They make it easy for the model to generalize or adapt 2.Sigmoid
with variety of data and to differentiate between the
output
3.Tanh
4.ReLU
The Nonlinear Activation Functions are mainly divided on 5.Leaky ReLU
the basis of their range or curves
6.Softmax
For further assistance, code and slide https://fahadhussaincs.blogspot.com/
YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Threshold Function?

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Sigmoid Function?
The Sigmoid Function curve looks like a S-shape
This function reduces extreme values or outliers in data without removing them.
It converts independent variables of near infinite range into simple probabilities
between 0 and 1, and most of its output will be very close to 0 or 1.

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Rectifier (Relu) Function?
ReLU is the most widely used activation function while designing networks today. First
things first, the ReLU function is non linear, which means we can easily backpropagate the
errors and have multiple layers of neurons being activated by the ReLU function.

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Leaky Relu Function?
Leaky ReLU function is nothing but an improved version of the ReLU function. As we saw that
for the ReLU function, the gradient is 0 for x<0, which made the neurons die for activations in
that region. Leaky ReLU is defined to address this problem. Instead of defining the Relu
function as 0 for x less than 0, we define it as a small linear component of x.

What we have done here is that we have simply replaced the horizontal line with a non-zero, non-horizontal line. Here
a is a small value like 0.01 or so.

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Sigmoid Function?
Pronounced “tanch,” tanh is a hyperbolic trigonometric function
The tangent represents a ratio between the opposite and adjacent sides of a right triangle,
tanh represents the ratio of the hyperbolic sine to the hyperbolic cosine: tanh(x) = sinh(x) /
cosh(x)
Unlike the Sigmoid function, the normalized range of tanh is –1 to 1 The advantage of tanh is
that it can deal more easily with negative numbers

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Softmax Function (for Multiple
Classification)?
Softmax function calculates the probabilities distribution of the event over ‘n’ different events. In general way of
saying, this function will calculate the probabilities of each target class over all possible target classes. Later the
calculated probabilities will be helpful for determining the target class for the given inputs.

The main advantage of using Softmax is the output probabilities range. The range will 0 to 1, and the sum of all
the probabilities will be equal to one. If the softmax function used for multi-classification model it returns the
probabilities of each class and the target class will have the high probability.

The formula computes the exponential (e-power) of the given input value and the sum of exponential values of
all the values in the inputs. Then the ratio of the exponential of the input value and the sum of exponential
values is the output of the softmax function.

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Activation Function Example

For further assistance, code and slide https://fahadhussaincs.blogspot.com/


YouTube Channel : https://www.youtube.com/fahadhussaintutorial
Thank You!
For further assistance, code and slide https://fahadhussaincs.blogspot.com/
YouTube Channel : https://www.youtube.com/fahadhussaintutorial

You might also like