Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
2 views

9 DL_ANN_ActivationFunctions

Activation functions are crucial for determining the output of nodes in neural networks, transforming linear inputs into linear or non-linear outputs. They must be monotonic, differentiable, and converge quickly, with common types including linear and non-linear functions like Sigmoid, Tanh, and ReLU. Each function has unique properties affecting training speed and gradient behavior, with implementations such as Softmax for multi-class classification.

Uploaded by

karunyakuraganti
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
2 views

9 DL_ANN_ActivationFunctions

Activation functions are crucial for determining the output of nodes in neural networks, transforming linear inputs into linear or non-linear outputs. They must be monotonic, differentiable, and converge quickly, with common types including linear and non-linear functions like Sigmoid, Tanh, and ReLU. Each function has unique properties affecting training speed and gradient behavior, with implementations such as Softmax for multi-class classification.

Uploaded by

karunyakuraganti
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Topic: Introduction To Activation Functions

HariBabu KVN
Activation Functions • Activation Functions are
applied over the linear
weighted summation of the
incoming information to a
node.
• Convert linear input signals
from perceptron to a
linear/non-linear output
signal.
• Decide whether to activate a
node or not.
Activation Functions
• Must be monotonic,
differentiable, and
quickly converging.
• Types of Activation
Functions:
• Linear
• Non-Linear
Linear Activation Functions

f(x) = ax + b
df(x)/dx = a
• Observations:
• Constant gradient
• Gradient does not
depend on the
change in the input
Non-Linear Activation Functions

• Sigmoid (Logistic)
• Hyperbolic Tangent (Tanh)
• Rectified Linear Unit (ReLU)
– Leaky ReLU
– Parametric ReLU
• Exponential Linear Unit (ELU)
Sigmoid

• Observations:
– Output: 0 to 1
– Outputs are not
zero-centered
– Can saturate and kill
(vanish) gradients
Tanh

• Observations:
– Output: -1 to 1
– Outputs are zero-centered
– Can saturate and kill
(vanish) gradients
– Gradient is more steeped
than Sigmoid resulting in
faster convergence
ReLU

• Observations:
– Greatly increases training
speed compared to Tanh and
Sigmoid
– Reduces likelihood of
vanishing gradient
– Issues: Dead nodes and
blowing up activations
Leaky ReLU

• Observations:
– Fixes dying ReLU issue
Parametric ReLU
Multi-Class Classification
Multi-Class Classification
Softmax
Softmax
Softmax
Binary Classification
Softmax Implementation

import numpy as np

def softmax(z):
'''Return the softmax output of a vector.'''
exp_z = np.exp(z)
sum = exp_z.sum()
softmax_z = np.round(exp_z/sum,3)
return softmax_z

You might also like