Types Of Activation Function in ANN

Last Updated : 22 Jan, 2021
Improve
Improve
Like Article
Like
Save
Share
Report

The biological neural network has been modeled in the form of Artificial Neural Networks with artificial neurons simulating the function of a biological neuron. The artificial neuron is depicted in the below picture:

Structure of an Artificial Neuron

Each neuron consists of three major components: 

  1. A set of ‘i’ synapses having weight wi. A signal xi forms the input to the i-th synapse having weight wi. The value of any weight may be positive or negative. A positive weight has an extraordinary effect, while a negative weight has an inhibitory effect on the output of the summation junction.
  2. A summation junction for the input signals is weighted by the respective synaptic weight. Because it is a linear combiner or adder of the weighted input signals, the output of the summation junction can be expressed as follows: y_{sum}=\sum_{i=1}^{n}w_ix_i
  3. A threshold activation function (or simply the activation function, also known as squashing function) results in an output signal only when an input signal exceeding a specific threshold value comes as an input. It is similar in behaviour to the biological neuron which transmits the signal only when the total input signal meets the firing threshold.

Types of Activation Function :

There are different types of activation functions. The most commonly used activation function are listed below:

A. Identity Function: Identity function is used as an activation function for the input layer. It is a linear function having the form

y_{out}=f(x)=x, \forall x

As obvious, the output remains the same as the input.

B. Threshold/step Function: It is a commonly used activation function. As depicted in the diagram, it gives 1 as output of the input is either 0 or positive. If the input is negative, it gives 0 as output. Expressing it mathematically, 

y_{out}=f(y_{sum})=\bigg\{\begin{matrix} 1, x \geq 0 \\ 0, x < 0 \end{matrix}

The threshold function is almost like the step function, with the only difference being a fact that \theta  is used as a threshold value instead of . Expressing mathematically,

y_{out}=f(y_{sum})=\bigg\{\begin{matrix} 1, x \geq \theta \\ 0, x < \theta \end{matrix}

C. ReLU (Rectified Linear Unit) Function: It is the most popularly used activation function in the areas of convolutional neural networks and deep learning. It is of the form:

f(x)=\bigg\{\begin{matrix} x, x \geq 0\\ 0, x < 0 \end{matrix}

This means that f(x) is zero when x is less than zero and f(x) is equal to x when x is above or equal to zero. This function is differentiable, except at a single point x = 0. In that sense, the derivative of a ReLU is actually a sub-derivative.

D. Sigmoid Function: It is by far the most commonly used activation function in neural networks. The need for sigmoid function stems from the fact that many learning algorithms require the activation function to be differentiable and hence continuous. There are two types of sigmoid function: 

1. Binary Sigmoid Function

A binary sigmoid function is of the form: y_{out}=f(x)=\frac{1}{1+e^{-kx}}

, where k = steepness or slope parameter, By varying the value of k, sigmoid function with different slopes can be obtained. It has a range of (0,1).  The slope of origin is k/4. As the value of k becomes very large, the sigmoid function becomes a threshold function. 

2. Bipolar Sigmoid Function

A bipolar sigmoid function is of the form y_{out}=f(x)=\frac{1-e^{-kx}}{1+e^{-kx}}

The range of values of sigmoid functions can be varied depending on the application. However, the range of (-1,+1) is most commonly adopted.

E. Hyperbolic Tangent Function: It is bipolar in nature. It is a widely adopted activation function for a special type of neural network known as Backpropagation Network. The hyperbolic tangent function is of the form

y_{out}=f(x)\frac{e^x-e^-x}{e^x+e^-x}  

This function is similar to the bipolar sigmoid function. 



V
versatile1990
Improve
Previous Article
Activation Functions
Next Article
Activation Functions in Pytorch

Similar Reads

Types Of Learning Rules in ANN
Learning rule enhances the Artificial Neural Network’s performance by applying this rule over the network. Thus learning rule updates the weights and bias levels of a network when certain conditions are met in the training process. it is a crucial part of the development of the Neural Network. Types Of Learning Rules in ANN 1. Hebbian Learning Rule
4 min read
Swish Activation Function
As the Machine Learning community keeps working on trying to identify complex patterns in the dataset for better results, Google proposed the Swish Activation function as an alternative to the popular ReLU activation function. The authors of the research paper show that using the Swish Activation function instead of ReLU improves the accuracy of th
4 min read
Swish activation function in Pytorch
Activation functions are a fundamental component of artificial neural networks. They introduce non-linearity into the model, allowing it to learn complex relationships in the data. One such activation function, the Swish activation function, has gained attention for its unique properties and potential advantages over the widely used Rectified Linea
6 min read
Why Is ReLU Used as an Activation Function?
Answer: ReLU is used as an activation function due to its simplicity, non-saturating nature, and effectiveness in combating the vanishing gradient problem, leading to faster training and improved performance in deep neural networks.ReLU (Rectified Linear Unit) is a popular activation function in neural networks for several reasons: Simplicity: ReLU
2 min read
What activation function should I use for a specific regression problem?
Answer: For regression problems, commonly used activation functions include linear activation, as it directly outputs the weighted sum of inputs without introducing non-linearity.In the realm of regression problems, selecting an appropriate activation function is pivotal for the successful training of a neural network. Unlike classification problem
2 min read
Choosing the Right Activation Function for Your Neural Network
Activation functions are a critical component in the design and performance of neural networks. They introduce non-linearity into the model, enabling it to learn and represent complex patterns in the data. Choosing the right activation function can significantly impact the efficiency and accuracy of a neural network. This article will guide you thr
10 min read
Introduction to ANN (Artificial Neural Networks) | Set 3 (Hybrid Systems)
Prerequisites: Genetic algorithms, Artificial Neural Networks, Fuzzy Logic Hybrid systems: A Hybrid system is an intelligent system that is framed by combining at least two intelligent technologies like Fuzzy Logic, Neural networks, Genetic algorithms, reinforcement learning, etc. The combination of different techniques in one computational model m
4 min read
Introduction to ANN | Set 4 (Network Architectures)
Prerequisites: Introduction to ANN | Set-1, Set-2, Set-3 An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by the brain. ANNs, like people, learn by examples. An ANN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Learning largely invol
5 min read
Heart Disease Prediction using ANN
Deep Learning is a technology of which mimics a human brain in the sense that it consists of multiple neurons with multiple layers like a human brain. The network so formed consists of an input layer, an output layer, and one or more hidden layers. The network tries to learn from the data that is fed into it and then performs predictions accordingl
3 min read
ANN - Implementation of Self Organizing Neural Network (SONN) from Scratch
Prerequisite: ANN | Self Organizing Neural Network (SONN) Learning Algorithm To implement a SONN, here are some essential consideration- Construct a Self Organizing Neural Network (SONN) or Kohonen Network with 100 neurons arranged in a 2-dimensional matrix with 10 rows and 10 columns Train the network with 1500 2-dimensional input vectors randomly
4 min read
Article Tags :
Practice Tags :