Answers.Body: <p>Let me give a very easy explanation...
  Activation Functions in neural networks are useful as they introduce non-linearity into the model, which helps in complex data pattern understanding.</p>
  <p><strong>Sigmoid Function</strong></p>
  <ul>
  <li>Sigmoid converts output values into range of (0,1).</li>
  <li>It is mostly used for binary classification problems.</li>
  <li>It suffers from Vanishing Gradient Problem. As the Input value becomes small or large. The gradient turns out very very small (near about zero).This will slow down the training process because weights and biases receives very small updates.</li>
  <li>E.g. :  <code>For a large input, say 10, sigmoid(10) ≈ 0.9999. The gradient is sigmoid(10) * (1 - sigmoid(10)) ≈ 0.0001, which is very small.</code>
  <a href="https://i.sstatic.net/JHDy9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JHDy9.png" alt="Sigmoid Function" /></a></li>
  </ul>
  <p><strong>ReLU Function</strong></p>
  <ul>
  <li>ReLU converts all the negative input to 0 and positive as it is</li>
  <li>It is mostly used in hidden layers which allows faster training because of the linearity in non-saturating form.</li>
  <li>It helps in the vanishing gradient problem of hidden layers.</li>
  <li>In some cases, there is a dyeing ReLU problem, where neurons sometimes only output zero(for all negative inputs).</li>
  <li>E.g. :<code> For an input of 5, ReLU(5) = 5. For an input of -3, ReLU(-3) = 0, and the gradient is 0.</code>
  <a href="https://i.sstatic.net/dTkbD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dTkbD.png" alt="ReLU Function" /></a></li>
  </ul>
  <p><strong>Tanh Function</strong></p>
  <ul>
  <li>The Tanh function outputs values in the range of (-1,1). It is somewhat similar to sigmoid but stretched for a wider range</li>
  <li>It is very useful when data is centered around zero, Useful in situations where input is negative.</li>
  <li>Tanh also suffers from vanishing gradient but lesser extent.</li>
  <li>E.g. :<code>For a large input, say 10, tanh(10) ≈ 1. The gradient is 1 - tanh(10)^2 ≈ 0, which is very small.</code>
  <a href="https://i.sstatic.net/iyofZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iyofZ.png" alt="Tanh Function" /></a></li>
  </ul>
  <p><strong>Other</strong>
  There are other activation functions like</p>
  <ul>
  <li><strong>Leaky ReLU:</strong> It is a variation of ReLU but allows a small negative result. This will prevent dead neuron problem.
  <a href="https://i.sstatic.net/CTuJ8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CTuJ8.png" alt="Leaky ReLu" /></a></li>
  <li><strong>PReLU(Parametric ReLU):</strong> It is modified Leaky ReLU where small slope of negative input is learned during training.
  <a href="https://i.sstatic.net/2SihY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2SihY.png" alt="PReLU" /></a></li>
  <li><strong>ELU(Exponential Linear Unit):</strong> Positive input will be as it is and for negative result exponential decay function is used. It is combination of sigmoid/tanh with ReLU.
  <a href="https://i.sstatic.net/K3oRu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/K3oRu.png" alt="ELU" /></a></li>
  <li><strong>SELU(Scaled Exponential Linear Unit):</strong> It will encourage self-normalizing properties for faster and more stable training.
  <a href="https://i.sstatic.net/OeSpS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OeSpS.png" alt="SELU" /></a></li>
  <li><strong>SoftMax:</strong> Converts vector of values into a probability distribution. Where probability is in proportion to its exponential. Mostly used in the output layer of multi-class classification.
  <a href="https://i.sstatic.net/lsexN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lsexN.png" alt="Softmax" /></a></li>
  <li>Others are SoftPlus, Swish, Hard Sigmoid, Hard Swish, Gaussian, etc...</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-01T03:43:55.960Z
Answers.LastActivityDate: 2023-12-01T04:00:36.110Z
Answers.LastEditDate: 2023-12-01T04:00:36.110Z
Answers.UserDisplayName: Hiren Namera
Body: <p>I need to specify the points and conditions that require me to use one activation function over another,</p>
  <ul>
  <li>Why is it not possible to use the sigmoid FUNCTION in hidden layers, and it is preferable to use Relu instead?</li>
  <li>When do we use the Relu function in hidden layers and when do we use Tanh?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-30T19:58:40.803Z
Comments.Text: Here is general guide/practice, https://www.turing.com/kb/how-to-choose-an-activation-function-for-deep-learning. Please add the task you want to use your neural network and what model under your consideration in your question. It will help us to give you good suggestion.
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks>
Title: Types of activation functions used in neural networks
UserDisplayName: maya sy
ThreadText: # Types of activation functions used in neural networks
  
  (Asked by: username_0 on 2023-11-30)
  
  I need to specify the points and conditions that require me to use one activation function over another,
  
  * Why is it not possible to use the sigmoid FUNCTION in hidden layers, and it is preferable to use Relu instead?
  * When do we use the Relu function in hidden layers and when do we use Tanh?
  
  **Tags:** neural-networks
  
  > **Comments:**
  > **username_1**: Here is general guide/practice, https://www.turing.com/kb/how-to-choose-an-activation-function-for-deep-learning. Please add the task you want to use your neural network and what model under your consideration in your question. It will help us to give you good suggestion.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-12-01. Score: 2
  
  Let me give a very easy explanation...
  Activation Functions in neural networks are useful as they introduce non-linearity into the model, which helps in complex data pattern understanding.
  
  **Sigmoid Function**
  
  * Sigmoid converts output values into range of (0,1).
  * It is mostly used for binary classification problems.
  * It suffers from Vanishing Gradient Problem. As the Input value becomes small or large. The gradient turns out very very small (near about zero).This will slow down the training process because weights and biases receives very small updates.
  * E.g. : `For a large input, say 10, sigmoid(10) ≈ 0.9999. The gradient is sigmoid(10) * (1 - sigmoid(10)) ≈ 0.0001, which is very small.`
    [![Sigmoid Function](https://i.sstatic.net/JHDy9.png)](https://i.sstatic.net/JHDy9.png)
  
  **ReLU Function**
  
  * ReLU converts all the negative input to 0 and positive as it is
  * It is mostly used in hidden layers which allows faster training because of the linearity in non-saturating form.
  * It helps in the vanishing gradient problem of hidden layers.
  * In some cases, there is a dyeing ReLU problem, where neurons sometimes only output zero(for all negative inputs).
  * E.g. : `For an input of 5, ReLU(5) = 5. For an input of -3, ReLU(-3) = 0, and the gradient is 0.`
    [![ReLU Function](https://i.sstatic.net/dTkbD.png)](https://i.sstatic.net/dTkbD.png)
  
  **Tanh Function**
  
  * The Tanh function outputs values in the range of (-1,1). It is somewhat similar to sigmoid but stretched for a wider range
  * It is very useful when data is centered around zero, Useful in situations where input is negative.
  * Tanh also suffers from vanishing gradient but lesser extent.
  * E.g. :`For a large input, say 10, tanh(10) ≈ 1. The gradient is 1 - tanh(10)^2 ≈ 0, which is very small.`
    [![Tanh Function](https://i.sstatic.net/iyofZ.png)](https://i.sstatic.net/iyofZ.png)
  
  **Other**
  There are other activation functions like
  
  * **Leaky ReLU:** It is a variation of ReLU but allows a small negative result. This will prevent dead neuron problem.
    [![Leaky ReLu](https://i.sstatic.net/CTuJ8.png)](https://i.sstatic.net/CTuJ8.png)
  * **PReLU(Parametric ReLU):** It is modified Leaky ReLU where small slope of negative input is learned during training.
    [![PReLU](https://i.sstatic.net/2SihY.png)](https://i.sstatic.net/2SihY.png)
  * **ELU(Exponential Linear Unit):** Positive input will be as it is and for negative result exponential decay function is used. It is combination of sigmoid/tanh with ReLU.
    [![ELU](https://i.sstatic.net/K3oRu.png)](https://i.sstatic.net/K3oRu.png)
  * **SELU(Scaled Exponential Linear Unit):** It will encourage self-normalizing properties for faster and more stable training.
    [![SELU](https://i.sstatic.net/OeSpS.png)](https://i.sstatic.net/OeSpS.png)
  * **SoftMax:** Converts vector of values into a probability distribution. Where probability is in proportion to its exponential. Mostly used in the output layer of multi-class classification.
    [![Softmax](https://i.sstatic.net/lsexN.png)](https://i.sstatic.net/lsexN.png)
  * Others are SoftPlus, Swish, Hard Sigmoid, Hard Swish, Gaussian, etc...
  
  
  ---
  
  