Answers.Body: <p>Yes an LSTM can use any of these.</p>
  <p>There are no hard rules of which to use. That is why they all exist.</p>
  <p>Some rules of thumb are:</p>
  <ol>
  <li><p>Relu is the cheapest computationally. Almost always worth trying first.</p>
  </li>
  <li><p>If you find neurons are dying try leaky relu instead.</p>
  </li>
  <li><p>Tanh is probably not going to help and is much slower. Likely only worth trying if computational resources are plentiful</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T12:10:39.763Z
Answers.LastActivityDate: 2023-03-14T12:10:39.763Z
Answers.UserDisplayName: chessprogrammer
Answers.Body: <p>Yes, you can use ReLU or LeakyReLU in an LSTM model.</p>
  <p>There aren't hard rules for choosing activation functions. Run your model with each activation function and pick the best performing one.</p>
  <p>See the following academic papers on activation functions for LSTM:</p>
  <ol>
  <li><a href="https://www.sciencedirect.com/science/article/pii/S1319157820303505" rel="noreferrer">Towards activation function search for long short-term model network: A differential evolution based approach</a></li>
  <li><a href="https://thesai.org/Downloads/Volume13No9/Paper_112-Flood_Prediction_using%20Deep_Learning_Models.pdf" rel="noreferrer">Flood Prediction using Deep Learning Models</a></li>
  </ol>
  <p>The following are some more articles which may be of interest although they don't specifically address LSTMs:</p>
  <ol>
  <li><a href="https://www.v7labs.com/blog/neural-networks-activation-functions#h5" rel="noreferrer">Activation Functions in Neural Networks [12 Types &amp; Use Cases]</a></li>
  <li><a href="https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/" rel="noreferrer">How to Choose an Activation Function for Deep Learning</a></li>
  <li><a href="https://towardsdatascience.com/how-to-choose-the-right-activation-function-for-neural-networks-3941ff0e6f9c" rel="noreferrer">How to Choose the Right Activation Function for Neural Networks</a></li>
  <li><a href="http://www.diva-portal.se/smash/get/diva2:1465073/FULLTEXT01.pdf" rel="noreferrer">Performance Analysis of Various Activation Functions Using LSTM Neural Network For Movie Recommendation Systems</a></li>
  <li><a href="https://medium.com/analytics-vidhya/activation-functions-all-you-need-to-know-355a850d025e" rel="noreferrer">Activation Functions — All You Need To Know!</a></li>
  <li><a href="https://arxiv.org/ftp/arxiv/papers/2012/2012.07564.pdf" rel="noreferrer">ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance</a></li>
  <li><a href="https://wandb.ai/shweta/Activation%20Functions/reports/Activation-Functions-Compared-With-Experiments--VmlldzoxMDQwOTQ" rel="noreferrer">Activation Functions Compared With Experiments</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T12:32:52.050Z
Answers.LastActivityDate: 2023-03-14T12:32:52.050Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?</p>
  
Tags: <training><long-short-term-memory><activation-functions>
Title: Can LSTM model use ReLU or LeakyReLU as the activation funtion?
UserDisplayName: BlueSnake
ThreadText: # Can LSTM model use ReLU or LeakyReLU as the activation funtion?
  
  (Asked by: username_0 on 2023-03-14)
  
  Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
  
  **Tags:** training, long-short-term-memory, activation-functions
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-03-14. Score: 5 | Accepted answer
  
  Yes, you can use ReLU or LeakyReLU in an LSTM model.
  
  There aren't hard rules for choosing activation functions. Run your model with each activation function and pick the best performing one.
  
  See the following academic papers on activation functions for LSTM:
  
  1. [Towards activation function search for long short-term model network: A differential evolution based approach](https://www.sciencedirect.com/science/article/pii/S1319157820303505)
  2. [Flood Prediction using Deep Learning Models](https://thesai.org/Downloads/Volume13No9/Paper_112-Flood_Prediction_using%20Deep_Learning_Models.pdf)
  
  The following are some more articles which may be of interest although they don't specifically address LSTMs:
  
  1. [Activation Functions in Neural Networks [12 Types & Use Cases]](https://www.v7labs.com/blog/neural-networks-activation-functions#h5)
  2. [How to Choose an Activation Function for Deep Learning](https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/)
  3. [How to Choose the Right Activation Function for Neural Networks](https://towardsdatascience.com/how-to-choose-the-right-activation-function-for-neural-networks-3941ff0e6f9c)
  4. [Performance Analysis of Various Activation Functions Using LSTM Neural Network For Movie Recommendation Systems](http://www.diva-portal.se/smash/get/diva2:1465073/FULLTEXT01.pdf)
  5. [Activation Functions — All You Need To Know!](https://medium.com/analytics-vidhya/activation-functions-all-you-need-to-know-355a850d025e)
  6. [ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance](https://arxiv.org/ftp/arxiv/papers/2012/2012.07564.pdf)
  7. [Activation Functions Compared With Experiments](https://wandb.ai/shweta/Activation%20Functions/reports/Activation-Functions-Compared-With-Experiments--VmlldzoxMDQwOTQ)
  
  
  ---
  
  ### Answer by username_1 on 2023-03-14. Score: 5
  
  Yes an LSTM can use any of these.
  
  There are no hard rules of which to use. That is why they all exist.
  
  Some rules of thumb are:
  
  1. Relu is the cheapest computationally. Almost always worth trying first.
  2. If you find neurons are dying try leaky relu instead.
  3. Tanh is probably not going to help and is much slower. Likely only worth trying if computational resources are plentiful
  
  
  ---
  
  