Answers.Body: <p>Currently, both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.</p>
  <p>Also, these AFs, more specifically ReLU, are very fast learning AF which makes them even more useful in research.</p>
  <p>However, depending on the type of NN you working on, it's always a good practice to pay some attention to new studies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T13:24:04.290Z
Answers.Text: Maybe you should provide some references that show that ReLU and ELU are the most widely used AFs. If there are some statistics, that would even be better.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T13:23:28.280Z
Answers.Text: @nbro I will have a look and update any answer I provide over time. I haven't found any good source to provide stat about it. Please let me know if you know any. Thanks for the suggestion and also for keeping this community well.
Answers.UserDisplayName: pedrum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T02:32:52.417Z
Answers.LastActivityDate: 2021-12-21T10:42:03.703Z
Answers.LastEditDate: 2021-12-21T10:42:03.703Z
Answers.UserDisplayName: pedrum
Body: <p>I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? E.g., ReLU, ELUs, etc. I'd appreciate any statistics or insight on this.</p>
  
Tags: <neural-networks><deep-learning><activation-functions>
Title: What activation functions are currently popular?
UserDisplayName: Schach21
ThreadText: # What activation functions are currently popular?
  
  (Asked by: username_0 on 2020-07-26)
  
  I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? E.g., ReLU, ELUs, etc. I'd appreciate any statistics or insight on this.
  
  **Tags:** neural-networks, deep-learning, activation-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-27. Score: 2 | Accepted answer
  
  Currently, both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.
  
  Also, these AFs, more specifically ReLU, are very fast learning AF which makes them even more useful in research.
  
  However, depending on the type of NN you working on, it's always a good practice to pay some attention to new studies.
  
  > **Comments:**
  > **username_2**: Maybe you should provide some references that show that ReLU and ELU are the most widely used AFs. If there are some statistics, that would even be better.  
  > **username_1**: @username_2 I will have a look and update any answer I provide over time. I haven't found any good source to provide stat about it. Please let me know if you know any. Thanks for the suggestion and also for keeping this community well.  
  
  ---
  
  