Body: <p>I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as <a href="https://ieeexplore.ieee.org/document/51993" rel="noreferrer">this paper</a> have worked on training networks with discontinuous activations yet this does not seem to have taken off. Does anybody have insight into why this happens, or better yet an article talking about this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T00:25:54.897Z
Comments.Text: [Here](https://ai.stackexchange.com/q/17609/2444) and [here](https://ai.stackexchange.com/q/19894/2444) are two related questions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T13:58:42.093Z
Comments.Text: I had seen both before posting but none of them discusses why they are not wildly used or why most activations are continuous.
Comments.UserDisplayName: ABIM
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T15:27:06.440Z
Comments.Text: Hi @BIM, check [this](https://stats.stackexchange.com/questions/271701/why-is-step-function-not-used-in-activation-functions-in-machine-learning) out, has some interesting thoughts. It's about step-function though.
Comments.UserDisplayName: mark mark
Tags: <neural-networks><reference-request><optimization><gradient-descent><activation-functions>
Title: Why are most commonly used activation functions continuous?
UserDisplayName: ABIM
ThreadText: # Why are most commonly used activation functions continuous?
  
  (Asked by: username_0 on 2021-01-27)
  
  I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as [this paper](https://ieeexplore.ieee.org/document/51993) have worked on training networks with discontinuous activations yet this does not seem to have taken off. Does anybody have insight into why this happens, or better yet an article talking about this?
  
  **Tags:** neural-networks, reference-request, optimization, gradient-descent, activation-functions
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/17609/2444) and [here](https://ai.stackexchange.com/q/19894/2444) are two related questions.  
  > **username_0**: I had seen both before posting but none of them discusses why they are not wildly used or why most activations are continuous.  
  > **username_2**: Hi @BIM, check [this](https://stats.stackexchange.com/questions/271701/why-is-step-function-not-used-in-activation-functions-in-machine-learning) out, has some interesting thoughts. It's about step-function though.  
  
  ---
  
  ## 0 answers
  
  