Answers.Body: <p>I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a> is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in <a href="https://arxiv.org/pdf/1710.05941.pdf" rel="nofollow noreferrer">Searching for Activation Functions</a>, </p>
  
  <blockquote>
    <p>the performance improvements of the other activation functions tend to be inconsistent across different models and datasets.</p>
  </blockquote>
  
  <p>Even if a new activation function did provide a meager improvement in performance across the board, I wouldn't be surprised if ReLU were still commonly used. It's the default for a lot of machine learning software already.</p>
  
  <p>Also research isn't ordinarily about eeking out one more percentage point in accuracy on a specific task. If I were entering in a competition, I might experiment with activation functions. But even then I'd rather use ReLU and save a little time while prototyping my architecture.</p>
  
  <p>As pointed out by @DuttaA in comments, softsign could potentially replace sigmoid and tanh in situations where a bounded output is desired. I haven't seen anyone compare them before, but softsign would at least be much faster. I'm guessing this replacement hasn't happened because of tradition and exposure. Not because of softsign's lack of infinite derivatives. I don't know if this happening would make softsign "widely used", but it would be something.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-18T06:17:22.583Z
Answers.Text: Sure, sigmoid and tanh are used sometimes. But they have specific use cases. And yes, things like the softmax output activation are obviously still used. I didn't mean to imply that they weren't. But most of the time, on a general neuron in the middle of a deep architecture, we choose ReLU. Actually, the proposed activation might best be used as a replacement for sigmoid and tanh, where bounded outputs are important, but I'm speculating.
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-17T23:08:19.130Z
Answers.LastActivityDate: 2019-03-18T09:32:03.817Z
Answers.LastEditDate: 2019-03-18T09:32:03.817Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>The Softsign (a.k.a. ElliotSig) activation function is really simple: </p>
  
  <p><span class="math-container">$$ f(x) = \frac{x}{1+|x|} $$</span></p>
  
  <p>It is bounded <span class="math-container">$[-1,1]$</span>, has a first derivative, it is monotonic, and it is computationally extremely simple (easy for, e.g., a GPU).</p>
  
  <p>Why it is not widely used in neural networks? Is it because it is not infinitely derivable?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-18T18:11:46.247Z
Comments.Text: @DuttaA - It is because the activation function is part of the calculations. I am just wondering if for a GPU it may be quicker to compute simple algebraic functions instead of exponentials.
Comments.UserDisplayName: Pietro
Tags: <machine-learning><activation-functions><performance>
Title: Why isn't the ElliotSig activation function widely used?
UserDisplayName: Pietro
ThreadText: # Why isn't the ElliotSig activation function widely used?
  
  (Asked by: username_0 on 2019-02-05)
  
  The Softsign (a.k.a. ElliotSig) activation function is really simple:
  
  $$ f(x) = \frac{x}{1+|x|} $$
  
  It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally extremely simple (easy for, e.g., a GPU).
  
  Why it is not widely used in neural networks? Is it because it is not infinitely derivable?
  
  **Tags:** machine-learning, activation-functions, performance
  > **Comments:**
  > **username_0**: @DuttaA - It is because the activation function is part of the calculations. I am just wondering if for a GPU it may be quicker to compute simple algebraic functions instead of exponentials.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-17. Score: 6 | Accepted answer
  
  I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function.
  
  [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in [Searching for Activation Functions](https://arxiv.org/pdf/1710.05941.pdf),
  
  > the performance improvements of the other activation functions tend to be inconsistent across different models and datasets.
  
  Even if a new activation function did provide a meager improvement in performance across the board, I wouldn't be surprised if ReLU were still commonly used. It's the default for a lot of machine learning software already.
  
  Also research isn't ordinarily about eeking out one more percentage point in accuracy on a specific task. If I were entering in a competition, I might experiment with activation functions. But even then I'd rather use ReLU and save a little time while prototyping my architecture.
  
  As pointed out by @DuttaA in comments, softsign could potentially replace sigmoid and tanh in situations where a bounded output is desired. I haven't seen anyone compare them before, but softsign would at least be much faster. I'm guessing this replacement hasn't happened because of tradition and exposure. Not because of softsign's lack of infinite derivatives. I don't know if this happening would make softsign "widely used", but it would be something.
  
  > **Comments:**
  > **username_1**: Sure, sigmoid and tanh are used sometimes. But they have specific use cases. And yes, things like the softmax output activation are obviously still used. I didn't mean to imply that they weren't. But most of the time, on a general neuron in the middle of a deep architecture, we choose ReLU. Actually, the proposed activation might best be used as a replacement for sigmoid and tanh, where bounded outputs are important, but I'm speculating.  
  
  ---
  
  