Answers.Body: <p>There are <a href="https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions" rel="nofollow noreferrer">a few traits that you want the activation function to have</a>, and cube roots rate as OK-ish:</p>
  
  <ul>
  <li><p><em>Nonlinear</em> – check.</p></li>
  <li><p><em>Continuously</em> differentiable – no. There is a problem at <span class="math-container">$x=0$</span>. Unlike other discontinuous functions like ReLU, although the gradient can be calculated near zero, it can be arbitrarily high as you approach <span class="math-container">$x=0$</span>, because <span class="math-container">$\frac{d}{dx}x^{\frac{1}{3}} = \frac{1}{3x^{\frac{2}{3}}}$</span> </p></li>
  <li><p><em>Range considerations</em> – limited range functions are more stable, large/infinite range functions are more efficient. You may need to reduce learning rates compared to e.g. tanh.</p></li>
  <li><p><em>Monotonic</em> – check.</p></li>
  <li><p><em>Monotonic derivative</em> - no.</p></li>
  <li><p><em>Approximates identity near the origin</em> – no, the approximation is bad near the origin.</p></li>
  </ul>
  
  <p>If you look through the list of current, successful activation functions, you will see a few that also fail to provide one or more desirable traits, yet are still used routinely.</p>
  
  <p>I would worry about the high gradients near <span class="math-container">$x=0$</span>, but other than that I think the function could work ok. It may sometimes be unstable during learning, as small changes near zero will result in large changes to output. You might be able to work around the high gradients simply in practice, by clipping them. If the raw calculation returns value greater than <span class="math-container">$1$</span> (or less than <span class="math-container">$-1$</span>) then treat the gradient as if it were <span class="math-container">$1$</span> (or <span class="math-container">$-1$</span>) for the rest of backpropagation. </p>
  
  <p>The only way of finding out if the function is competitive with other more standard activation functions is to try it on some standard data sets and make comparisons.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-30T15:52:47.403Z
Answers.LastActivityDate: 2018-09-30T16:08:06.890Z
Answers.LastEditDate: 2018-09-30T16:08:06.890Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to design a neural network on Python.</p>
  
  <p>Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph:
  <a href="https://i.sstatic.net/jSqOd.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jSqOd.jpg" alt="A graph of the cube root function"></a></p>
  
  <p>Is this suitable?</p>
  
Tags: <neural-networks><activation-functions>
Title: Is the cube root function suitable as a n activation function?
UserDisplayName: Vikhyat Agarwal
ThreadText: # Is the cube root function suitable as a n activation function?
  
  (Asked by: username_0 on 2018-09-30)
  
  I am trying to design a neural network on Python.
  
  Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph:
  [![A graph of the cube root function](https://i.sstatic.net/jSqOd.jpg)](https://i.sstatic.net/jSqOd.jpg)
  
  Is this suitable?
  
  **Tags:** neural-networks, activation-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-30. Score: 3 | Accepted answer
  
  There are [a few traits that you want the activation function to have](https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions), and cube roots rate as OK-ish:
  
  * *Nonlinear* – check.
  * *Continuously* differentiable – no. There is a problem at $x=0$. Unlike other discontinuous functions like ReLU, although the gradient can be calculated near zero, it can be arbitrarily high as you approach $x=0$, because $\frac{d}{dx}x^{\frac{1}{3}} = \frac{1}{3x^{\frac{2}{3}}}$
  * *Range considerations* – limited range functions are more stable, large/infinite range functions are more efficient. You may need to reduce learning rates compared to e.g. tanh.
  * *Monotonic* – check.
  * *Monotonic derivative* - no.
  * *Approximates identity near the origin* – no, the approximation is bad near the origin.
  
  If you look through the list of current, successful activation functions, you will see a few that also fail to provide one or more desirable traits, yet are still used routinely.
  
  I would worry about the high gradients near $x=0$, but other than that I think the function could work ok. It may sometimes be unstable during learning, as small changes near zero will result in large changes to output. You might be able to work around the high gradients simply in practice, by clipping them. If the raw calculation returns value greater than $1$ (or less than $-1$) then treat the gradient as if it were $1$ (or $-1$) for the rest of backpropagation.
  
  The only way of finding out if the function is competitive with other more standard activation functions is to try it on some standard data sets and make comparisons.
  
  
  ---
  
  