Answers.Body: <p><strong>Update:</strong> the issue is that your <code>odd_pow()</code> is <em>numerically unstable</em>, and so to fix the exploding gradients, which cause a NaN loss, you simply need to add a small constant when computing the power:</p>
  <pre><code>def odd_pow(input, exponent, eps=1e-4):
      return input.sign() * (input.abs() + eps).pow(exponent)
  </code></pre>
  <p>this ensures that the <code>pow</code> is not computed at zero, avoiding NaNs.</p>
  <hr />
  <p><strong>Note:</strong> for reference I also keep my old answer, since also the activation <span class="math-container">$\sigma(x)$</span> suffers the same problem.</p>
  <p>The issue is that the function <span class="math-container">$\sigma(x) = x^{1/3}$</span> is zero for <span class="math-container">$x\le 0$</span>, and so its derivative becomes NaN for negative <span class="math-container">$x$</span> and infinity for <span class="math-container">$x=0$</span> as shown here:
  <a href="https://i.sstatic.net/rdl0c.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rdl0c.png" alt="enter image description here" /></a>
  where the blue curve is the derivative of <span class="math-container">$\sigma(x)$</span>, which is depicted in green.</p>
  <p>To avoid either NaN or exploding gradients, you need to add a small value (usually called an <em>epsilon</em>) to make the computation of the derivative more numerically stable. The following should work:</p>
  <pre><code>def custom_activation(x, eps=1e-3):
      # set negative values to zero
      tmp = torch.nn.functional.relu(x)
      # add the small epsilon `eps`
      return torch.pow(tmp + eps, exponent=1 / 3.0)
  </code></pre>
  <p>Notice that as you lower the <code>eps</code>, e.g. at 1e-5, the derivative would be larger. If you really need a small epsilon, then you should also consider to clip the individual gradients (there should be a <code>clip_grad</code> functionality related to the optimizer) to ensure stable training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T13:54:35.463Z
Answers.Text: The function $\sigma(x) = x^{1/3}$ is not zero for $x \leq 0$—I am taking the real root.
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T14:02:37.977Z
Answers.Text: The zero at $x=0$ is not the issue. It's the derivative is $\frac{1}{3}x^{-\frac{2}{3}}$ which is undefined at zero. With their intended $\sigma'$ function, which should simply have derivative of $1$ at $x=0$, the OP looks to have an implementation problem here, not a theoretical one.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T16:17:02.247Z
Answers.Text: @NeilSlater It could be that torch isn't taking the real roots for the negative values of $x^{-2/3}$.
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T18:02:18.610Z
Answers.Text: @user572780: I would have thought that the `.abs()` would have stopped that issue. I cannot see clearly what is wrong, although I suspect one or more sub-components of your `custom_activation` function is confusing the auto-differentiation.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T13:00:17.420Z
Answers.Text: @NeilSlater I implemented and tested the code I provide here in Tensorflow, on a simple MNIST classification problem and it trains without any issue
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T13:44:58.490Z
Answers.Text: In TensorFlow? Do you mean PyTorch? Either way, it is *not* the OP's activation function, and there's nothing wrong with the OP's function $\sigma'(x)$ in theory AFAICS.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T13:52:05.057Z
Answers.Text: There's also nothing wrong with the activation function presented in this answer. However, the *reason* given for the OP's problem in the first sentence is incorrect, and the suggestion to use a different function is also not an answer to the original question.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-13T08:20:49.400Z
Answers.Text: @NeilSlater Sorry but I made a bit of confusion here :) As you said, both $\sigma$ and $\sigma'$ are fine in theory. The issue, in practice, is numerical instability around the zero (I think). I've updated my answer and the issue should be fixed now
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-13T12:03:12.520Z
Answers.Text: I think you need to take a careful look at OP's definition of $\sigma'(x)$ The function is linear near the origin. You seem to be talking about the OP's first function -  at least a lot of your comments are correct when applied to that
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-14T15:31:22.757Z
Answers.Text: @NeilSlater Yes, I was referring to the first function, $\sigma(x)$ because as far as I understood $\sigma'$ was a replacement for that, since not working as intended. So I aimed at just fixing $\sigma$
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T13:45:01.490Z
Answers.LastActivityDate: 2023-09-13T08:19:04.297Z
Answers.LastEditDate: 2023-09-13T08:19:04.297Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I implemented the following activation function:
  <span class="math-container">$$\sigma(x) = x^{1/3},$$</span>
  which returns NaNs after some epochs. I think this is due to the derivative exploding close to <span class="math-container">$0$</span>. To fix this issue, I implemented a modified activation function:
  <span class="math-container">$$
  \sigma'(x) = 
  \begin{cases} 
   x, &amp; \text{if } -1 \leq x \leq 1, \\
   x^{1/3}, &amp; \text{otherwise.}
  \end{cases}
  $$</span>
  Unfortunately, <span class="math-container">$\sigma'$</span> suffers from the same issue. Why is that?</p>
  <p>Here is my code:</p>
  <pre><code>def odd_pow(input, exponent):
      return input.sign() * input.abs().pow(exponent)
  
  def custom_activation(x):
      # This will create a mask where values are between -1 and 1
      mask = (x &gt;= -1) &amp; (x &lt;= 1)
      # For values between -1 and 1, apply x
      # For other values, apply x^1/3
      # Use where to combine results based on the mask
      return torch.where(mask, odd_pow(x, 1), odd_pow(x, 1/3))
  </code></pre>
  <p>Note: Both ReLU and Tanh don't return NaNs, but I am looking for homogeneous activation functions <span class="math-container">$\sigma(ax) = g(a)\sigma(x)$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-11T13:13:14.470Z
Comments.Text: Does the derivative cause the NaNs, or the normal activation function?
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-11T13:13:55.197Z
Comments.Text: @RobinvanHoorn I think the derivative is what is causing the NaNs.
Comments.UserDisplayName: user572780
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-13T08:21:31.173Z
Comments.Text: You need to use a small constant to prevent exploding gradients. Please, see my answer.
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-13T14:10:12.710Z
Comments.Text: Please look at the gradients being backpropagated to see where the issue comes from.
Comments.UserDisplayName: Lelouch
Tags: <activation-functions>
Title: Why does my activation function cause NaNs?
UserDisplayName: user572780
ThreadText: # Why does my activation function cause NaNs?
  
  (Asked by: username_0 on 2023-09-11)
  
  I implemented the following activation function:
  $$\sigma(x) = x^{1/3},$$
  which returns NaNs after some epochs. I think this is due to the derivative exploding close to $0$. To fix this issue, I implemented a modified activation function:
  $$
  \sigma'(x) =
  \begin{cases}
  x, & \text{if } -1 \leq x \leq 1, \\
  x^{1/3}, & \text{otherwise.}
  \end{cases}
  $$
  Unfortunately, $\sigma'$ suffers from the same issue. Why is that?
  
  Here is my code:
  
  ```
  def odd_pow(input, exponent):
      return input.sign() * input.abs().pow(exponent)
  
  def custom_activation(x):
      # This will create a mask where values are between -1 and 1
      mask = (x >= -1) & (x <= 1)
      # For values between -1 and 1, apply x
      # For other values, apply x^1/3
      # Use where to combine results based on the mask
      return torch.where(mask, odd_pow(x, 1), odd_pow(x, 1/3))
  
  ```
  
  Note: Both ReLU and Tanh don't return NaNs, but I am looking for homogeneous activation functions $\sigma(ax) = g(a)\sigma(x)$.
  
  **Tags:** activation-functions
  
  > **Comments:**
  > **username_1**: Does the derivative cause the NaNs, or the normal activation function?  
  > **username_0**: @username_1 I think the derivative is what is causing the NaNs.  
  > **username_2**: You need to use a small constant to prevent exploding gradients. Please, see my answer.  
  > **username_3**: Please look at the gradients being backpropagated to see where the issue comes from.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-11. Score: 1
  
  **Update:** the issue is that your `odd_pow()` is *numerically unstable*, and so to fix the exploding gradients, which cause a NaN loss, you simply need to add a small constant when computing the power:
  
  ```
  def odd_pow(input, exponent, eps=1e-4):
      return input.sign() * (input.abs() + eps).pow(exponent)
  
  ```
  
  this ensures that the `pow` is not computed at zero, avoiding NaNs.
  
  ---
  
  **Note:** for reference I also keep my old answer, since also the activation $\sigma(x)$ suffers the same problem.
  
  The issue is that the function $\sigma(x) = x^{1/3}$ is zero for $x\le 0$, and so its derivative becomes NaN for negative $x$ and infinity for $x=0$ as shown here:
  [![enter image description here](https://i.sstatic.net/rdl0c.png)](https://i.sstatic.net/rdl0c.png)
  where the blue curve is the derivative of $\sigma(x)$, which is depicted in green.
  
  To avoid either NaN or exploding gradients, you need to add a small value (usually called an *epsilon*) to make the computation of the derivative more numerically stable. The following should work:
  
  ```
  def custom_activation(x, eps=1e-3):
      # set negative values to zero
      tmp = torch.nn.functional.relu(x)
      # add the small epsilon `eps`
      return torch.pow(tmp + eps, exponent=1 / 3.0)
  
  ```
  
  Notice that as you lower the `eps`, e.g. at 1e-5, the derivative would be larger. If you really need a small epsilon, then you should also consider to clip the individual gradients (there should be a `clip_grad` functionality related to the optimizer) to ensure stable training.
  
  > **Comments:**
  > **username_0**: The function $\sigma(x) = x^{1/3}$ is not zero for $x \leq 0$—I am taking the real root.  
  > **username_4**: The zero at $x=0$ is not the issue. It's the derivative is $\frac{1}{3}x^{-\frac{2}{3}}$ which is undefined at zero. With their intended $\sigma'$ function, which should simply have derivative of $1$ at $x=0$, the OP looks to have an implementation problem here, not a theoretical one.  
  > **username_0**: @username_4 It could be that torch isn't taking the real roots for the negative values of $x^{-2/3}$.  
  > **username_4**: @username_0: I would have thought that the `.abs()` would have stopped that issue. I cannot see clearly what is wrong, although I suspect one or more sub-components of your `custom_activation` function is confusing the auto-differentiation.  
  > **username_2**: @username_4 I implemented and tested the code I provide here in Tensorflow, on a simple MNIST classification problem and it trains without any issue  
  > **username_4**: In TensorFlow? Do you mean PyTorch? Either way, it is *not* the OP's activation function, and there's nothing wrong with the OP's function $\sigma'(x)$ in theory AFAICS.  
  > **username_4**: There's also nothing wrong with the activation function presented in this answer. However, the *reason* given for the OP's problem in the first sentence is incorrect, and the suggestion to use a different function is also not an answer to the original question.  
  > **username_2**: @username_4 Sorry but I made a bit of confusion here :) As you said, both $\sigma$ and $\sigma'$ are fine in theory. The issue, in practice, is numerical instability around the zero (I think). I've updated my answer and the issue should be fixed now  
  > **username_4**: I think you need to take a careful look at OP's definition of $\sigma'(x)$ The function is linear near the origin. You seem to be talking about the OP's first function -  at least a lot of your comments are correct when applied to that  
  > **username_2**: @username_4 Yes, I was referring to the first function, $\sigma(x)$ because as far as I understood $\sigma'$ was a replacement for that, since not working as intended. So I aimed at just fixing $\sigma$  
  
  ---
  
  