<p><strong>Update:</strong> the issue is that your <code>odd_pow()</code> is <em>numerically unstable</em>, and so to fix the exploding gradients, which cause a NaN loss, you simply need to add a small constant when computing the power:</p>
<pre><code>def odd_pow(input, exponent, eps=1e-4):
    return input.sign() * (input.abs() + eps).pow(exponent)
</code></pre>
<p>this ensures that the <code>pow</code> is not computed at zero, avoiding NaNs.</p>
<hr />
<p><strong>Note:</strong> for reference I also keep my old answer, since also the activation <span class="math-container">$\sigma(x)$</span> suffers the same problem.</p>
<p>The issue is that the function <span class="math-container">$\sigma(x) = x^{1/3}$</span> is zero for <span class="math-container">$x\le 0$</span>, and so its derivative becomes NaN for negative <span class="math-container">$x$</span> and infinity for <span class="math-container">$x=0$</span> as shown here:
<a href="https://i.sstatic.net/rdl0c.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rdl0c.png" alt="enter image description here" /></a>
where the blue curve is the derivative of <span class="math-container">$\sigma(x)$</span>, which is depicted in green.</p>
<p>To avoid either NaN or exploding gradients, you need to add a small value (usually called an <em>epsilon</em>) to make the computation of the derivative more numerically stable. The following should work:</p>
<pre><code>def custom_activation(x, eps=1e-3):
    # set negative values to zero
    tmp = torch.nn.functional.relu(x)
    # add the small epsilon `eps`
    return torch.pow(tmp + eps, exponent=1 / 3.0)
</code></pre>
<p>Notice that as you lower the <code>eps</code>, e.g. at 1e-5, the derivative would be larger. If you really need a small epsilon, then you should also consider to clip the individual gradients (there should be a <code>clip_grad</code> functionality related to the optimizer) to ensure stable training.</p>

CC BY-SA 4.0
2023-09-11T13:54:35.463Z
The function $\sigma(x) = x^{1/3}$ is not zero for $x \leq 0$—I am taking the real root.
user572780
CC BY-SA 4.0
2023-09-11T14:02:37.977Z
The zero at $x=0$ is not the issue. It's the derivative is $\frac{1}{3}x^{-\frac{2}{3}}$ which is undefined at zero. With their intended $\sigma'$ function, which should simply have derivative of $1$ at $x=0$, the OP looks to have an implementation problem here, not a theoretical one.
Neil Slater
CC BY-SA 4.0
2023-09-11T16:17:02.247Z
@NeilSlater It could be that torch isn't taking the real roots for the negative values of $x^{-2/3}$.
user572780
CC BY-SA 4.0
2023-09-11T18:02:18.610Z
@user572780: I would have thought that the `.abs()` would have stopped that issue. I cannot see clearly what is wrong, although I suspect one or more sub-components of your `custom_activation` function is confusing the auto-differentiation.
Neil Slater
CC BY-SA 4.0
2023-09-12T13:00:17.420Z
@NeilSlater I implemented and tested the code I provide here in Tensorflow, on a simple MNIST classification problem and it trains without any issue
Luca Anzalone
CC BY-SA 4.0
2023-09-12T13:44:58.490Z
In TensorFlow? Do you mean PyTorch? Either way, it is *not* the OP's activation function, and there's nothing wrong with the OP's function $\sigma'(x)$ in theory AFAICS.
Neil Slater
CC BY-SA 4.0
2023-09-12T13:52:05.057Z
There's also nothing wrong with the activation function presented in this answer. However, the *reason* given for the OP's problem in the first sentence is incorrect, and the suggestion to use a different function is also not an answer to the original question.
Neil Slater
CC BY-SA 4.0
2023-09-13T08:20:49.400Z
@NeilSlater Sorry but I made a bit of confusion here :) As you said, both $\sigma$ and $\sigma'$ are fine in theory. The issue, in practice, is numerical instability around the zero (I think). I've updated my answer and the issue should be fixed now
Luca Anzalone
CC BY-SA 4.0
2023-09-13T12:03:12.520Z
I think you need to take a careful look at OP's definition of $\sigma'(x)$ The function is linear near the origin. You seem to be talking about the OP's first function -  at least a lot of your comments are correct when applied to that
Neil Slater
CC BY-SA 4.0
2023-09-14T15:31:22.757Z
@NeilSlater Yes, I was referring to the first function, $\sigma(x)$ because as far as I understood $\sigma'$ was a replacement for that, since not working as intended. So I aimed at just fixing $\sigma$
Luca Anzalone
CC BY-SA 4.0
2023-09-11T13:45:01.490Z
2023-09-13T08:19:04.297Z
2023-09-13T08:19:04.297Z
Luca Anzalone
<p>I implemented the following activation function:
<span class="math-container">$$\sigma(x) = x^{1/3},$$</span>
which returns NaNs after some epochs. I think this is due to the derivative exploding close to <span class="math-container">$0$</span>. To fix this issue, I implemented a modified activation function:
<span class="math-container">$$
\sigma'(x) = 
\begin{cases} 
 x, &amp; \text{if } -1 \leq x \leq 1, \\
 x^{1/3}, &amp; \text{otherwise.}
\end{cases}
$$</span>
Unfortunately, <span class="math-container">$\sigma'$</span> suffers from the same issue. Why is that?</p>
<p>Here is my code:</p>
<pre><code>def odd_pow(input, exponent):
    return input.sign() * input.abs().pow(exponent)

def custom_activation(x):
    # This will create a mask where values are between -1 and 1
    mask = (x &gt;= -1) &amp; (x &lt;= 1)
    # For values between -1 and 1, apply x
    # For other values, apply x^1/3
    # Use where to combine results based on the mask
    return torch.where(mask, odd_pow(x, 1), odd_pow(x, 1/3))
</code></pre>
<p>Note: Both ReLU and Tanh don't return NaNs, but I am looking for homogeneous activation functions <span class="math-container">$\sigma(ax) = g(a)\sigma(x)$</span>.</p>

CC BY-SA 4.0
2023-09-11T13:13:14.470Z
Does the derivative cause the NaNs, or the normal activation function?
Robin van Hoorn
CC BY-SA 4.0
2023-09-11T13:13:55.197Z
@RobinvanHoorn I think the derivative is what is causing the NaNs.
user572780
CC BY-SA 4.0
2023-09-13T08:21:31.173Z
You need to use a small constant to prevent exploding gradients. Please, see my answer.
Luca Anzalone
CC BY-SA 4.0
2023-09-13T14:10:12.710Z
Please look at the gradients being backpropagated to see where the issue comes from.
Lelouch
<activation-functions>
Why does my activation function cause NaNs?
user572780
# Why does my activation function cause NaNs?

(Asked by: username_0 on 2023-09-11)

I implemented the following activation function:
$$\sigma(x) = x^{1/3},$$
which returns NaNs after some epochs. I think this is due to the derivative exploding close to $0$. To fix this issue, I implemented a modified activation function:
$$
\sigma'(x) = 
\begin{cases} 
 x, & \text{if } -1 \leq x \leq 1, \\
 x^{1/3}, & \text{otherwise.}
\end{cases}
$$
Unfortunately, $\sigma'$ suffers from the same issue. Why is that?
Here is my code:
def odd_pow(input, exponent):
    return input.sign() * input.abs().pow(exponent)

def custom_activation(x):
    # This will create a mask where values are between -1 and 1
    mask = (x >= -1) & (x <= 1)
    # For values between -1 and 1, apply x
    # For other values, apply x^1/3
    # Use where to combine results based on the mask
    return torch.where(mask, odd_pow(x, 1), odd_pow(x, 1/3))

Note: Both ReLU and Tanh don't return NaNs, but I am looking for homogeneous activation functions $\sigma(ax) = g(a)\sigma(x)$.

**Tags:** activation-functions

> **Comments:**
> **username_1**: Does the derivative cause the NaNs, or the normal activation function?  
> **username_0**: @RobinvanHoorn I think the derivative is what is causing the NaNs.  
> **username_2**: You need to use a small constant to prevent exploding gradients. Please, see my answer.  
> **username_3**: Please look at the gradients being backpropagated to see where the issue comes from.  

---

## 1 answers

### Answer by username_2 on 2023-09-11. Score: 1

Update: the issue is that your odd_pow() is numerically unstable, and so to fix the exploding gradients, which cause a NaN loss, you simply need to add a small constant when computing the power:
def odd_pow(input, exponent, eps=1e-4):
    return input.sign() * (input.abs() + eps).pow(exponent)

this ensures that the pow is not computed at zero, avoiding NaNs.

Note: for reference I also keep my old answer, since also the activation $\sigma(x)$ suffers the same problem.
The issue is that the function $\sigma(x) = x^{1/3}$ is zero for $x\le 0$, and so its derivative becomes NaN for negative $x$ and infinity for $x=0$ as shown here:

where the blue curve is the derivative of $\sigma(x)$, which is depicted in green.
To avoid either NaN or exploding gradients, you need to add a small value (usually called an epsilon) to make the computation of the derivative more numerically stable. The following should work:
def custom_activation(x, eps=1e-3):
    # set negative values to zero
    tmp = torch.nn.functional.relu(x)
    # add the small epsilon `eps`
    return torch.pow(tmp + eps, exponent=1 / 3.0)

Notice that as you lower the eps, e.g. at 1e-5, the derivative would be larger. If you really need a small epsilon, then you should also consider to clip the individual gradients (there should be a clip_grad functionality related to the optimizer) to ensure stable training.

> **Comments:**
> **username_0**: The function $\sigma(x) = x^{1/3}$ is not zero for $x \leq 0$—I am taking the real root.  
> **username_4**: The zero at $x=0$ is not the issue. It's the derivative is $\frac{1}{3}x^{-\frac{2}{3}}$ which is undefined at zero. With their intended $\sigma'$ function, which should simply have derivative of $1$ at $x=0$, the OP looks to have an implementation problem here, not a theoretical one.  
> **username_0**: @NeilSlater It could be that torch isn't taking the real roots for the negative values of $x^{-2/3}$.  
> **username_4**: @username_0: I would have thought that the `.abs()` would have stopped that issue. I cannot see clearly what is wrong, although I suspect one or more sub-components of your `custom_activation` function is confusing the auto-differentiation.  
> **username_2**: @NeilSlater I implemented and tested the code I provide here in Tensorflow, on a simple MNIST classification problem and it trains without any issue  
> **username_4**: In TensorFlow? Do you mean PyTorch? Either way, it is *not* the OP's activation function, and there's nothing wrong with the OP's function $\sigma'(x)$ in theory AFAICS.  
> **username_4**: There's also nothing wrong with the activation function presented in this answer. However, the *reason* given for the OP's problem in the first sentence is incorrect, and the suggestion to use a different function is also not an answer to the original question.  
> **username_2**: @NeilSlater Sorry but I made a bit of confusion here :) As you said, both $\sigma$ and $\sigma'$ are fine in theory. The issue, in practice, is numerical instability around the zero (I think). I've updated my answer and the issue should be fixed now  
> **username_4**: I think you need to take a careful look at OP's definition of $\sigma'(x)$ The function is linear near the origin. You seem to be talking about the OP's first function -  at least a lot of your comments are correct when applied to that  
> **username_2**: @NeilSlater Yes, I was referring to the first function, $\sigma(x)$ because as far as I understood $\sigma'$ was a replacement for that, since not working as intended. So I aimed at just fixing $\sigma$  

---

