<p>Creating custom gradient for <code>tf.abs</code> may solve the problem:</p>
<pre><code>@tf.custom_gradient
def abs_with_grad(x):
  y = tf.abs(x);

  def grad(div): # Derivation intermediate value
    g = 1; # Use 1 to make the chain rule just skip abs
    return div*g;

  return y,grad;
</code></pre>
<p>Use <code>1</code> as above to skip thru' abs or, use the actual abs grad (<a href="https://ai.stackexchange.com/users/69830/samuel-k">Samual K</a>):</p>
<pre><code>g = tf.where(x&lt;0, -1, 1) #now the gradient at 0 would be one. This way u dont have dead  weights.
# With/without:
g = tf.where(x==0, 0, g) #if you realy want the gradient 0 at 0 add  this.
</code></pre>

CC BY-SA 4.0
2021-02-18T10:54:53.127Z
2023-03-28T07:04:52.420Z
2023-03-28T07:04:52.420Z
Dan D.
<p>By convention, the <span class="math-container">$\mathrm{ReLU}$</span> activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for <code>tf.nn.relu</code>. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.</p>
<p>In a way this is compatible with the Python philosophy that <em>explicit is better than implicit</em>. If you mean to use <span class="math-container">$\mathrm{ReLU}$</span>, it's probably best to use <code>tf.nn.relu</code> if it is suitable for your use case.</p>
<p>[1] Vinod Nair and Geoffrey Hinton. <em>Rectified Linear Units Improve Restricted Boltzmann Machines</em>. ICML'10 (2010). <a href="https://www.cs.toronto.edu/%7Efritz/absps/reluICML.pdf" rel="nofollow noreferrer">URL</a>.</p>

CC BY-SA 4.0
2021-02-17T22:25:19.023Z
Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0?
zedsdead
CC BY-SA 4.0
2021-02-18T09:45:11.563Z
@zedsdead Could you just substitute in `tf.nn.relu` in the place you're using `tf.abs`?
htl
CC BY-SA 4.0
2021-02-20T09:47:06.017Z
How about `relu(x) + relu(-x)`? :)
NikoNyrh
CC BY-SA 4.0
2021-02-23T16:36:48.480Z
@htl Well, they are not the same thing but NikoNyrh’s answer below is probably the easiest solution.
zedsdead
CC BY-SA 4.0
2021-02-17T19:31:30.030Z
2021-02-17T19:31:30.030Z
htl
<p>I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.</p>

<tensorflow><backpropagation><relu><gradient>
Why is tf.abs non-differentiable in Tensorflow?
zedsdead
# Why is tf.abs non-differentiable in Tensorflow?

(Asked by: username_0 on 2021-02-17)

I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.

**Tags:** tensorflow, backpropagation, relu, gradient


---

## 2 answers

### Answer by username_2 on 2021-02-17. Score: 4 | Accepted answer

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for `tf.nn.relu`. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.

In a way this is compatible with the Python philosophy that *explicit is better than implicit*. If you mean to use $\mathrm{ReLU}$, it's probably best to use `tf.nn.relu` if it is suitable for your use case.

[1] Vinod Nair and Geoffrey Hinton. *Rectified Linear Units Improve Restricted Boltzmann Machines*. ICML'10 (2010). [URL](https://www.cs.toronto.edu/%7Efritz/absps/reluICML.pdf).

> **Comments:**
> **username_0**: Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0?  
> **username_2**: @username_0 Could you just substitute in `tf.nn.relu` in the place you're using `tf.abs`?  
> **username_3**: How about `relu(x) + relu(-x)`? :)  
> **username_0**: @username_2 Well, they are not the same thing but username_3’s answer below is probably the easiest solution.  

---

### Answer by username_1 on 2021-02-18. Score: 2

Creating custom gradient for tf.abs may solve the problem:
@tf.custom_gradient
def abs_with_grad(x):
  y = tf.abs(x);

  def grad(div): # Derivation intermediate value
    g = 1; # Use 1 to make the chain rule just skip abs
    return div*g;

  return y,grad;

Use 1 as above to skip thru' abs or, use the actual abs grad (Samual K):
g = tf.where(x<0, -1, 1) #now the gradient at 0 would be one. This way u dont have dead  weights.
# With/without:
g = tf.where(x==0, 0, g) #if you realy want the gradient 0 at 0 add  this.


---

