Answers.Body: <p>Creating custom gradient for <code>tf.abs</code> may solve the problem:</p>
  <pre><code>@tf.custom_gradient
  def abs_with_grad(x):
    y = tf.abs(x);
  
    def grad(div): # Derivation intermediate value
      g = 1; # Use 1 to make the chain rule just skip abs
      return div*g;
  
    return y,grad;
  </code></pre>
  <p>Use <code>1</code> as above to skip thru' abs or, use the actual abs grad (<a href="https://ai.stackexchange.com/users/69830/samuel-k">Samual K</a>):</p>
  <pre><code>g = tf.where(x&lt;0, -1, 1) #now the gradient at 0 would be one. This way u dont have dead  weights.
  # With/without:
  g = tf.where(x==0, 0, g) #if you realy want the gradient 0 at 0 add  this.
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-18T10:54:53.127Z
Answers.LastActivityDate: 2023-03-28T07:04:52.420Z
Answers.LastEditDate: 2023-03-28T07:04:52.420Z
Answers.UserDisplayName: Dan D.
Answers.Body: <p>By convention, the <span class="math-container">$\mathrm{ReLU}$</span> activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for <code>tf.nn.relu</code>. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.</p>
  <p>In a way this is compatible with the Python philosophy that <em>explicit is better than implicit</em>. If you mean to use <span class="math-container">$\mathrm{ReLU}$</span>, it's probably best to use <code>tf.nn.relu</code> if it is suitable for your use case.</p>
  <p>[1] Vinod Nair and Geoffrey Hinton. <em>Rectified Linear Units Improve Restricted Boltzmann Machines</em>. ICML'10 (2010). <a href="https://www.cs.toronto.edu/%7Efritz/absps/reluICML.pdf" rel="nofollow noreferrer">URL</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-17T22:25:19.023Z
Answers.Text: Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0?
Answers.UserDisplayName: zedsdead
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-18T09:45:11.563Z
Answers.Text: @zedsdead Could you just substitute in `tf.nn.relu` in the place you're using `tf.abs`?
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-20T09:47:06.017Z
Answers.Text: How about `relu(x) + relu(-x)`? :)
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-23T16:36:48.480Z
Answers.Text: @htl Well, they are not the same thing but NikoNyrh’s answer below is probably the easiest solution.
Answers.UserDisplayName: zedsdead
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-17T19:31:30.030Z
Answers.LastActivityDate: 2021-02-17T19:31:30.030Z
Answers.UserDisplayName: htl
Body: <p>I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.</p>
  
Tags: <tensorflow><backpropagation><relu><gradient>
Title: Why is tf.abs non-differentiable in Tensorflow?
UserDisplayName: zedsdead
ThreadText: # Why is tf.abs non-differentiable in Tensorflow?
  
  (Asked by: username_0 on 2021-02-17)
  
  I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.
  
  **Tags:** tensorflow, backpropagation, relu, gradient
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-02-17. Score: 4 | Accepted answer
  
  By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for `tf.nn.relu`. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.
  
  In a way this is compatible with the Python philosophy that *explicit is better than implicit*. If you mean to use $\mathrm{ReLU}$, it's probably best to use `tf.nn.relu` if it is suitable for your use case.
  
  [1] Vinod Nair and Geoffrey Hinton. *Rectified Linear Units Improve Restricted Boltzmann Machines*. ICML'10 (2010). [URL](https://www.cs.toronto.edu/%7Efritz/absps/reluICML.pdf).
  
  > **Comments:**
  > **username_0**: Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0?  
  > **username_2**: @username_0 Could you just substitute in `tf.nn.relu` in the place you're using `tf.abs`?  
  > **username_3**: How about `relu(x) + relu(-x)`? :)  
  > **username_0**: @username_2 Well, they are not the same thing but username_3’s answer below is probably the easiest solution.  
  
  ---
  
  ### Answer by username_1 on 2021-02-18. Score: 2
  
  Creating custom gradient for `tf.abs` may solve the problem:
  
  ```
  @tf.custom_gradient
  def abs_with_grad(x):
    y = tf.abs(x);
  
    def grad(div): # Derivation intermediate value
      g = 1; # Use 1 to make the chain rule just skip abs
      return div*g;
  
    return y,grad;
  
  ```
  
  Use `1` as above to skip thru' abs or, use the actual abs grad ([Samual K](https://ai.stackexchange.com/users/69830/samuel-k)):
  
  ```
  g = tf.where(x<0, -1, 1) #now the gradient at 0 would be one. This way u dont have dead  weights.
  # With/without:
  g = tf.where(x==0, 0, g) #if you realy want the gradient 0 at 0 add  this.
  
  ```
  
  
  ---
  
  