Answers.Body: <blockquote>
  <p>This idea seems pretty convincing</p>
  </blockquote>
  <p>Indeed, you don't have to use the exact gradient of the activation function during the backward step.</p>
  <p>The gradient of the activation function is a multiplicative part of the gradient of the loss. However, the exact gradient of the loss is almost never used. It is used neither in stochastic gradient descent, mini-batch, training with momentum, Adam, Newton nor in most other methods.</p>
  <p>As only the approximate gradient of the loss is used in practice, we don't have to keep any part of it exact, so there is no need to keep the gradient of the activation function exact. It can be taken approximately, and any approximation is fine (including from the paper you referenced) as long as the loss decreases during training.</p>
  <blockquote>
  <p>So why is it that vanishing gradients have not been addressed in this way in conventional deep learning frameworks such as Pytorch or Tensorflow?</p>
  </blockquote>
  <p>This can be done in PyTorch and TensorFlow. You can modify the gradient by specifying the <code>backward()</code> method in PyTorch or <code>grad()</code> method in TensorFlow:</p>
  <p><a href="https://www.kaggle.com/code/aleksandradeis/extending-pytorch-with-custom-activation-functions/notebook" rel="nofollow noreferrer">Extending PyTorch with Custom Activation Functions</a></p>
  <p><a href="https://www.tensorflow.org/api_docs/python/tf/custom_gradient" rel="nofollow noreferrer">tf.custom_gradient</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-11T12:01:38.163Z
Answers.Text: Thank you for your reply. Your point about SGD makes sense to me. I also appreciate you mentioning how one can manually adapt the backward/grad method. However, I think you did not directly answer my second question: why are artificial derivatives not used as much? Is it because people simply use batch normalisation when they observe vanishing gradients?
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-11T12:55:41.290Z
Answers.Text: I am not sure they are not used. I showed that it's easy to use them with popular frameworks, although batch norm is even easier in TensorFlow and PyTorch: 1 line of code instead of a few, maybe that's why. Previously, I encountered their usage in discussions of what to do with discontinuous activation functions (including in stackexchange and kaggle) long before the paper you refer to was published. If that problem can be solved by using approximate gradients instead of exact ones, I suppose that many other problems of activation functions including vanishing gradients can be solved similarly
Answers.UserDisplayName: Vladislav Gladkikh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T08:15:31.120Z
Answers.LastActivityDate: 2022-10-10T08:15:31.120Z
Answers.UserDisplayName: Vladislav Gladkikh
Body: <p>While looking into the vanishing gradient problem, I came across a paper (<a href="https://ieeexplore.ieee.org/abstract/document/9336631" rel="nofollow noreferrer">https://ieeexplore.ieee.org/abstract/document/9336631</a>) that used artificial derivatives in lieu of the real derivatives. For a visualization, see the attached image: <a href="https://i.sstatic.net/vRENQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vRENQ.png" alt="" /></a></p>
  <p>As you can see, the artificial derivative of the ReLU is the sigmoid function, which smoothes out the derivative to circumvent the dying ReLU problem. The authors apply similar smoothing operations for other activation functions such as the sigmoid. This idea seems pretty convincing, and it surprises me that it has not been used thus far, especially considering there is no additional computational cost in replacing the derivatives, as the authors claim. So why is it that vanishing gradients have not been addressed in this way in conventional deep learning frameworks such as Pytorch or Tensorflow?</p>
  
Tags: <activation-functions><vanishing-gradient-problem>
Title: Why aren't artificial derivatives used more often to solve the vanishing gradient problem?
UserDisplayName: postnubilaphoebus
ThreadText: # Why aren't artificial derivatives used more often to solve the vanishing gradient problem?
  
  (Asked by: username_0 on 2022-10-09)
  
  While looking into the vanishing gradient problem, I came across a paper (<https://ieeexplore.ieee.org/abstract/document/9336631>) that used artificial derivatives in lieu of the real derivatives. For a visualization, see the attached image: [![](https://i.sstatic.net/vRENQ.png)](https://i.sstatic.net/vRENQ.png)
  
  As you can see, the artificial derivative of the ReLU is the sigmoid function, which smoothes out the derivative to circumvent the dying ReLU problem. The authors apply similar smoothing operations for other activation functions such as the sigmoid. This idea seems pretty convincing, and it surprises me that it has not been used thus far, especially considering there is no additional computational cost in replacing the derivatives, as the authors claim. So why is it that vanishing gradients have not been addressed in this way in conventional deep learning frameworks such as Pytorch or Tensorflow?
  
  **Tags:** activation-functions, vanishing-gradient-problem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-10. Score: 4 | Accepted answer
  
  > This idea seems pretty convincing
  
  Indeed, you don't have to use the exact gradient of the activation function during the backward step.
  
  The gradient of the activation function is a multiplicative part of the gradient of the loss. However, the exact gradient of the loss is almost never used. It is used neither in stochastic gradient descent, mini-batch, training with momentum, Adam, Newton nor in most other methods.
  
  As only the approximate gradient of the loss is used in practice, we don't have to keep any part of it exact, so there is no need to keep the gradient of the activation function exact. It can be taken approximately, and any approximation is fine (including from the paper you referenced) as long as the loss decreases during training.
  
  > So why is it that vanishing gradients have not been addressed in this way in conventional deep learning frameworks such as Pytorch or Tensorflow?
  
  This can be done in PyTorch and TensorFlow. You can modify the gradient by specifying the `backward()` method in PyTorch or `grad()` method in TensorFlow:
  
  [Extending PyTorch with Custom Activation Functions](https://www.kaggle.com/code/aleksandradeis/extending-pytorch-with-custom-activation-functions/notebook)
  
  [tf.custom\_gradient](https://www.tensorflow.org/api_docs/python/tf/custom_gradient)
  
  > **Comments:**
  > **username_0**: Thank you for your reply. Your point about SGD makes sense to me. I also appreciate you mentioning how one can manually adapt the backward/grad method. However, I think you did not directly answer my second question: why are artificial derivatives not used as much? Is it because people simply use batch normalisation when they observe vanishing gradients?  
  > **username_1**: I am not sure they are not used. I showed that it's easy to use them with popular frameworks, although batch norm is even easier in TensorFlow and PyTorch: 1 line of code instead of a few, maybe that's why. Previously, I encountered their usage in discussions of what to do with discontinuous activation functions (including in stackexchange and kaggle) long before the paper you refer to was published. If that problem can be solved by using approximate gradients instead of exact ones, I suppose that many other problems of activation functions including vanishing gradients can be solved similarly  
  
  ---
  
  