Answers.Body: <p>Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0. </p>
  
  <p>Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to directly derive gradients of that loss with respect to the softmax input. The result is very elegant and extremely easy to implement. Try to derive that yourself!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-17T08:10:36.367Z
Answers.Text: Thanks so much! I went through the calculus and I got the SoftMax minus 1, is that correct? Also for implementation, do I simply use this gradient in backpropagation and feed it back through the network like normal?
Answers.UserDisplayName: Archie Shahidullah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-17T17:36:47.197Z
Answers.Text: You are on the right path! Note however that the negative log-likelihood depends on correct labels, while there are no labels in your expression for the NLL-softmax gradient. We have a contradiction: a supervised learning which does not use supervision. Try to correct that!
Answers.UserDisplayName: ssegvic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T12:46:22.530Z
Answers.Text: You say "For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.", but you don't explain why that must be the case, from a calculus perspective. I think this answer would improve if you explain that more in detail.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T13:34:25.280Z
Answers.Text: @nbro I assume that if OP is able to write a backprop for tanh, then he should know what is the derivative of ReLU.
Answers.UserDisplayName: ssegvic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T13:47:12.490Z
Answers.Text: @ssegvic How does the derivative of ReLU come from the derivative of tanh? It's just a convention that the derivative of ReLU at zero is 0, given that the left and right derivatives at zero are different. You don't explain this in your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T18:29:57.543Z
Answers.Text: If you know how to differentiate tanh, then you know how to differentiate ReLU. 
  
  My answer is agnostic with repect to the derivative of ReLU in zero.
Answers.UserDisplayName: ssegvic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T10:41:08.120Z
Answers.LastActivityDate: 2019-02-16T15:06:03.950Z
Answers.LastEditDate: 2019-02-16T15:06:03.950Z
Answers.UserDisplayName: ssegvic
Body: <p>I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation functions such as <span class="math-container">$\tanh$</span> and the sigmoid function. However, these are normalised in their outputs. A function like ReLU is unbounded so its outputs can blow up really fast. In my understanding, a classification layer, usually using the SoftMax function, is added at the end to squash the outputs between 0 and 1. </p>
  
  <p>How does backpropagation work with this? Do I just treat the SoftMax function as another activation function and compute its gradient? If so, what is that gradient and how would I implement it? If not, how does the training process work? If possible, a pseudocode answer is preferred.</p>
  
Tags: <neural-networks><backpropagation><relu>
Title: How does backpropagation with unbounded activation functions such as ReLU work?
UserDisplayName: Archie Shahidullah
ThreadText: # How does backpropagation with unbounded activation functions such as ReLU work?
  
  (Asked by: username_0 on 2019-02-16)
  
  I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation functions such as $\tanh$ and the sigmoid function. However, these are normalised in their outputs. A function like ReLU is unbounded so its outputs can blow up really fast. In my understanding, a classification layer, usually using the SoftMax function, is added at the end to squash the outputs between 0 and 1.
  
  How does backpropagation work with this? Do I just treat the SoftMax function as another activation function and compute its gradient? If so, what is that gradient and how would I implement it? If not, how does the training process work? If possible, a pseudocode answer is preferred.
  
  **Tags:** neural-networks, backpropagation, relu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-16. Score: 3 | Accepted answer
  
  Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.
  
  Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to directly derive gradients of that loss with respect to the softmax input. The result is very elegant and extremely easy to implement. Try to derive that yourself!
  
  > **Comments:**
  > **username_0**: Thanks so much! I went through the calculus and I got the SoftMax minus 1, is that correct? Also for implementation, do I simply use this gradient in backpropagation and feed it back through the network like normal?  
  > **username_1**: You are on the right path! Note however that the negative log-likelihood depends on correct labels, while there are no labels in your expression for the NLL-softmax gradient. We have a contradiction: a supervised learning which does not use supervision. Try to correct that!  
  > **username_2**: You say "For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.", but you don't explain why that must be the case, from a calculus perspective. I think this answer would improve if you explain that more in detail.  
  > **username_1**: @nbro I assume that if OP is able to write a backprop for tanh, then he should know what is the derivative of ReLU.  
  > **username_2**: @ssegvic How does the derivative of ReLU come from the derivative of tanh? It's just a convention that the derivative of ReLU at zero is 0, given that the left and right derivatives at zero are different. You don't explain this in your answer.  
  > **username_1**: If you know how to differentiate tanh, then you know how to differentiate ReLU.   My answer is agnostic with repect to the derivative of ReLU in zero.  
  
  ---
  
  