Answers.Body: <p>There is no benefit to using ReLU as the output activation of a neural network. As you said, the network will ignore training labels below zero and it will train on labels above zero as if no output activation were present. However, the problem you're describing can also occur for individual units of hidden layers, where ReLU activations are common. This is known as the dead ReLU problem. In practice, it is rarely a problem but it can be avoided with smooth rectifiers like ELU and Swish. Another interesting ideas is CRelu, which concatenates both positive and negative parts of the pre-activation, resulting in twice as many outputs, half of which always receive a non-zero gradient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T03:46:00.763Z
Answers.LastActivityDate: 2020-04-28T04:07:18.310Z
Answers.LastEditDate: 2020-04-28T04:07:18.310Z
Answers.UserDisplayName: danijar
Body: <p>The derivative of ReLU is 0 if its output is lower than 0 -  <span class="math-container">$d ReLU(x)/dReLU$</span> is <span class="math-container">$0$</span> if <span class="math-container">$x &lt; 0$</span>. Let's denote some net's output by <span class="math-container">$Out$</span>, so if this net's last layer is ReLU then we get that <span class="math-container">$dOut/dReLU$</span> is <span class="math-container">$0$</span> if <span class="math-container">$Out &lt; 0$</span>. Subsequently, for every parameter <span class="math-container">$p$</span> in the net we would get that <span class="math-container">$dOut/dp$</span> is <span class="math-container">$0$</span>. Does that mean that for every sample <span class="math-container">$x$</span> such that <span class="math-container">$Out(x) &lt; 0$</span> the net doesn't learn at all from that sample since the derivative for each parameter is <span class="math-container">$0$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-27T23:33:57.043Z
Comments.Text: I have never seen the ReLU being used as the activation function of the last layer.
Comments.UserDisplayName: nbro
Tags: <deep-learning><backpropagation><relu>
Title: Does net with ReLU not learn when output < 0?
UserDisplayName: Gilad Deutsch
ThreadText: # Does net with ReLU not learn when output < 0?
  
  (Asked by: username_0 on 2020-04-27)
  
  The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/dReLU$ is $0$ if $Out < 0$. Subsequently, for every parameter $p$ in the net we would get that $dOut/dp$ is $0$. Does that mean that for every sample $x$ such that $Out(x) < 0$ the net doesn't learn at all from that sample since the derivative for each parameter is $0$?
  
  **Tags:** deep-learning, backpropagation, relu
  
  > **Comments:**
  > **username_1**: I have never seen the ReLU being used as the activation function of the last layer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-28. Score: 1
  
  There is no benefit to using ReLU as the output activation of a neural network. As you said, the network will ignore training labels below zero and it will train on labels above zero as if no output activation were present. However, the problem you're describing can also occur for individual units of hidden layers, where ReLU activations are common. This is known as the dead ReLU problem. In practice, it is rarely a problem but it can be avoided with smooth rectifiers like ELU and Swish. Another interesting ideas is CRelu, which concatenates both positive and negative parts of the pre-activation, resulting in twice as many outputs, half of which always receive a non-zero gradient.
  
  
  ---
  
  