Answers.Body: <p>You are right about the fact that we use nonlinearity for classes that cant be seperated by a straight line. If you think about curves,as we know from the calculus, you can approximate a nonlinear function with lines. If you have infinite amount of lines you can mimic exactly the same function. When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents. And ReLU solves the vanishing gradient problem so they are superior to sigmoid in my opinion too.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-19T08:38:18.147Z
Answers.Text: Thanks for the answer!
  Quick follow up, I don't understand how it solves the vanishing gradient problem since there are still infinitly many points where the derivative of ReLU is zero.
Answers.UserDisplayName: Emil Wormbs
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-19T08:58:14.313Z
Answers.Text: https://stats.stackexchange.com/q/176794
Answers.UserDisplayName: Ege Keyvan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-19T05:13:14.290Z
Answers.Text: Ege is correct. The dying relu problem only occurs in very specific instances(dying relu is where a neuron always outputs the same value). For example, when using gradient descent and backpropagation the forward pass can clamp the weights to zero. Karpathy has some good info about that in his backprop [walkthrough](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b)
Answers.UserDisplayName: hisairnessag3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-17T15:05:34.363Z
Answers.Text: "When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents": please, clarify.
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-18T15:32:19.283Z
Answers.LastActivityDate: 2018-01-18T15:32:19.283Z
Answers.UserDisplayName: Ege Keyvan
Body: <p>I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as <a href="https://www.quora.com/How-does-the-ReLu-solve-the-vanishing-gradient-problem" rel="nofollow noreferrer">ReLU solves the vanishing gradient problem</a>. However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all? </p>
  
  <p>Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x&lt;0 = 0</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-17T23:37:29.370Z
Comments.Text: Your premise doesn't follow your conclusion. It seems you're implying that we use ReLU on the model's weights. We multiply the weights by the input which might not necessarily yield values [0, 1].
Comments.UserDisplayName: Daniel
Tags: <machine-learning><image-recognition><tensorflow>
Title: If neurons are only defined for values between 0 and 1, how does ReLU differ from the identity?
UserDisplayName: Emil Wormbs
ThreadText: # If neurons are only defined for values between 0 and 1, how does ReLU differ from the identity?
  
  (Asked by: username_0 on 2018-01-18)
  
  I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as [ReLU solves the vanishing gradient problem](https://www.quora.com/How-does-the-ReLu-solve-the-vanishing-gradient-problem). However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all?
  
  Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x<0 = 0
  
  **Tags:** machine-learning, image-recognition, tensorflow
  > **Comments:**
  > **username_1**: Your premise doesn't follow your conclusion. It seems you're implying that we use ReLU on the model's weights. We multiply the weights by the input which might not necessarily yield values [0, 1].  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-01-18. Score: -1
  
  You are right about the fact that we use nonlinearity for classes that cant be seperated by a straight line. If you think about curves,as we know from the calculus, you can approximate a nonlinear function with lines. If you have infinite amount of lines you can mimic exactly the same function. When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents. And ReLU solves the vanishing gradient problem so they are superior to sigmoid in my opinion too.
  
  > **Comments:**
  > **username_0**: Thanks for the answer! Quick follow up, I don't understand how it solves the vanishing gradient problem since there are still infinitly many points where the derivative of ReLU is zero.  
  > **username_2**: https://stats.stackexchange.com/q/176794  
  > **username_3**: Ege is correct. The dying relu problem only occurs in very specific instances(dying relu is where a neuron always outputs the same value). For example, when using gradient descent and backpropagation the forward pass can clamp the weights to zero. Karpathy has some good info about that in his backprop [walkthrough](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b)  
  > **username_4**: "When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents": please, clarify.  
  
  ---
  
  