Answers.Body: <p>The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too.</p>
  
  <p>Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input.
  Remember that only layer outputs, not their weights, are passed through ReLu, so, weights of a model could contain negative values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T05:28:27.080Z
Answers.LastActivityDate: 2020-05-16T07:24:04.663Z
Answers.LastEditDate: 2020-05-16T07:24:04.663Z
Answers.UserDisplayName: SpiderRico
Body: <p>When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU? </p>
  
  <p>As far as I know, features are never negative or converted to negative in any other type of layer. </p>
  
  <p>Is it that we can use the RELU with a different "inflection" point than zero, so we can make the neuron start describing a lineal response just after this "new zero"? </p>
  
Tags: <convolutional-neural-networks><linear-regression><relu><non-linear-regression>
Title: If features are always positives, why do we use RELU activation functions?
UserDisplayName: sujeto1
ThreadText: # If features are always positives, why do we use RELU activation functions?
  
  (Asked by: username_0 on 2020-05-16)
  
  When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU?
  
  As far as I know, features are never negative or converted to negative in any other type of layer.
  
  Is it that we can use the RELU with a different "inflection" point than zero, so we can make the neuron start describing a lineal response just after this "new zero"?
  
  **Tags:** convolutional-neural-networks, linear-regression, relu, non-linear-regression
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-16. Score: 3
  
  The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too.
  
  Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input.
  Remember that only layer outputs, not their weights, are passed through ReLu, so, weights of a model could contain negative values.
  
  
  ---
  
  