Answers.Body: <p>Applying a ReLU activation function to a non-negative number is the same as applying the identity activation function.</p>
  
  <p>What you may wish to do is apply <span class="math-container">$\ln(|\vec{x}|)$</span>, <span class="math-container">$\arctan(x[0], x[1])$</span>, ..., <span class="math-container">$\arctan(x[0], x[N-1])$</span>, where <span class="math-container">$N$</span> is the number of dimensions to <span class="math-container">$\vec{x}$</span>, and pass this directly into a parameter matrix multiplication (which involves deliberately using an identity activation in the input layer in some frameworks). This typically improves speed, accuracy, and reliability of convergence when the magnitudes exhibit a distribution close to an exponential of Gaussian and the directional components are near Gaussian when expressed in radians.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-28T01:11:06.487Z
Answers.LastActivityDate: 2018-12-28T01:11:06.487Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>Suppose I want to predict the position of a sensor based on its reading.</p>
  
  <p>I can first predict the unit vector and predict the distance to be multiplied to this vector.
  And I know that distance will never be negative because all the negative parts are inside unit vector already.</p>
  
  <p><strong>Should I apply ReLU to the distance before multiplying it to the unit vector?</strong></p>
  
  <p>I'm thinking that this can be helpful to eliminate the network from needing too much training data by restricting the output ranges the network could give. But I also think that it could make the learning slower when the ReLU unit dies (value=0) so the gradient doesn't flow properly somehow.</p>
  
Tags: <neural-networks><deep-learning><tensorflow><keras>
Title: Should I apply ReLU to non negative output?
UserDisplayName: offchan
ThreadText: # Should I apply ReLU to non negative output?
  
  (Asked by: username_0 on 2018-12-25)
  
  Suppose I want to predict the position of a sensor based on its reading.
  
  I can first predict the unit vector and predict the distance to be multiplied to this vector.
  And I know that distance will never be negative because all the negative parts are inside unit vector already.
  
  **Should I apply ReLU to the distance before multiplying it to the unit vector?**
  
  I'm thinking that this can be helpful to eliminate the network from needing too much training data by restricting the output ranges the network could give. But I also think that it could make the learning slower when the ReLU unit dies (value=0) so the gradient doesn't flow properly somehow.
  
  **Tags:** neural-networks, deep-learning, tensorflow, keras
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-28. Score: 1 | Accepted answer
  
  Applying a ReLU activation function to a non-negative number is the same as applying the identity activation function.
  
  What you may wish to do is apply $\ln(|\vec{x}|)$, $\arctan(x[0], x[1])$, ..., $\arctan(x[0], x[N-1])$, where $N$ is the number of dimensions to $\vec{x}$, and pass this directly into a parameter matrix multiplication (which involves deliberately using an identity activation in the input layer in some frameworks). This typically improves speed, accuracy, and reliability of convergence when the magnitudes exhibit a distribution close to an exponential of Gaussian and the directional components are near Gaussian when expressed in radians.
  
  
  ---
  
  