Answers.Body: <p>The activation function you choose depends on the application you are building/data that you have got to work with. It is hard to recommend one over the other, without taking this into account.</p>
  
  <p>Here is a short-summary of the advantages and disadvantages of some common activation functions:
  <a href="https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/" rel="nofollow noreferrer">https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/</a></p>
  
  <blockquote>
    <p>What does the author mean with ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.</p>
  </blockquote>
  
  <p>ReLU is good for inputs > 0, since ReLU = 0 if input &lt; 0(which would kill the neuron, if the gradient is = 0)</p>
  
  <p>To remedy this, you could look into using a Leaky-ReLU instead.
  (Which avoids killing the neuron by returning a non-zero value in the cases of input &lt;= 0)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T06:24:13.430Z
Answers.Text: This solved a big part of my question, but would but would there be a case that i specifically would use a linear activation instead of relu?
Answers.UserDisplayName: jennifer ruurs
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T13:38:33.413Z
Answers.Text: That depends again on your specific application.
  
  There are two major drawbacks of linear activation functions:
  
  1.You cant use back-propagation in training(since the derivative is a constant, it does not convey which weight influenced the input the most).
  
  2.Linear activation functions are only applicable to shallow networks, since the derivative of a linear function is a constant(multiple layers of linear functions is just another linear function).
  
  So, in the case that you have a shallow network - that does not rely on backpropagation - then you can use a linear activation function.
Answers.UserDisplayName: Krrrl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T13:39:33.520Z
Answers.Text: I think it is fair to say that you can think of the linear activation function as an artifact of the earlier stages of neural network development - when work was done on single/few perceptrons, rather than larger layered networks.
Answers.UserDisplayName: Krrrl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T22:17:20.057Z
Answers.LastActivityDate: 2019-10-21T22:17:20.057Z
Answers.UserDisplayName: Krrrl
Answers.Body: <p>Nothing is written on stone in here, but as a rule of thumb linear activation is not very common.
  A linear activation function in a hidden layer can collapse more neurons in more layers.
  Linear activation can be implemented in the last layer if a scale of the outputs is not used. (This is the most common use I have seen.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-21T18:08:09.753Z
Answers.LastActivityDate: 2020-09-25T02:45:28.143Z
Answers.LastEditDate: 2020-09-25T02:45:28.143Z
Answers.UserDisplayName: Joaquin Torrens
Body: <p>I have read this post:
  <a href="https://ai.stackexchange.com/q/7088/2444">How to choose an activation function?</a>.</p>
  
  <p>There is enough literature about activation functions, but when should I use a linear activation instead of ReLU? </p>
  
  <p>What does the author mean with <em>ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.</em>?</p>
  
  <p>Is there a more detail answer to this?</p>
  
Tags: <neural-networks><deep-learning><comparison><activation-functions>
Title: When should I use a linear activation instead of ReLU?
UserDisplayName: jennifer ruurs
ThreadText: # When should I use a linear activation instead of ReLU?
  
  (Asked by: username_0 on 2019-10-21)
  
  I have read this post:
  [How to choose an activation function?](https://ai.stackexchange.com/q/7088/2444).
  
  There is enough literature about activation functions, but when should I use a linear activation instead of ReLU?
  
  What does the author mean with *ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.*?
  
  Is there a more detail answer to this?
  
  **Tags:** neural-networks, deep-learning, comparison, activation-functions
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-10-21. Score: 2 | Accepted answer
  
  The activation function you choose depends on the application you are building/data that you have got to work with. It is hard to recommend one over the other, without taking this into account.
  
  Here is a short-summary of the advantages and disadvantages of some common activation functions:
  <https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/>
  
  > What does the author mean with ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.
  
  ReLU is good for inputs > 0, since ReLU = 0 if input < 0(which would kill the neuron, if the gradient is = 0)
  
  To remedy this, you could look into using a Leaky-ReLU instead.
  (Which avoids killing the neuron by returning a non-zero value in the cases of input <= 0)
  
  > **Comments:**
  > **username_0**: This solved a big part of my question, but would but would there be a case that i specifically would use a linear activation instead of relu?  
  > **username_1**: That depends again on your specific application.  There are two major drawbacks of linear activation functions:  1.You cant use back-propagation in training(since the derivative is a constant, it does not convey which weight influenced the input the most).  2.Linear activation functions are only applicable to shallow networks, since the derivative of a linear function is a constant(multiple layers of linear functions is just another linear function).  So, in the case that you have a shallow network - that does not rely on backpropagation - then you can use a linear activation function.  
  > **username_1**: I think it is fair to say that you can think of the linear activation function as an artifact of the earlier stages of neural network development - when work was done on single/few perceptrons, rather than larger layered networks.  
  
  ---
  
  ### Answer by username_2 on 2020-09-21. Score: 1
  
  Nothing is written on stone in here, but as a rule of thumb linear activation is not very common.
  A linear activation function in a hidden layer can collapse more neurons in more layers.
  Linear activation can be implemented in the last layer if a scale of the outputs is not used. (This is the most common use I have seen.)
  
  
  ---
  
  