Answers.Body: <p>Why wouldn't they work?</p>
  <p>Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer">Sigmoid</a> function, then the output is squashed from <span class="math-container">$[0,1]$</span>. If the entire layer uses a <a href="https://en.wikipedia.org/wiki/Softmax_function" rel="nofollow noreferrer">SoftMax</a> function, then the output of all neurons is squashed from <span class="math-container">$[0,1]$</span> and their sum equals 1. In other others, they represent a set of probabilities, where you can then use <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">cross-entropy</a> to optimize their values (cross-entropy measures the difference between two probability distributions).</p>
  <p><a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a> and <a href="https://arxiv.org/abs/1511.07289" rel="nofollow noreferrer">ELU</a> are simply other types of functions, whose output is not limited to the range <span class="math-container">$[0, 1]$</span>. They are differentiable, like other activation functions, and so they can be used in any neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-24T22:58:34.253Z
Answers.Text: I think the OP was confused about the fact that the cross-entropy _may_ require a probability vector as input, so, by using activation functions that do not have values in the range $[0, 1]$, the output of the neural network may not be a probability vector. You partially address this by mentioning the softmax, but I think this answer could definitely be improved by explaining more in detail the cross-entropy, and it's relation to the softmax and other activation functions of the hidden neurons.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-09T20:47:17.330Z
Answers.LastActivityDate: 2021-01-24T22:53:22.933Z
Answers.LastEditDate: 2021-01-24T22:53:22.933Z
Answers.UserDisplayName: BlueMoon93
Answers.Body: <p>Christopher Olah's <a href="http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/" rel="nofollow noreferrer">blog post</a> describes it better that I ever could.
  Basically, most data we come across can't be separated with a single line, but with some kind of curve. Non-linearities allow us to distort the input space in ways that make the data linearly separable, making classification more accuarate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-24T22:43:59.730Z
Answers.Text: This does not really answer the question. This answers another question, which is "Why do we need non-linearities?". Please, read the **actual** question again.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-09T14:21:31.417Z
Answers.LastActivityDate: 2018-02-09T14:21:31.417Z
Answers.UserDisplayName: Daniel
Body: <p>Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?</p>
  <p>My understanding is that neurons can only produce values between 0 and 1, and that this assumption can be used in things like cross-entropy. Are my assumptions just completely wrong?</p>
  <p>Is there any reference that explains this?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-09T21:09:42.727Z
Comments.Text: [This discussion](https://stackoverflow.com/q/46659525/712995) might be helpful
Comments.UserDisplayName: Maxim
Tags: <neural-networks><reference-request><activation-functions><relu><cross-entropy>
Title: Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?
UserDisplayName: Emil Wormbs
ThreadText: # Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?
  
  (Asked by: username_0 on 2018-01-09)
  
  Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?
  
  My understanding is that neurons can only produce values between 0 and 1, and that this assumption can be used in things like cross-entropy. Are my assumptions just completely wrong?
  
  Is there any reference that explains this?
  
  **Tags:** neural-networks, reference-request, activation-functions, relu, cross-entropy
  
  > **Comments:**
  > **username_1**: [This discussion](https://stackoverflow.com/q/46659525/712995) might be helpful  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-01-09. Score: 2
  
  Why wouldn't they work?
  
  Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the [Sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) function, then the output is squashed from $[0,1]$. If the entire layer uses a [SoftMax](https://en.wikipedia.org/wiki/Softmax_function) function, then the output of all neurons is squashed from $[0,1]$ and their sum equals 1. In other others, they represent a set of probabilities, where you can then use [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) to optimize their values (cross-entropy measures the difference between two probability distributions).
  
  [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) and [ELU](https://arxiv.org/abs/1511.07289) are simply other types of functions, whose output is not limited to the range $[0, 1]$. They are differentiable, like other activation functions, and so they can be used in any neural network.
  
  > **Comments:**
  > **username_3**: I think the OP was confused about the fact that the cross-entropy _may_ require a probability vector as input, so, by using activation functions that do not have values in the range $[0, 1]$, the output of the neural network may not be a probability vector. You partially address this by mentioning the softmax, but I think this answer could definitely be improved by explaining more in detail the cross-entropy, and it's relation to the softmax and other activation functions of the hidden neurons.  
  
  ---
  
  ### Answer by username_4 on 2018-02-09. Score: 1
  
  Christopher Olah's [blog post](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/) describes it better that I ever could.
  Basically, most data we come across can't be separated with a single line, but with some kind of curve. Non-linearities allow us to distort the input space in ways that make the data linearly separable, making classification more accuarate.
  
  > **Comments:**
  > **username_3**: This does not really answer the question. This answers another question, which is "Why do we need non-linearities?". Please, read the **actual** question again.  
  
  ---
  
  