Answers.Body: <p>From what I've read on deep nets and neuroscience, your interpretation is backwards.  Most 'neurons' in deep nets use RELU activations, which is strictly excitatory, while mammal brains are composed of both excitatory and inhibitory synapses/neurons.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Excitatory_postsynaptic_potential" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Excitatory_postsynaptic_potential</a></p>
  
  <p><a href="https://en.wikipedia.org/wiki/Inhibitory_postsynaptic_potential" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Inhibitory_postsynaptic_potential</a></p>
  
  <p>I think the reason for strictly excitatory activations like RELU in deep nets is that they provide a continuous gradient for backpropagation during learning.  Inhibitory activations make the gradient discontinuous, greatly complicating the propagation of feedback/error.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T10:48:13.117Z
Answers.Text: But sigmoids have negativ output as well, which acts inhibitory. For some problems you need activation functions that can be negativ. I think ReLU are widely used in vision because vision works largely by combining features, not so much by determining features that definitely shouldn't be there for certain classes.
Answers.UserDisplayName: BlindKungFuMaster
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-18T22:16:12.313Z
Answers.LastActivityDate: 2017-05-18T22:16:12.313Z
Answers.UserDisplayName: Randy
Answers.Body: <p>I think you are confused. The reason why some neural neural networks have neurons with an output in range <code>(-1, 1)</code> all depends on the activation function used. Some networks even have neurons with an output range of <code>(-Infinity, +Infinity)</code> ( aka the identity function ).</p>
  
  <p>I advise you to take a look at this list: <a href="https://en.wikipedia.org/wiki/Activation_function" rel="nofollow noreferrer">activation functions w/ ranges</a>. </p>
  
  <p>Nowadays a lot of neural networks have a <em>mixture</em> of different activation functions. E.g. combining <code>ReLU</code> with <code>Sigmoid</code> or <code>TanH</code>. </p>
  
  <p><a href="https://github.com/wagenaartje/neataptic" rel="nofollow noreferrer">This neural network library</a> even shows that for most simple problems, neural networks prefer a variety of activation functions. </p>
  
  <p>The red lines just mean that that connection has a low value in comparison with other connections.</p>
  
  <p><em>There is no rule of thumb for activation functions, some problems benefit from a certain function, some don't.</em></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-17T17:32:04.037Z
Answers.LastActivityDate: 2017-05-17T17:32:04.037Z
Answers.UserDisplayName: Thomas Wagenaar
Body: <p>Something I like about neural network AI is that we already have a blueprint for it - we know in great detail how different types of neurons work, and we don't have to invent the AI, we can just replicate what we already know works (neural networks) in a simplified way and train it. </p>
  
  <p>So what's confusing me right now is why many popular neural models I've seen when studying neural networks include both inhibitory and stimulating connections. In real neural networks, from my understanding, there is no negative signal being transferred, rather the signals sent between neurons are comparable to values between 0.1 and 1. There's no mechanic for an inverse (inhibiting) signal to be sent. </p>
  
  <p>For example, in this network (seen overlaying a snake being simulated), the red lines represent inhibiting connections, and the blue lines represent stimulating neurons:</p>
  
  <p><a href="https://i.sstatic.net/nLT4I.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nLT4I.png" alt="enter image description here"></a></p>
  
  <p>Is this just an inconsequential detail of neural network design, where there's really no significant difference between a range of 0 to 1 and a range of -1 to 1? Or is there a reason that connections in our simulated neural networks benefit from being able to a express a range from -1 to 1 rather than just 0 to 1?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-18T06:41:37.427Z
Comments.Text: If by real life neural networks, you mean the squishy kind in somebodies skull, those do have inhibitory connection. If not, what is the difference between "real life neural networks" and "virtual neural networks"?
Comments.UserDisplayName: BlindKungFuMaster
Tags: <neural-networks>
Title: Why are inhibitory connections often used in virtual neural networks when they don't seem to exist in real life neural networks?
UserDisplayName: J.Todd
ThreadText: # Why are inhibitory connections often used in virtual neural networks when they don't seem to exist in real life neural networks?
  
  (Asked by: username_0 on 2017-05-17)
  
  Something I like about neural network AI is that we already have a blueprint for it - we know in great detail how different types of neurons work, and we don't have to invent the AI, we can just replicate what we already know works (neural networks) in a simplified way and train it.
  
  So what's confusing me right now is why many popular neural models I've seen when studying neural networks include both inhibitory and stimulating connections. In real neural networks, from my understanding, there is no negative signal being transferred, rather the signals sent between neurons are comparable to values between 0.1 and 1. There's no mechanic for an inverse (inhibiting) signal to be sent.
  
  For example, in this network (seen overlaying a snake being simulated), the red lines represent inhibiting connections, and the blue lines represent stimulating neurons:
  
  [![enter image description here](https://i.sstatic.net/nLT4I.png)](https://i.sstatic.net/nLT4I.png)
  
  Is this just an inconsequential detail of neural network design, where there's really no significant difference between a range of 0 to 1 and a range of -1 to 1? Or is there a reason that connections in our simulated neural networks benefit from being able to a express a range from -1 to 1 rather than just 0 to 1?
  
  **Tags:** neural-networks
  
  > **Comments:**
  > **username_1**: If by real life neural networks, you mean the squishy kind in somebodies skull, those do have inhibitory connection. If not, what is the difference between "real life neural networks" and "virtual neural networks"?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-05-17. Score: 2 | Accepted answer
  
  I think you are confused. The reason why some neural neural networks have neurons with an output in range `(-1, 1)` all depends on the activation function used. Some networks even have neurons with an output range of `(-Infinity, +Infinity)` ( aka the identity function ).
  
  I advise you to take a look at this list: [activation functions w/ ranges](https://en.wikipedia.org/wiki/Activation_function).
  
  Nowadays a lot of neural networks have a *mixture* of different activation functions. E.g. combining `ReLU` with `Sigmoid` or `TanH`.
  
  [This neural network library](https://github.com/wagenaartje/neataptic) even shows that for most simple problems, neural networks prefer a variety of activation functions.
  
  The red lines just mean that that connection has a low value in comparison with other connections.
  
  *There is no rule of thumb for activation functions, some problems benefit from a certain function, some don't.*
  
  
  ---
  
  ### Answer by username_2 on 2017-05-18. Score: 2
  
  From what I've read on deep nets and neuroscience, your interpretation is backwards. Most 'neurons' in deep nets use RELU activations, which is strictly excitatory, while mammal brains are composed of both excitatory and inhibitory synapses/neurons.
  
  <https://en.wikipedia.org/wiki/Excitatory_postsynaptic_potential>
  
  <https://en.wikipedia.org/wiki/Inhibitory_postsynaptic_potential>
  
  I think the reason for strictly excitatory activations like RELU in deep nets is that they provide a continuous gradient for backpropagation during learning. Inhibitory activations make the gradient discontinuous, greatly complicating the propagation of feedback/error.
  
  > **Comments:**
  > **username_1**: But sigmoids have negativ output as well, which acts inhibitory. For some problems you need activation functions that can be negativ. I think ReLU are widely used in vision because vision works largely by combining features, not so much by determining features that definitely shouldn't be there for certain classes.  
  
  ---
  
  