Answers.Body: <p>In the case of artificial neural networks, your question can be (partially) answered by looking at the definition of the operation that an artificial neuron performs. An artificial neuron is usually defined as a linear combination of its inputs, followed by the application of a non-linear activation function (e.g. the <a href="http://mathworld.wolfram.com/HyperbolicTangent.html" rel="nofollow noreferrer">hyperbolic tangent</a> or <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a>). More formally, a neuron <span class="math-container">$i$</span> in layer <span class="math-container">$l$</span> performs the following operation</p>
  
  <p><span class="math-container">\begin{align}
  o_i^l = \sigma \left(\sum_{j=1}^N w_j o_j^{l-1} \right) \tag{1}\label{1},
  \end{align}</span></p>
  
  <p>where <span class="math-container">$o_j^{l-1}$</span> is the output from neuron <span class="math-container">$j$</span> in layer <span class="math-container">$l-1$</span> (the previous layer), <span class="math-container">$w_j$</span> the corresponding weight, <span class="math-container">$\sigma$</span> an activation function and <span class="math-container">$N$</span> the number of neurons from layer <span class="math-container">$l-1$</span> connected to neuron <span class="math-container">$i$</span> in layer <span class="math-container">$l$</span>.</p>
  
  <p>Let's assume that <span class="math-container">$\sigma$</span> is the <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a>, which is defined as follows</p>
  
  <p><span class="math-container">$$
  \sigma(x)=\max(0, x)
  $$</span></p>
  
  <p>which means that all negative numbers become <span class="math-container">$0$</span> and all non-negative numbers become themselves.</p>
  
  <p>In equation \ref{1}, if <span class="math-container">$w_j$</span> and <span class="math-container">$o_j^{l-1}$</span> have the same sign, then the product <span class="math-container">$w_j o_j^{l-1}$</span> is non-negative (positive or zero), else it is negative (or zero). Therefore, the sign of the output of neuron <span class="math-container">$j$</span> in layer <span class="math-container">$l-1$</span> alone does not fully determine the effect on <span class="math-container">$o_i^l$</span>, but the sign of the <span class="math-container">$w_j$</span> is also required. </p>
  
  <p>Let's suppose that the product <span class="math-container">$w_j o_j^{l-1}$</span> is negative, then, of course, this will negatively contribute to the sum in equation \ref{1}. In any case, even if the sum <span class="math-container">$\sum_{j=1}^N w_j o_j^{l-1}$</span> is negative, if <span class="math-container">$\sigma$</span> is the ReLU, no matter the magnitude of the negative number, <span class="math-container">$o_i^l$</span> will always be zero. However, if the activation function is <a href="http://mathworld.wolfram.com/HyperbolicTangent.html" rel="nofollow noreferrer">hyperbolic tangent</a>, the magnitude of a negative <span class="math-container">$\sum_{j=1}^N w_j o_j^{l-1}$</span> affects the magnitude of <span class="math-container">$o_i^l$</span>. More precisely, the more negative the sum is, the closest <span class="math-container">$o_i^l$</span> is to <span class="math-container">$-1$</span>. </p>
  
  <p>To conclude, in general, the effect of the sign of an output of an artificial neuron on neighboring neurons depends on the activation function and the learned weights, which depend on the error the neural network is making (assuming the neural network is trained with gradient descent combined with back-propagation), which in turn depends on the training dataset, the loss function, the architecture of the neural network, etc.</p>
  
  <p>Biological neurons and synapses are more complex than artificial ones. Nevertheless, <a href="https://www.sciencedirect.com/topics/agricultural-and-biological-sciences/synapse" rel="nofollow noreferrer">biological synapses are usually classified as either excitatory or inhibitory</a>, so they can have an excitatory or inhibitory effect on connected neurons.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T22:41:16.637Z
Answers.Text: So what means when we receive a negative in next neuron? Could you give me an example
Answers.UserDisplayName: Mahdi Amrollahi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T22:43:05.950Z
Answers.Text: @MehdiAmrollahi I am exactly explaining this in my answer. Please, read it again until you understand it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-20T03:40:55.883Z
Answers.Text: Maybe I do not understand the question - isn't it the case the the artificial neurons in layer X are only affected by those in layer X-1 (providing inputs), not other neurons in layer X?
Answers.UserDisplayName: George White
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-20T16:06:40.287Z
Answers.Text: @GeorgeWhite I think this is a nice question you should ask on the main website. I'll answer it (if nobody does it).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T22:13:14.560Z
Answers.LastActivityDate: 2019-12-18T22:50:04.147Z
Answers.LastEditDate: 2019-12-18T22:50:04.147Z
Answers.UserDisplayName: nbro
Body: <p>Artificial neural networks are composed of multiple neurons that are connected to each other. When the output of an artificial neuron is zero, it does not have any effect on neighboring neurons. When the output is positive, the neuron has an effect on neighboring neurons. </p>
  
  <p>What does it mean when the output of a neuron is negative (which can e.g. occur when the activation function of a neuron is the hyperbolic tangent)? What effect would this output have on neighboring neurons? </p>
  
  <p>Do biological neural networks also have this property?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-18T17:57:34.367Z
Comments.Text: I mean both of them
Comments.UserDisplayName: Mahdi Amrollahi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-19T06:28:50.017Z
Comments.Text: I think you might be misinterpreting what a neural network is doing with its outputs. The most an output of a neuron can generally mean, is how sensitive it is to a certain feature of the input. If you were to look at a 100 layer neural network, and saw an individual neuron was negative, there is very little you can infer from this. It doesn't really *mean* anything tangible. If you were to research this particular frozen state of the network, you could eventually interpret exactly what the network is doing, but for all intents and purposes, a negative output from a neuron doesnt tell you much
Comments.UserDisplayName: Recessive
Tags: <neural-networks><artificial-neuron><biology>
Title: What effect does a negative output of a neuron have on neighbouring neurons?
UserDisplayName: Mahdi Amrollahi
ThreadText: # What effect does a negative output of a neuron have on neighbouring neurons?
  
  (Asked by: username_0 on 2019-12-18)
  
  Artificial neural networks are composed of multiple neurons that are connected to each other. When the output of an artificial neuron is zero, it does not have any effect on neighboring neurons. When the output is positive, the neuron has an effect on neighboring neurons.
  
  What does it mean when the output of a neuron is negative (which can e.g. occur when the activation function of a neuron is the hyperbolic tangent)? What effect would this output have on neighboring neurons?
  
  Do biological neural networks also have this property?
  
  **Tags:** neural-networks, artificial-neuron, biology
  > **Comments:**
  > **username_0**: I mean both of them  
  > **username_1**: I think you might be misinterpreting what a neural network is doing with its outputs. The most an output of a neuron can generally mean, is how sensitive it is to a certain feature of the input. If you were to look at a 100 layer neural network, and saw an individual neuron was negative, there is very little you can infer from this. It doesn't really *mean* anything tangible. If you were to research this particular frozen state of the network, you could eventually interpret exactly what the network is doing, but for all intents and purposes, a negative output from a neuron doesnt tell you much  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-12-18. Score: 3 | Accepted answer
  
  In the case of artificial neural networks, your question can be (partially) answered by looking at the definition of the operation that an artificial neuron performs. An artificial neuron is usually defined as a linear combination of its inputs, followed by the application of a non-linear activation function (e.g. the [hyperbolic tangent](http://mathworld.wolfram.com/HyperbolicTangent.html) or [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks))). More formally, a neuron $i$ in layer $l$ performs the following operation
  
  \begin{align}
  o\_i^l = \sigma \left(\sum\_{j=1}^N w\_j o\_j^{l-1} \right) \tag{1}\label{1},
  \end{align}
  
  where $o\_j^{l-1}$ is the output from neuron $j$ in layer $l-1$ (the previous layer), $w\_j$ the corresponding weight, $\sigma$ an activation function and $N$ the number of neurons from layer $l-1$ connected to neuron $i$ in layer $l$.
  
  Let's assume that $\sigma$ is the [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), which is defined as follows
  
  $$
  \sigma(x)=\max(0, x)
  $$
  
  which means that all negative numbers become $0$ and all non-negative numbers become themselves.
  
  In equation \ref{1}, if $w\_j$ and $o\_j^{l-1}$ have the same sign, then the product $w\_j o\_j^{l-1}$ is non-negative (positive or zero), else it is negative (or zero). Therefore, the sign of the output of neuron $j$ in layer $l-1$ alone does not fully determine the effect on $o\_i^l$, but the sign of the $w\_j$ is also required.
  
  Let's suppose that the product $w\_j o\_j^{l-1}$ is negative, then, of course, this will negatively contribute to the sum in equation \ref{1}. In any case, even if the sum $\sum\_{j=1}^N w\_j o\_j^{l-1}$ is negative, if $\sigma$ is the ReLU, no matter the magnitude of the negative number, $o\_i^l$ will always be zero. However, if the activation function is [hyperbolic tangent](http://mathworld.wolfram.com/HyperbolicTangent.html), the magnitude of a negative $\sum\_{j=1}^N w\_j o\_j^{l-1}$ affects the magnitude of $o\_i^l$. More precisely, the more negative the sum is, the closest $o\_i^l$ is to $-1$.
  
  To conclude, in general, the effect of the sign of an output of an artificial neuron on neighboring neurons depends on the activation function and the learned weights, which depend on the error the neural network is making (assuming the neural network is trained with gradient descent combined with back-propagation), which in turn depends on the training dataset, the loss function, the architecture of the neural network, etc.
  
  Biological neurons and synapses are more complex than artificial ones. Nevertheless, [biological synapses are usually classified as either excitatory or inhibitory](https://www.sciencedirect.com/topics/agricultural-and-biological-sciences/synapse), so they can have an excitatory or inhibitory effect on connected neurons.
  
  > **Comments:**
  > **username_0**: So what means when we receive a negative in next neuron? Could you give me an example  
  > **username_2**: @MehdiAmrollahi I am exactly explaining this in my answer. Please, read it again until you understand it.  
  > **username_3**: Maybe I do not understand the question - isn't it the case the the artificial neurons in layer X are only affected by those in layer X-1 (providing inputs), not other neurons in layer X?  
  > **username_2**: @GeorgeWhite I think this is a nice question you should ask on the main website. I'll answer it (if nobody does it).  
  
  ---
  
  