Answers.Body: <p>The brains of mammals do not use an activation function.  Only machine learning designs based on the perceptron multiply the vector of outputs from a prior layer by a parameter matrix and pass the result statelessly into a mathematical function.</p>
  
  <p>Although the spike aggregation behavior has been partly modeled, and in far more detail than the 1952 Hodgkin and Huxley model, all the models require statefulness to functionally approximate biological neurons.  RNNs and their derivatives are an attempt to correct that shortcoming in the perceptron design.</p>
  
  <p>In addition to that distinction, although the signal strength summing into activation functions are parametrized, traditional ANNs, CNNs, and RNNs, are statically connected, something Intel claims they will correct with the Nirvana architecture in 2019 (which places into silicon that which we would call layer set up in Python or Java now.</p>
  
  <p>There are at least three important biological neuron features that make the activation mechanism more than a function of a scalar input producing a scalar output, which renders questionable any algebraic comparison.</p>
  
  <ul>
  <li>State held as neuroplastic (changing) connectivity, and this is not just how many neurons in a layer but also the direction of signal propagation in three dimensions and the topology of the network, which is organized, but chaotically so</li>
  <li>The state held within the cytoplasm and its organelles, which is only partly understood as of 2018</li>
  <li>The fact that there is a temporal alignment factor, that pulses through a biological circuit may arrive via synapses in such a way that they aggregate but the peaks of the pulses are not coincident in time, so the activation probability is not as high as if they were temporally aligned.</li>
  </ul>
  
  <p>The decision about what activation function to use has largely been based on the analysis of convergence on a theoretical level combined with testing permutations to see which ones show the most desirable combinations of speed, accuracy, and reliability in ctheonvergence.  By reliability is meant that convergence on the global optimum (not some local minimum of the error function) is reached at all for the majority of input cases.</p>
  
  <p>This bifurcated research between the forks of practical machine learning and biological simulations and modeling.  The two branches may rejoin at some point with the emergence of spiking - Accuracy
   - Reliability (completes) networks.  The machine learning branch may borrow inspiration from the biological, such as the case of visual and auditory pathways in brains.</p>
  
  <p>They have parallels and relationships that may be exploited to aid in progress along both forks, but gaining knowledge by comparing the shapes of activation functions is confounded by the above three differences, especially the temporal alignment factor and the entire timing of brain circuits which cannot be modeled using iterations.  The brain is a true parallel computing architecture, not reliant on loops or even time sharing in the CPU and data buses.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T00:50:50.507Z
Answers.LastActivityDate: 2018-08-13T00:50:50.507Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>The answer is <strong>We do not know</strong>. Odds are, we will not know for quite a while. The reason for this is we cannot understand the "code" of the human brain, nor can we simply feed it values and get results. This limits us to measuring currents of the input and output on test subjects, and we have had few such test subjects that are <em>human</em>. Thus, we know almost nothing about the human brain, including the activation function.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-18T11:58:55.093Z
Answers.LastActivityDate: 2018-04-18T11:58:55.093Z
Answers.UserDisplayName: FreezePhoenix
Answers.Body: <p>The human brain does have an activation function which is sigmoid like when considering the membrane potential at the Axon Initial Segment (AIS) as input and the rate of action potential as the output.</p>
  <p>There is a threshold membrane potential under witch no action potential is generated, and there is a maximum output rate the neuron may produce due to the hyper polarisation after the action potential spike.</p>
  <p>The biological neuron adjust the slope of the sigmoid and its mean (or the threshold if you like). The adjustment of the mean (and threshold) is quite fast as the AIS has a particular structure allowing to move the voltage sensitive sodium pores with an intracellular muscle (actin) and skeleton (microtubules).</p>
  <p><img src="https://www.jneurosci.org/content/jneuro/38/9/2135/F3.large.jpg" alt="this image of the AIS" />.</p>
  <p>From &quot;The Axon Initial Segment: An Updated Viewpoint&quot; Christophe Leterrier,
  Journal of Neuroscience 28 February 2018, 38 (9) 2135-2145; <a href="https://doi.org/10.1523/JNEUROSCI.1922-17.2018" rel="nofollow noreferrer">https://doi.org/10.1523/JNEUROSCI.1922-17.2018</a></p>
  <p>The sigmoid change is normalizing the output. When the membrane potential is artificially strongly stimulated, the average output rate is reduced, and when the stimulation is reduced, the average output rate is increased.</p>
  <p>This normalization makes sense with recursive neural network. The system must be maintained in a stable regime.</p>
  <p>Note that the sigmoid is not symmetric and that the biological signal processing is stochastic. The action potential spike is a dirac which thus doesn't carry any information in itself. It is the relative time of spikes that carries the information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-02T08:52:51.240Z
Answers.LastActivityDate: 2024-07-02T09:18:42.010Z
Answers.LastEditDate: 2024-07-02T09:18:42.010Z
Answers.UserDisplayName: chmike
Answers.Body: <p>The basic neuron firing is similar to a spiking function. However, there is both an immediate (enough positives charges leads to overcoming activation threshold for the axon hillock, leading to more jumping in), and feedback effect. not every neuron “fires” the same way, as transcriptors can produce more channels to increase influx of charge and strengthen an activation. Axon terminal also have vesicles with neurotransmitters. Only a few hundred neurotransmitters exist, ex dopamine, similar to an alphabet which can be used to create different words or effects in different areas of the brain.</p>
  <p>So while the basic concept can be analogous to spiking / Hebbian activation, in reality there is modulation of states which make it more complex.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-04T15:31:07.740Z
Answers.LastActivityDate: 2024-07-04T15:31:07.740Z
Answers.UserDisplayName: lnx
Answers.Body: <p>My interpretation of the question was 'what activation function in an artificial neural network (ANN) is closest to that found in the brain?'</p>
  
  <p>Whilst I agree with the selected answer above, that a single neuron outputs a dirac, if you think of a neuron in an ANN as modelling the output firing rate, rather than the current output, then I believe ReLU might be closest?</p>
  
  <p><a href="http://jackterwilliger.com/biological-neural-networks-part-i-spiking-neurons/" rel="nofollow noreferrer">http://jackterwilliger.com/biological-neural-networks-part-i-spiking-neurons/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-27T16:58:59.240Z
Answers.LastActivityDate: 2019-06-27T16:58:59.240Z
Answers.UserDisplayName: tea_pea
Answers.Body: <p>The thing you were reading about is known as the <a href="https://en.wikipedia.org/wiki/Action_potential" rel="noreferrer">action potential</a>. It is a mechanism that governs how information flows within a neuron.</p>
  
  <p>It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards the resting potential if it is left alone, but incoming electric activations from dendrites can shift its electric potential. </p>
  
  <p>If the neuron reaches a certain threshold in electric potential (the activation potential), the entire neuron and its connecting axons goes through a chain reaction of ionic exchange inside/outside the cell that results in a "wave of propagation" through the axon.</p>
  
  <p>TL;DR: Once a neuron reaches a certain activation potential, it electrically discharges. But if the electric potential of the neuron doesn't reach that value then the neuron does not activate.</p>
  
  <blockquote>
    <p>Does the human brain use a specific activation function? </p>
  </blockquote>
  
  <p>IIRC neurons in different parts of the brain behave a bit differently, and the way this question is phrased sounds as if you are asking if there is a specific implementation of neuronal activation (as opposed to us modelling it). </p>
  
  <p>But in general behave relatively similar to each other (Neurons communicate with each other via neurochemicals, information propagates inside a neuron via a mechanism known as the action potential...) But the details and the differences they cause could be significant.</p>
  
  <p>There are various <a href="https://en.wikipedia.org/wiki/Biological_neuron_model" rel="noreferrer">biological neuron models</a>, but the <a href="http://neuronaldynamics.epfl.ch/online/Ch2.S2.html" rel="noreferrer">Hodgkin-Huxley Model</a> is the most notable. </p>
  
  <p>Also note that a general description of neurons don't give you a general description of neuronal dynamics a la cognition (understanding a tree doesn't give you complete understanding of a forest)</p>
  
  <p>But, the method of which information propagates inside a neuron is in general quite well understood as sodium / potassium ionic exchange.</p>
  
  <blockquote>
    <p>It (activation potential) sounds a lot like ReLU...</p>
  </blockquote>
  
  <p>It's only like ReLU in the sense that they require a threshold before anything happens. But ReLU can have variable output while neurons are all-or-nothing. </p>
  
  <p>Also ReLU (and other activation functions in general) are differentiable with respect to input space. This is very important for backprop.</p>
  
  <p>This is a ReLU function, with the X-axis being input value and Y-axis being output value.
  <a href="https://i.sstatic.net/vD7Ut.png" rel="noreferrer"><img src="https://i.sstatic.net/vD7Ut.png" alt="enter image description here"></a></p>
  
  <p>And this is the action potential with the X-axis being time, and Y being output value.<a href="https://i.sstatic.net/EsnF7.png" rel="noreferrer"><img src="https://i.sstatic.net/EsnF7.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-20T06:55:24.657Z
Answers.Text: It's a step function if the x-axis is the input, but a Delta Dirac if the x-axis is time, you mean?
Answers.UserDisplayName: mlman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-04T21:18:25.897Z
Answers.Text: You are right but the point i'm trying to get at (about it being continuous and moving back to resting potential), ill upload a picture to make myself more clear
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T11:34:18.397Z
Answers.Text: Seems more like a sigmoid that a step function
Answers.UserDisplayName: Zwiebak
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-18T14:21:39.670Z
Answers.LastActivityDate: 2018-05-04T22:09:14.397Z
Answers.LastEditDate: 2018-05-04T22:09:14.397Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Body: <p>Does the human brain use a specific activation function?</p>
  <p>I've tried doing some research, and as it's a threshold for whether the signal is sent through a neuron or not, it sounds a lot like ReLU. However, I can't find a single article confirming this. Or is it more like a step function (it sends 1 if it's above the threshold, instead of the input value)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-03T10:22:21.813Z
Comments.Text: It is implicit in this question that an artificial neuron in a neural net is a model or equivalent to a natural neuron, but this assumption is false.
Comments.UserDisplayName: pasaba por aqui
Tags: <activation-functions><neuroscience><brain>
Title: What activation function does the human brain use?
UserDisplayName: mlman
ThreadText: # What activation function does the human brain use?
  
  (Asked by: username_0 on 2018-04-18)
  
  Does the human brain use a specific activation function?
  
  I've tried doing some research, and as it's a threshold for whether the signal is sent through a neuron or not, it sounds a lot like ReLU. However, I can't find a single article confirming this. Or is it more like a step function (it sends 1 if it's above the threshold, instead of the input value)?
  
  **Tags:** activation-functions, neuroscience, brain
  
  > **Comments:**
  > **username_1**: It is implicit in this question that an artificial neuron in a neural net is a model or equivalent to a natural neuron, but this assumption is false.  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_7 on 2018-04-18. Score: 19 | Accepted answer
  
  The thing you were reading about is known as the [action potential](https://en.wikipedia.org/wiki/Action_potential). It is a mechanism that governs how information flows within a neuron.
  
  It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards the resting potential if it is left alone, but incoming electric activations from dendrites can shift its electric potential.
  
  If the neuron reaches a certain threshold in electric potential (the activation potential), the entire neuron and its connecting axons goes through a chain reaction of ionic exchange inside/outside the cell that results in a "wave of propagation" through the axon.
  
  TL;DR: Once a neuron reaches a certain activation potential, it electrically discharges. But if the electric potential of the neuron doesn't reach that value then the neuron does not activate.
  
  > Does the human brain use a specific activation function?
  
  IIRC neurons in different parts of the brain behave a bit differently, and the way this question is phrased sounds as if you are asking if there is a specific implementation of neuronal activation (as opposed to us modelling it).
  
  But in general behave relatively similar to each other (Neurons communicate with each other via neurochemicals, information propagates inside a neuron via a mechanism known as the action potential...) But the details and the differences they cause could be significant.
  
  There are various [biological neuron models](https://en.wikipedia.org/wiki/Biological_neuron_model), but the [Hodgkin-Huxley Model](http://neuronaldynamics.epfl.ch/online/Ch2.S2.html) is the most notable.
  
  Also note that a general description of neurons don't give you a general description of neuronal dynamics a la cognition (understanding a tree doesn't give you complete understanding of a forest)
  
  But, the method of which information propagates inside a neuron is in general quite well understood as sodium / potassium ionic exchange.
  
  > It (activation potential) sounds a lot like ReLU...
  
  It's only like ReLU in the sense that they require a threshold before anything happens. But ReLU can have variable output while neurons are all-or-nothing.
  
  Also ReLU (and other activation functions in general) are differentiable with respect to input space. This is very important for backprop.
  
  This is a ReLU function, with the X-axis being input value and Y-axis being output value.
  [![enter image description here](https://i.sstatic.net/vD7Ut.png)](https://i.sstatic.net/vD7Ut.png)
  
  And this is the action potential with the X-axis being time, and Y being output value.[![enter image description here](https://i.sstatic.net/EsnF7.png)](https://i.sstatic.net/EsnF7.png)
  
  > **Comments:**
  > **username_0**: It's a step function if the x-axis is the input, but a Delta Dirac if the x-axis is time, you mean?  
  > **username_7**: You are right but the point i'm trying to get at (about it being continuous and moving back to resting potential), ill upload a picture to make myself more clear  
  > **username_8**: Seems more like a sigmoid that a step function  
  
  ---
  
  ### Answer by username_2 on 2018-08-13. Score: 6
  
  The brains of mammals do not use an activation function. Only machine learning designs based on the perceptron multiply the vector of outputs from a prior layer by a parameter matrix and pass the result statelessly into a mathematical function.
  
  Although the spike aggregation behavior has been partly modeled, and in far more detail than the 1952 Hodgkin and Huxley model, all the models require statefulness to functionally approximate biological neurons. RNNs and their derivatives are an attempt to correct that shortcoming in the perceptron design.
  
  In addition to that distinction, although the signal strength summing into activation functions are parametrized, traditional ANNs, CNNs, and RNNs, are statically connected, something Intel claims they will correct with the Nirvana architecture in 2019 (which places into silicon that which we would call layer set up in Python or Java now.
  
  There are at least three important biological neuron features that make the activation mechanism more than a function of a scalar input producing a scalar output, which renders questionable any algebraic comparison.
  
  * State held as neuroplastic (changing) connectivity, and this is not just how many neurons in a layer but also the direction of signal propagation in three dimensions and the topology of the network, which is organized, but chaotically so
  * The state held within the cytoplasm and its organelles, which is only partly understood as of 2018
  * The fact that there is a temporal alignment factor, that pulses through a biological circuit may arrive via synapses in such a way that they aggregate but the peaks of the pulses are not coincident in time, so the activation probability is not as high as if they were temporally aligned.
  
  The decision about what activation function to use has largely been based on the analysis of convergence on a theoretical level combined with testing permutations to see which ones show the most desirable combinations of speed, accuracy, and reliability in ctheonvergence. By reliability is meant that convergence on the global optimum (not some local minimum of the error function) is reached at all for the majority of input cases.
  
  This bifurcated research between the forks of practical machine learning and biological simulations and modeling. The two branches may rejoin at some point with the emergence of spiking - Accuracy
  - Reliability (completes) networks. The machine learning branch may borrow inspiration from the biological, such as the case of visual and auditory pathways in brains.
  
  They have parallels and relationships that may be exploited to aid in progress along both forks, but gaining knowledge by comparing the shapes of activation functions is confounded by the above three differences, especially the temporal alignment factor and the entire timing of brain circuits which cannot be modeled using iterations. The brain is a true parallel computing architecture, not reliant on loops or even time sharing in the CPU and data buses.
  
  
  ---
  
  ### Answer by username_3 on 2018-04-18. Score: 1
  
  The answer is **We do not know**. Odds are, we will not know for quite a while. The reason for this is we cannot understand the "code" of the human brain, nor can we simply feed it values and get results. This limits us to measuring currents of the input and output on test subjects, and we have had few such test subjects that are *human*. Thus, we know almost nothing about the human brain, including the activation function.
  
  
  ---
  
  ### Answer by username_4 on 2024-07-02. Score: 1
  
  The human brain does have an activation function which is sigmoid like when considering the membrane potential at the Axon Initial Segment (AIS) as input and the rate of action potential as the output.
  
  There is a threshold membrane potential under witch no action potential is generated, and there is a maximum output rate the neuron may produce due to the hyper polarisation after the action potential spike.
  
  The biological neuron adjust the slope of the sigmoid and its mean (or the threshold if you like). The adjustment of the mean (and threshold) is quite fast as the AIS has a particular structure allowing to move the voltage sensitive sodium pores with an intracellular muscle (actin) and skeleton (microtubules).
  
  ![this image of the AIS](https://www.jneurosci.org/content/jneuro/38/9/2135/F3.large.jpg).
  
  From "The Axon Initial Segment: An Updated Viewpoint" Christophe Leterrier,
  Journal of Neuroscience 28 February 2018, 38 (9) 2135-2145; <https://doi.org/10.1523/JNEUROSCI.1922-17.2018>
  
  The sigmoid change is normalizing the output. When the membrane potential is artificially strongly stimulated, the average output rate is reduced, and when the stimulation is reduced, the average output rate is increased.
  
  This normalization makes sense with recursive neural network. The system must be maintained in a stable regime.
  
  Note that the sigmoid is not symmetric and that the biological signal processing is stochastic. The action potential spike is a dirac which thus doesn't carry any information in itself. It is the relative time of spikes that carries the information.
  
  
  ---
  
  ### Answer by username_5 on 2024-07-04. Score: 0
  
  The basic neuron firing is similar to a spiking function. However, there is both an immediate (enough positives charges leads to overcoming activation threshold for the axon hillock, leading to more jumping in), and feedback effect. not every neuron “fires” the same way, as transcriptors can produce more channels to increase influx of charge and strengthen an activation. Axon terminal also have vesicles with neurotransmitters. Only a few hundred neurotransmitters exist, ex dopamine, similar to an alphabet which can be used to create different words or effects in different areas of the brain.
  
  So while the basic concept can be analogous to spiking / Hebbian activation, in reality there is modulation of states which make it more complex.
  
  
  ---
  
  ### Answer by username_6 on 2019-06-27. Score: 0
  
  My interpretation of the question was 'what activation function in an artificial neural network (ANN) is closest to that found in the brain?'
  
  Whilst I agree with the selected answer above, that a single neuron outputs a dirac, if you think of a neuron in an ANN as modelling the output firing rate, rather than the current output, then I believe ReLU might be closest?
  
  <http://jackterwilliger.com/biological-neural-networks-part-i-spiking-neurons/>
  
  
  ---
  
  