Answers.Body: <p>I am not an DL expert but these are my short thoughts on it:</p>
  
  <p>I think this is because it is believed (from an information theoretic point of view) to be the good way to avoid that the network falls into some wired state from beginning on. Remember: DNNs are nonlinear approximators for continuous functions. So they have some storage capacity to learn an amount of n function to map from input to output. When you look on topic like data leakage you will see that NNs quickly try to cheat you if they can :D. The optimization applied during training will heavily be affected by the init state. So starting with an random initialization at least avoids that your neurons do all the same at the beginning etc. </p>
  
  <p><strong>Biological reasoning:</strong> 
  From the viewpoint of a neurobiologist I can recommend you to read <a href="https://en.m.wikipedia.org/wiki/Hebbian_theory" rel="nofollow noreferrer">Hebbian rule</a> and how neural systems work (eg. <a href="https://www.quora.com/How-do-axons-branching-out-from-neurons-find-their-targets" rel="nofollow noreferrer">google how neurons find targets</a>) in general and then to compare it to what is known about how dendrite cells in the cerebrum develop their interconnections in the first 3 years after birth. In summary there are behavioral patterns in nature which could look similar, inspiring and even reasonable. But, I would say the reason why this random init. is recommend is backed by mathematical and information theoretical assumptions rather then pure biological arguments.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-09T14:02:40.890Z
Answers.LastActivityDate: 2019-01-10T08:00:31.110Z
Answers.LastEditDate: 2019-01-10T08:00:31.110Z
Answers.UserDisplayName: PlagTag
Answers.Body: <p><strong>In short</strong></p>
  
  <p>I mentioned <a href="https://psychology.stackexchange.com/a/10886/4055">in another post</a>, how the Artificial Neural Network (ANN) weights are a relatively crude abstraction of connections between neurons in the brain. Similarly, <strong>the random weight initialization step in ANNs is a simple procedure that abstracts the complexity of <a href="https://en.wikipedia.org/wiki/Development_of_the_nervous_system" rel="nofollow noreferrer">central nervous system development</a> and <a href="https://en.wikipedia.org/wiki/Synaptogenesis#Central_nervous_system_synapse_formation" rel="nofollow noreferrer">synaptogenesis</a>.</strong></p>
  
  <p><strong>A bit more detail</strong> (with the most relevant parts <em>italicized</em> below)</p>
  
  <p>The neocortex (one of its columns, more specifically) is a region of the brain that somewhat resembles an ANN. It has a laminar structure with layers that receive and send axons from other brain regions. Those layers can be viewed as "input" and "output" layers of an ANN (axons "send" signals, dendrites "receive"). Other layers are intermediate-processing layers and can be viewed as the ANN "hidden" layers.</p>
  
  <p>When building an ANN, the programmer can set the number of layers and the number of units in each layer. In the neocortex, the number of layers and layer cell counts are determined mostly by genes (however, see: <a href="https://en.wikipedia.org/wiki/Neuroplasticity#Human_echolocation" rel="nofollow noreferrer">Human echolocation</a> for an example of post-birth brain plasticity). Chemical cues guide the positions of the cell bodies and create the laminar structure. They also seem to guide long term axonal connections between distant brain regions. The cells then sprout dendrites in certain characteristic "tree-like" patterns (see: <a href="http://www.neuromorpho.org/index.jsp" rel="nofollow noreferrer">NeuroMorpho.org</a> for examples). <em>The dendrites will then form synapses with axons or other cell bodies they encounter along the way, generally based on the encountered cell type.</em> </p>
  
  <p>This last phase is probably the most analogous to the idea of random weight initialization in ANNs. Based on where the cell is positioned and its type, <em>the encountered other neurons will be somewhat random and so will the connections to them</em>. These connections are probably not going to be very strong initially but will have room to get stronger during learning (probably analogous to initial random weights between 0 and ~0.1, with 1 being the strongest possible connection). Furthermore, most cells are either inhibitory or excitatory (analogous to negative and positive weights). </p>
  
  <p>Keep in mind this randomization process has a heavy spatial component in real brains. The neurons are small and so they will make these connections to nearby neurons that are 10-200 microns away. The long-distance connections between brain regions are mostly "programmed-in" via genes. In most ANNs, there is generally no distance-based aspect to the initialization of connection weights (although convolutional ANNs implicitly perform something like distance-based wiring by using the sliding window).</p>
  
  <p>There is also the <a href="https://en.wikipedia.org/wiki/Synaptic_pruning" rel="nofollow noreferrer">synaptic pruning</a> phenomenon, which might be analogous to creating many low weight connections in an ANN initially (birth), training it for some number of epochs (adolescence), and then removing most low-weight connections (consolidation in adulthood).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T20:06:27.133Z
Answers.Text: Do you by chance also have some hints on https://ai.stackexchange.com/questions/10416/is-the-thatcher-face-illusion-only-limited-to-face-recognition I haven't received any answer yet.
Answers.UserDisplayName: Leo Gallucci
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T04:04:10.700Z
Answers.LastActivityDate: 2019-01-10T04:04:10.700Z
Answers.UserDisplayName: Justas
Body: <p>When trying to map artificial neuronal models to biological facts it was not possible to find an answer regarding the <strong>biological justification</strong> of randomly initializing the weights.</p>
  
  <p>Perhaps this is not yet known from our current understanding of biological neurons?</p>
  
Tags: <machine-learning><training><artificial-neuron><neurons><biology>
Title: How do biological neurons weights get initialized?
UserDisplayName: Leo Gallucci
ThreadText: # How do biological neurons weights get initialized?
  
  (Asked by: username_0 on 2019-01-09)
  
  When trying to map artificial neuronal models to biological facts it was not possible to find an answer regarding the **biological justification** of randomly initializing the weights.
  
  Perhaps this is not yet known from our current understanding of biological neurons?
  
  **Tags:** machine-learning, training, artificial-neuron, neurons, biology
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-01-10. Score: 4 | Accepted answer
  
  **In short**
  
  I mentioned [in another post](https://psychology.stackexchange.com/a/10886/4055), how the Artificial Neural Network (ANN) weights are a relatively crude abstraction of connections between neurons in the brain. Similarly, **the random weight initialization step in ANNs is a simple procedure that abstracts the complexity of [central nervous system development](https://en.wikipedia.org/wiki/Development_of_the_nervous_system) and [synaptogenesis](https://en.wikipedia.org/wiki/Synaptogenesis#Central_nervous_system_synapse_formation).**
  
  **A bit more detail** (with the most relevant parts *italicized* below)
  
  The neocortex (one of its columns, more specifically) is a region of the brain that somewhat resembles an ANN. It has a laminar structure with layers that receive and send axons from other brain regions. Those layers can be viewed as "input" and "output" layers of an ANN (axons "send" signals, dendrites "receive"). Other layers are intermediate-processing layers and can be viewed as the ANN "hidden" layers.
  
  When building an ANN, the programmer can set the number of layers and the number of units in each layer. In the neocortex, the number of layers and layer cell counts are determined mostly by genes (however, see: [Human echolocation](https://en.wikipedia.org/wiki/Neuroplasticity#Human_echolocation) for an example of post-birth brain plasticity). Chemical cues guide the positions of the cell bodies and create the laminar structure. They also seem to guide long term axonal connections between distant brain regions. The cells then sprout dendrites in certain characteristic "tree-like" patterns (see: [NeuroMorpho.org](http://www.neuromorpho.org/index.jsp) for examples). *The dendrites will then form synapses with axons or other cell bodies they encounter along the way, generally based on the encountered cell type.*
  
  This last phase is probably the most analogous to the idea of random weight initialization in ANNs. Based on where the cell is positioned and its type, *the encountered other neurons will be somewhat random and so will the connections to them*. These connections are probably not going to be very strong initially but will have room to get stronger during learning (probably analogous to initial random weights between 0 and ~0.1, with 1 being the strongest possible connection). Furthermore, most cells are either inhibitory or excitatory (analogous to negative and positive weights).
  
  Keep in mind this randomization process has a heavy spatial component in real brains. The neurons are small and so they will make these connections to nearby neurons that are 10-200 microns away. The long-distance connections between brain regions are mostly "programmed-in" via genes. In most ANNs, there is generally no distance-based aspect to the initialization of connection weights (although convolutional ANNs implicitly perform something like distance-based wiring by using the sliding window).
  
  There is also the [synaptic pruning](https://en.wikipedia.org/wiki/Synaptic_pruning) phenomenon, which might be analogous to creating many low weight connections in an ANN initially (birth), training it for some number of epochs (adolescence), and then removing most low-weight connections (consolidation in adulthood).
  
  > **Comments:**
  > **username_0**: Do you by chance also have some hints on https://ai.stackexchange.com/questions/10416/is-the-thatcher-face-illusion-only-limited-to-face-recognition I haven't received any answer yet.  
  
  ---
  
  ### Answer by username_1 on 2019-01-09. Score: 0
  
  I am not an DL expert but these are my short thoughts on it:
  
  I think this is because it is believed (from an information theoretic point of view) to be the good way to avoid that the network falls into some wired state from beginning on. Remember: DNNs are nonlinear approximators for continuous functions. So they have some storage capacity to learn an amount of n function to map from input to output. When you look on topic like data leakage you will see that NNs quickly try to cheat you if they can :D. The optimization applied during training will heavily be affected by the init state. So starting with an random initialization at least avoids that your neurons do all the same at the beginning etc.
  
  **Biological reasoning:**
  From the viewpoint of a neurobiologist I can recommend you to read [Hebbian rule](https://en.m.wikipedia.org/wiki/Hebbian_theory) and how neural systems work (eg. [google how neurons find targets](https://www.quora.com/How-do-axons-branching-out-from-neurons-find-their-targets)) in general and then to compare it to what is known about how dendrite cells in the cerebrum develop their interconnections in the first 3 years after birth. In summary there are behavioral patterns in nature which could look similar, inspiring and even reasonable. But, I would say the reason why this random init. is recommend is backed by mathematical and information theoretical assumptions rather then pure biological arguments.
  
  
  ---
  
  