Answers.Body: <p>I may not be able to fully address your question but I want to provide some comparisons between ANNs and the Brain.</p>
  <ol>
  <li><strong>Plasticity.</strong> The brain can adapt/grow/remove its connections among neurons as required. For example, learning a new skill (like playing a new instrument) causes the grow of new connections that you can see as shortcuts between the interested parts of the brain such that to maximize your performance at that task. Instead, ANNs are just static: once the training is completed, the values of the weights are fixed regardless the new experience (or data samples) they are asked to predict.</li>
  <li><strong>Modularity and multi-modality.</strong> The brain is made of many parts (or modules) which are often interconnected in a very complex way. Moreover, the brain can process multi-modal information like visual, audio, and, e.g., tactile stimuli. In these days, modern ANNs can also exploits multi-modal inputs although still not as good as our brain.</li>
  <li><strong>Distributed memory.</strong> The brain has specialized areas, like the Hippocampus if I'm not wrong, that handle memory related stuff. Moreover, some argue that the memories are also stored in many different places because the Hippocampus takes a small area within the brain. Also, memories seem to be stored in a sort of &quot;compressed&quot; way such that you retain only the vital information, which can be later &quot;reconstructed&quot; or even &quot;retrieved&quot;. Regarding this point there could be a similarity with (variational) auto-encoders (a kind of ANN), which learns to reconstruct its inputs from a compressed representation. In some applications, the goal is to learn an efficient thus compact representation of the input data.</li>
  <li><strong>Attention.</strong> Each fraction of second the brain experience tons of data (by its multiple senses), which, most of them, is discarded for the sake of not becoming crazy just in few seconds. In the brain we distinguish between <em>voluntary</em> (top-down) and <em>automatic</em> (bottom-up) types of attention. Basically, in the former case we decide what to attend whereas in the latter case, which is also less understood, there are things that are found to capture our attention (like a sudden movement or change in luminosity/color.) In Transformer-based ANNs you have an attention mechanism that can select which parts of the input to focus more. Also, in general, there are portion of the input that, let's say, capture the attention of (or better said, <em>mislead</em>) the ANN. Personally I find this mechanism in ANNs to be similar to bottom-up attention but, also in this case, this is very little understood.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-20T21:27:28.137Z
Answers.LastActivityDate: 2023-10-20T21:27:28.137Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>The essence and foundations of memory in ANNs (neural network) and the brain...</p>
  <p>I guess that the neural network or neuron worked in saving small/tiny portion of information.</p>
  <p>Here are two separate sentences from <a href="https://www.cs.cmu.edu/%7E./epxing/Class/10715/reading/McCulloch.and.Pitts.pdf" rel="nofollow noreferrer">a research paper by Walter Pitts in 1943</a>:</p>
  <ul> This illusion makes very clear the dependence of the correspondence between perception and the “external world” upon the specific structural properties of the intervening nervous net. </ul>
  <ul> From the irreciprocity of causality it follows that even if the net be known, though we may predict future from present activities, we can deduce neither afferent from central, nor central from efferent, nor past from present activities--conclusions which are reinforced by the contradictory testimony of eye-witnesses, by the difficulty of diagnosing differentially the organically diseased, the hysteric and the malingerer, and by comparing one’s own m&#x332;e&#x332;m&#x332;o&#x332;r&#x332;i&#x332;e&#x332;s&#x332; or recollections with his contemporaneous records. </ul>
  <p>    Reciprocity: mutual dependence, action, or influence</p>
  <p>The paper title is &quot;A Logical Calculus of The Idea Immanent in Nervous Activity&quot;. If you want to see how mimicking the perception of brain started, you may read it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-20T22:57:39.657Z
Answers.LastActivityDate: 2023-10-21T06:35:53.813Z
Answers.LastEditDate: 2023-10-21T06:35:53.813Z
Answers.UserDisplayName: Cloud Cho
Body: <p>I am learning Computational Neuroscience. If I do reasoning in my mind, especially Classifying an object and Deducing to figure out its qualities, I have to memorize the category and the category's definition, qualities in the past.</p>
  <p>So what are the Essence and foundations of Memory in ANNs and the Brain? Memories can fade, so memories are not simply the parameters in the ANNs. Are there any Transformers-like parts in our brain, since the attention mechanism is quite an easy way to calculate the similarity between two items?</p>
  <p>Thank you very much for all your discussions and answers.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-17T17:22:26.673Z
Comments.Text: there are little to no dualities between ANN and how the brain works, we like to pretend that, but no... and btw, attention in Transformers, has nothing to do with the "brain" attention, however it has some dualities
Comments.UserDisplayName: Alberto
Tags: <agi><biology><memory>
Title: What is the Essence of Memory of ANNs and Brain?
UserDisplayName: Forest Liu
ThreadText: # What is the Essence of Memory of ANNs and Brain?
  
  (Asked by: username_0 on 2023-10-17)
  
  I am learning Computational Neuroscience. If I do reasoning in my mind, especially Classifying an object and Deducing to figure out its qualities, I have to memorize the category and the category's definition, qualities in the past.
  
  So what are the Essence and foundations of Memory in ANNs and the Brain? Memories can fade, so memories are not simply the parameters in the ANNs. Are there any Transformers-like parts in our brain, since the attention mechanism is quite an easy way to calculate the similarity between two items?
  
  Thank you very much for all your discussions and answers.
  
  **Tags:** agi, biology, memory
  
  > **Comments:**
  > **username_1**: there are little to no dualities between ANN and how the brain works, we like to pretend that, but no... and btw, attention in Transformers, has nothing to do with the "brain" attention, however it has some dualities  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-10-20. Score: 2
  
  I may not be able to fully address your question but I want to provide some comparisons between ANNs and the Brain.
  
  1. **Plasticity.** The brain can adapt/grow/remove its connections among neurons as required. For example, learning a new skill (like playing a new instrument) causes the grow of new connections that you can see as shortcuts between the interested parts of the brain such that to maximize your performance at that task. Instead, ANNs are just static: once the training is completed, the values of the weights are fixed regardless the new experience (or data samples) they are asked to predict.
  2. **Modularity and multi-modality.** The brain is made of many parts (or modules) which are often interconnected in a very complex way. Moreover, the brain can process multi-modal information like visual, audio, and, e.g., tactile stimuli. In these days, modern ANNs can also exploits multi-modal inputs although still not as good as our brain.
  3. **Distributed memory.** The brain has specialized areas, like the Hippocampus if I'm not wrong, that handle memory related stuff. Moreover, some argue that the memories are also stored in many different places because the Hippocampus takes a small area within the brain. Also, memories seem to be stored in a sort of "compressed" way such that you retain only the vital information, which can be later "reconstructed" or even "retrieved". Regarding this point there could be a similarity with (variational) auto-encoders (a kind of ANN), which learns to reconstruct its inputs from a compressed representation. In some applications, the goal is to learn an efficient thus compact representation of the input data.
  4. **Attention.** Each fraction of second the brain experience tons of data (by its multiple senses), which, most of them, is discarded for the sake of not becoming crazy just in few seconds. In the brain we distinguish between *voluntary* (top-down) and *automatic* (bottom-up) types of attention. Basically, in the former case we decide what to attend whereas in the latter case, which is also less understood, there are things that are found to capture our attention (like a sudden movement or change in luminosity/color.) In Transformer-based ANNs you have an attention mechanism that can select which parts of the input to focus more. Also, in general, there are portion of the input that, let's say, capture the attention of (or better said, *mislead*) the ANN. Personally I find this mechanism in ANNs to be similar to bottom-up attention but, also in this case, this is very little understood.
  
  
  ---
  
  ### Answer by username_3 on 2023-10-20. Score: 1
  
  The essence and foundations of memory in ANNs (neural network) and the brain...
  
  I guess that the neural network or neuron worked in saving small/tiny portion of information.
  
  Here are two separate sentences from [a research paper by Walter Pitts in 1943](https://www.cs.cmu.edu/%7E./epxing/Class/10715/reading/McCulloch.and.Pitts.pdf):
  
  This illusion makes very clear the dependence of the correspondence between perception and the “external world” upon the specific structural properties of the intervening nervous net.
  
  From the irreciprocity of causality it follows that even if the net be known, though we may predict future from present activities, we can deduce neither afferent from central, nor central from efferent, nor past from present activities--conclusions which are reinforced by the contradictory testimony of eye-witnesses, by the difficulty of diagnosing differentially the organically diseased, the hysteric and the malingerer, and by comparing one’s own m̲e̲m̲o̲r̲i̲e̲s̲ or recollections with his contemporaneous records.
  
      Reciprocity: mutual dependence, action, or influence
  
  The paper title is "A Logical Calculus of The Idea Immanent in Nervous Activity". If you want to see how mimicking the perception of brain started, you may read it.
  
  
  ---
  
  