Answers.Body: <p><em><strong>In a nutshell : Memorizing is not Learning</strong></em></p>
  <p>So, first let's just remind the classical use of a neural net, in <a href="https://en.wikipedia.org/wiki/Supervised_learning" rel="nofollow noreferrer">Supervised Learning</a> :</p>
  <ul>
  <li>You have a set of <span class="math-container">$(x_{train}, y_{train}) \in X \times Y$</span> pairs, and you want to extract a general mapping law from <span class="math-container">$X$</span> to <span class="math-container">$Y$</span></li>
  <li>You use a neural net function <span class="math-container">$f_{\theta} : x \rightarrow f_{\theta}(x)$</span>, with <span class="math-container">$\theta$</span> the weights (parameters) of your net.</li>
  <li>You optimise <span class="math-container">$f_{\theta}$</span> by minimizing the prediction error, represented by the loss function.</li>
  </ul>
  <p>Can this solve your question ? Well, I don't think so. With this scheme, your neural net will <em>learn</em> an appropriate mapping from the set <span class="math-container">$X$</span> to the set <span class="math-container">$Y$</span>, but this mapping is appropriate according to your loss function , not to your <span class="math-container">$(x_{train}, y_{train})$</span> pairs.</p>
  <p>Imagine that a small part of the data is wrongly labelled. A properly trained net learns to extract relevant features and thus will predict correctly the label, not like you did. So the net doesn't memorize your pairs, it infers a general law from the data, and this law may not respect each <span class="math-container">$(x_{train}, y_{train})$</span>.
  So classical Supervised Deep Learning should not memorize <span class="math-container">$(x_{train}, y_{train})$</span>  pairs.</p>
  <p>However, you could <em>memorize</em> using a net with too many parameters :  it's <a href="https://en.wikipedia.org/wiki/Overfitting" rel="nofollow noreferrer">Overfitting</a> !</p>
  <ul>
  <li>In this case, you set up the net with too many parameters. That gives too much degrees of freedom to your net, and the net will use these DoFs to exactly fit rightly each <span class="math-container">$(x_{train}, y_{train})$</span>  pair you feed during training.</li>
  <li>However, for an input <span class="math-container">$x$</span> that it never saw during training, <span class="math-container">$f_{\theta}(x)$</span> would have no meaning. That's why we say an overfitted net did not learn, and overfitting is feared by many DL practitioner.</li>
  </ul>
  <p>But as long as you want only to memorize, and not to learn, a overfitted net may be the a solution. An other solution for memorization may be <a href="https://en.wikipedia.org/wiki/Expert_system" rel="nofollow noreferrer">Expert Systems</a>, I don't know them enough to explain them, but you may check that if you want.</p>
  <p><strong>What about the brain ?</strong></p>
  <p>The matter in answering this question is that we don't really know how does the brain work. I highly recommend <a href="https://www.semanticscholar.org/paper/The-recent-excitement-about-neural-networks-Crick/66752d19577b2007d4b5e3a20c6ddeb8f1d1e600" rel="nofollow noreferrer">this article</a> discussing neural networks and the brain.</p>
  <p>Some thoughts to start :</p>
  <ol>
  <li>The brain has an incredibly huge amount of <em>parameters</em>, and has a great plasticity. In that sense, we could draw a parallel with overfitted neural networks :  so the brain could be also able to overfit, and thus to memorize by this mean.</li>
  <li>Our brain is not a feed forward network at all, we can't delimitate any <em>layer</em>, just some rough zones where we know that some specific information is processed. This makes any parallel between neural nets and the brain difficult.</li>
  <li>It's still unclear how our brain updates itself. There's no backpropagation for instance. Our overfitted networks also stem from the update processes (for instance, adding regularization to the loss helps avoiding underfitting), but we have no idea of how this works in the brain, so that's another hurdle to drawing parallels !</li>
  <li>A more personal thought : the brain is able to both learn and memorize (<em>&quot;The exception that proves the rule&quot;</em> motto shows that I think), while learning and memorizing are antonyms for neural nets...</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-20T09:53:01.397Z
Answers.Text: How can you learn without memorizing?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-20T11:07:34.650Z
Answers.Text: ahah good question :) your comment asks for a proper definition of memorizing (and maybe I should have done it in my answer). If you consider memorizing as just 'storing the raw input data', then it has little to do with learning, as learning is a statistical optimization problem (for neural nets). When you learn, what you memorize are patterns, trends that you extracted from the raw data.
Answers.UserDisplayName: 16Aghnar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-20T08:11:48.963Z
Answers.LastActivityDate: 2020-08-23T13:33:07.687Z
Answers.LastEditDate: 2020-08-23T13:33:07.687Z
Answers.UserDisplayName: 16Aghnar
Body: <p>Let's say I have pairs of keys and values of the form <span class="math-container">$(x_1, y_1), \dots, (x_N, y_N)$</span>. Then I give a neural net a key and a value, <span class="math-container">$(x_i, y_i)$</span>. For example, <span class="math-container">$x_i$</span> could be <span class="math-container">$4$</span> and <span class="math-container">$y_i$</span> could be <span class="math-container">$3$</span>, but this does not have to be the case.</p>
  <p>Is there a way to teach the neural net to output the <span class="math-container">$y_i$</span> variable every time it receives the corresponding <span class="math-container">$x_i$</span>?</p>
  <p>By the way, how do our brains perform this function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-20T09:58:45.673Z
Comments.Text: Why do you need a neural network to perform this extremely simple task of outputting 3 whenever it receives 4? Can you provide more details about your problem? For example, do you have multiple **different** parts of numbers or just one pair or what? Also, ask only one question per post, i.e. what is your main question? Is that about mapping 4 to 3 or about brains? Moreover, have you ever heard of **supervised learning**?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-20T11:54:01.123Z
Comments.Text: So, I know there are way more simple ways to do this, but I am just curious about how to replicate how our brains memorize things in a neural net. And it wouldn’t be just mapping 4 to 3. It would be any arbitrary association of numbers, like how our brains can memorize anything
Comments.UserDisplayName: iamPres
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-23T13:11:46.950Z
Comments.Text: @DuttaA as far as I know, RBMs do unsupervised learning (visible neurons are for the X data only). If I'm wrong, could you developp your point in an answer ? I'd be pleased to read you.
Comments.UserDisplayName: 16Aghnar
Tags: <neural-networks><machine-learning><deep-learning><supervised-learning>
Title: How can we teach a neural net to make arbitrary data associations?
UserDisplayName: iamPres
ThreadText: # How can we teach a neural net to make arbitrary data associations?
  
  (Asked by: username_0 on 2020-08-19)
  
  Let's say I have pairs of keys and values of the form $(x\_1, y\_1), \dots, (x\_N, y\_N)$. Then I give a neural net a key and a value, $(x\_i, y\_i)$. For example, $x\_i$ could be $4$ and $y\_i$ could be $3$, but this does not have to be the case.
  
  Is there a way to teach the neural net to output the $y\_i$ variable every time it receives the corresponding $x\_i$?
  
  By the way, how do our brains perform this function?
  
  **Tags:** neural-networks, machine-learning, deep-learning, supervised-learning
  
  > **Comments:**
  > **username_1**: Why do you need a neural network to perform this extremely simple task of outputting 3 whenever it receives 4? Can you provide more details about your problem? For example, do you have multiple **different** parts of numbers or just one pair or what? Also, ask only one question per post, i.e. what is your main question? Is that about mapping 4 to 3 or about brains? Moreover, have you ever heard of **supervised learning**?  
  > **username_0**: So, I know there are way more simple ways to do this, but I am just curious about how to replicate how our brains memorize things in a neural net. And it wouldn’t be just mapping 4 to 3. It would be any arbitrary association of numbers, like how our brains can memorize anything  
  > **username_2**: @DuttaA as far as I know, RBMs do unsupervised learning (visible neurons are for the X data only). If I'm wrong, could you developp your point in an answer ? I'd be pleased to read you.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-20. Score: 3 | Accepted answer
  
  ***In a nutshell : Memorizing is not Learning***
  
  So, first let's just remind the classical use of a neural net, in [Supervised Learning](https://en.wikipedia.org/wiki/Supervised_learning) :
  
  * You have a set of $(x\_{train}, y\_{train}) \in X \times Y$ pairs, and you want to extract a general mapping law from $X$ to $Y$
  * You use a neural net function $f\_{\theta} : x \rightarrow f\_{\theta}(x)$, with $\theta$ the weights (parameters) of your net.
  * You optimise $f\_{\theta}$ by minimizing the prediction error, represented by the loss function.
  
  Can this solve your question ? Well, I don't think so. With this scheme, your neural net will *learn* an appropriate mapping from the set $X$ to the set $Y$, but this mapping is appropriate according to your loss function , not to your $(x\_{train}, y\_{train})$ pairs.
  
  Imagine that a small part of the data is wrongly labelled. A properly trained net learns to extract relevant features and thus will predict correctly the label, not like you did. So the net doesn't memorize your pairs, it infers a general law from the data, and this law may not respect each $(x\_{train}, y\_{train})$.
  So classical Supervised Deep Learning should not memorize $(x\_{train}, y\_{train})$ pairs.
  
  However, you could *memorize* using a net with too many parameters : it's [Overfitting](https://en.wikipedia.org/wiki/Overfitting) !
  
  * In this case, you set up the net with too many parameters. That gives too much degrees of freedom to your net, and the net will use these DoFs to exactly fit rightly each $(x\_{train}, y\_{train})$ pair you feed during training.
  * However, for an input $x$ that it never saw during training, $f\_{\theta}(x)$ would have no meaning. That's why we say an overfitted net did not learn, and overfitting is feared by many DL practitioner.
  
  But as long as you want only to memorize, and not to learn, a overfitted net may be the a solution. An other solution for memorization may be [Expert Systems](https://en.wikipedia.org/wiki/Expert_system), I don't know them enough to explain them, but you may check that if you want.
  
  **What about the brain ?**
  
  The matter in answering this question is that we don't really know how does the brain work. I highly recommend [this article](https://www.semanticscholar.org/paper/The-recent-excitement-about-neural-networks-Crick/66752d19577b2007d4b5e3a20c6ddeb8f1d1e600) discussing neural networks and the brain.
  
  Some thoughts to start :
  
  1. The brain has an incredibly huge amount of *parameters*, and has a great plasticity. In that sense, we could draw a parallel with overfitted neural networks : so the brain could be also able to overfit, and thus to memorize by this mean.
  2. Our brain is not a feed forward network at all, we can't delimitate any *layer*, just some rough zones where we know that some specific information is processed. This makes any parallel between neural nets and the brain difficult.
  3. It's still unclear how our brain updates itself. There's no backpropagation for instance. Our overfitted networks also stem from the update processes (for instance, adding regularization to the loss helps avoiding underfitting), but we have no idea of how this works in the brain, so that's another hurdle to drawing parallels !
  4. A more personal thought : the brain is able to both learn and memorize (*"The exception that proves the rule"* motto shows that I think), while learning and memorizing are antonyms for neural nets...
  
  > **Comments:**
  > **username_1**: How can you learn without memorizing?  
  > **username_2**: ahah good question :) your comment asks for a proper definition of memorizing (and maybe I should have done it in my answer). If you consider memorizing as just 'storing the raw input data', then it has little to do with learning, as learning is a statistical optimization problem (for neural nets). When you learn, what you memorize are patterns, trends that you extracted from the raw data.  
  
  ---
  
  