Answers.Body: <p>The answer to your question is that the <strong>capacity</strong> of your model (i.e. the number and type of function that your model can compute) generally increases with the number of parameters. So, a bigger model can potentially <strong>approximate better</strong> the function represented by your training data, but, at the same time, it may not take into account the test data, a phenomenon known as <strong>over-fitting</strong> the training data (i.e. fitting &quot;too much&quot; the training data).</p>
  <p>In theory, you want to fit the training data perfectly, so over-fitting should not make sense, right? The problem is that, if we just fit all the (training) data, there is no way of <em>empirically</em> checking that our model will perform well on <strong>unseen data</strong>, i.e. will it <strong>generalize</strong> to data not seen during training?
  We split our data into training and test data because of this: we want to understand whether our model will perform well also on unseen data or not.</p>
  <p>There are also some <strong>theoretical bounds</strong> that ensure you that, probabilistically and approximately, you can generalize: if you have more training data than a certain threshold, the probability that you perform badly is small. However, these theoretical bounds are often not taken into account in practice because, for example, we may not be able to collect more data to ensure that the bounds are satisfied.</p>
  <blockquote>
  <p>Surely the representations/patterns learnt in the training data will be found too in the test data.</p>
  </blockquote>
  <p>This is possibly the wrong assumption and the reason why you are confused. You may assume that both your training data and test data come from the same distribution <span class="math-container">$p(x, y)$</span>, but it does not necessarily mean that they have the same patterns. For example, I can sample e.g. 13 numbers from a Gaussian <span class="math-container">$N(0, 1)$</span>, the first 10 numbers could be very close to <span class="math-container">$0$</span> and the last <span class="math-container">$3$</span> could be close to <span class="math-container">$1$</span>. If you split this data so that your training data contains different patterns than the test data, then it is not guaranteed that you will perform well also on the test data.</p>
  <p>Finally, note that, in supervised learning, our ultimate goal when we fit models to labeled data is to learn a function (or a probability distribution over functions), where we often assume that both the training and test data are input-output pairs from our unknown target function, i.e. <span class="math-container">$y_i = f(x_i)$</span>, where <span class="math-container">$(x_i, y_i) \in D$</span> (where <span class="math-container">$D$</span> is your labelled dataset), and <span class="math-container">$f$</span> is the <strong>unknown target function</strong> (i.e. the function we want to compute with our model), so, if our model performs well on the training data but not on the test data and we assume that both training and test data come from the same function <span class="math-container">$f$</span>, there is no way that our model is computing our target function <span class="math-container">$f$</span> if it performs badly on the test data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-01T11:44:08.597Z
Answers.LastActivityDate: 2020-09-01T12:01:35.060Z
Answers.LastEditDate: 2020-09-01T12:01:35.060Z
Answers.UserDisplayName: nbro
Body: <p>I read the following from a book:</p>
  <blockquote>
  <p>You can intuitively understand the dimensionality of your representation space as “how much freedom you’re allowing the model to have when learning internal representations.” Having more units (a higher-dimensional representation space) allows your model to learn more-complex representations, but it makes the model more computationally expensive and may lead to learning unwanted patterns (patterns that will improve performance on the training data but not on the test data).</p>
  </blockquote>
  <p>Why does using a higher representation space lead to performance increase on the training data but not on the test data?</p>
  <p>Surely the representations/patterns learnt in the training data will be found too in the test data.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-01T11:22:11.213Z
Comments.Text: Which book did you read that from?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-01T23:19:34.963Z
Comments.Text: I read it from "Deep Learning with Python" by François Chollet. You can also read it here: 
  https://rpubs.com/kennyboy/503107
Comments.UserDisplayName: THAT_AI_GUY
Tags: <neural-networks><machine-learning><deep-learning><deep-neural-networks>
Title: Why does using a higher representation space lead to performance increase on the training data but not on the test data?
UserDisplayName: THAT_AI_GUY
ThreadText: # Why does using a higher representation space lead to performance increase on the training data but not on the test data?
  
  (Asked by: username_0 on 2020-09-01)
  
  I read the following from a book:
  
  > You can intuitively understand the dimensionality of your representation space as “how much freedom you’re allowing the model to have when learning internal representations.” Having more units (a higher-dimensional representation space) allows your model to learn more-complex representations, but it makes the model more computationally expensive and may lead to learning unwanted patterns (patterns that will improve performance on the training data but not on the test data).
  
  Why does using a higher representation space lead to performance increase on the training data but not on the test data?
  
  Surely the representations/patterns learnt in the training data will be found too in the test data.
  
  **Tags:** neural-networks, machine-learning, deep-learning, deep-neural-networks
  
  > **Comments:**
  > **username_1**: Which book did you read that from?  
  > **username_0**: I read it from "Deep Learning with Python" by François Chollet. You can also read it here:  https://rpubs.com/kennyboy/503107  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-01. Score: 1 | Accepted answer
  
  The answer to your question is that the **capacity** of your model (i.e. the number and type of function that your model can compute) generally increases with the number of parameters. So, a bigger model can potentially **approximate better** the function represented by your training data, but, at the same time, it may not take into account the test data, a phenomenon known as **over-fitting** the training data (i.e. fitting "too much" the training data).
  
  In theory, you want to fit the training data perfectly, so over-fitting should not make sense, right? The problem is that, if we just fit all the (training) data, there is no way of *empirically* checking that our model will perform well on **unseen data**, i.e. will it **generalize** to data not seen during training?
  We split our data into training and test data because of this: we want to understand whether our model will perform well also on unseen data or not.
  
  There are also some **theoretical bounds** that ensure you that, probabilistically and approximately, you can generalize: if you have more training data than a certain threshold, the probability that you perform badly is small. However, these theoretical bounds are often not taken into account in practice because, for example, we may not be able to collect more data to ensure that the bounds are satisfied.
  
  > Surely the representations/patterns learnt in the training data will be found too in the test data.
  
  This is possibly the wrong assumption and the reason why you are confused. You may assume that both your training data and test data come from the same distribution $p(x, y)$, but it does not necessarily mean that they have the same patterns. For example, I can sample e.g. 13 numbers from a Gaussian $N(0, 1)$, the first 10 numbers could be very close to $0$ and the last $3$ could be close to $1$. If you split this data so that your training data contains different patterns than the test data, then it is not guaranteed that you will perform well also on the test data.
  
  Finally, note that, in supervised learning, our ultimate goal when we fit models to labeled data is to learn a function (or a probability distribution over functions), where we often assume that both the training and test data are input-output pairs from our unknown target function, i.e. $y\_i = f(x\_i)$, where $(x\_i, y\_i) \in D$ (where $D$ is your labelled dataset), and $f$ is the **unknown target function** (i.e. the function we want to compute with our model), so, if our model performs well on the training data but not on the test data and we assume that both training and test data come from the same function $f$, there is no way that our model is computing our target function $f$ if it performs badly on the test data.
  
  
  ---
  
  