Answers.Body: <p><strong>TL;DR</strong>: <em><strong>You always need held-out (unseen) data</strong></em> to know if a model 'generalises' to new data and to evaluate it (to compare models, hyper-parameters, assess performance over the training phase etc.)</p>
  <hr />
  <p>If the NN is trained on training data (X,Y), to predict Y from X, the system &quot;sees&quot; Y and it could simply learn to perfectly map each observed X to its observed Y, giving high training accuracy. But, then for unseen X it could give complete gibberish (low test accuracy) - you simply have no idea what it might be doing elsewhere. If all it does is get the training data right, it's effectively memorised it and is useless as a &quot;model&quot; for any other X. Typically, the whole point of the model, is to bake in some parametric rules such that if the parameters are tuned so the model fits <em>some</em> samples of X (the training set), then it should also fit other points in X-space (test set / whatever data comes in when deployed)</p>
  <p>The fact you can create new data (X,Y) each epoch makes no difference, e.g. the NN might simply memorise each new batch of data and forget the previous. It might be mildly reassuring if you then tested on a previous batch of data and found the model still performed well, but that doesn't rule out that the NN just memorises more data.</p>
  <p>You ultimately always need unseen test data to know if the model generalises, i.e. is any use in practice for new data.</p>
  <p>I don't think there is any harm &quot;using new data each epoch&quot;. In effect that is not a &quot;new epoch&quot;, it is a new batch within one very long epoch using a large (essentially unbounded) data set.</p>
  <ul>
  <li>If you save all the data and retrain on it again, that is a second epoch.</li>
  <li>Not saving the data would have implications for reproducibility (for comparison/debugging etc), unless process A can reproduce it again identically (i.e. system A effectively encodes the data set), so no harm in continuously generating new data.  In the limit, you will see the entire distribution.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T06:27:23.170Z
Answers.Text: I don't understand what you mean by memorisation, how does that play a role in this scenario? If the model doesn't see the data more than once, what effect would the potential memorisation have?
Answers.UserDisplayName: n-l-i
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T06:33:53.313Z
Answers.Text: Are you talking about the case of training on data X and then evaluating the model again on data X? I was thinking about the scenario of training on data X and monitoring the loss and other metrics the model got during the training for that batch, not by revisiting the same data once more.
Answers.UserDisplayName: n-l-i
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T08:31:12.947Z
Answers.Text: You want to learn a function from X-space to Y-space (could be infinite spaces) from a finite sample of pairs. Imagine you see those pairs.
   - what do you predict for a seen X? Probably the Y seen with it (or the average if several)
   - what do you predict for an unseen X? Who knows, depends on the "model"
  At the least you want to predict correctly for the training data but if that's *all* you do, it's essentially "memorisation", the NN acts as a look-up table. But typically it's a continuous fn so makes "predictions" for all X, it's those you want to evaluate, they could be arbitrary.
Answers.UserDisplayName: Carl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T08:34:51.250Z
Answers.Text: response edited to hopefully clarify
Answers.UserDisplayName: Carl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T05:12:40.730Z
Answers.Text: I still don't think we are talking about the same thing. "If all it does is get the training data right, it's effectively memorised it and is useless as a 'model' for any other X." If it memories its training data and gets low accuracy for other data, then it will get low accuracy on subsequent training data since it would be training exclusively on unseen data. So if it has poor generalisability then that would never be a surprise, even in the case of no validation dataset. Your conclusions might be fully correct but I don't think your reasoning supports them.
Answers.UserDisplayName: n-l-i
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:29:54.770Z
Answers.Text: If the model trains on some training data, it *might* get low accuracy on some new data (without training on it) or not. But if you train on that data and evaluate the model on it, it tells you nothing about how it the model will perform on the next batch. If when new data comes in, you evaluate the model on it immediately (with frozen parameters/without training on it) and the model performs well, then you are evaluating generalisability. But the minute you train the model on data and evaluate on that data, you are only testing its ability to fit (or worst case memorise) not generalise
Answers.UserDisplayName: Carl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-30T15:18:50.630Z
Answers.LastActivityDate: 2023-08-01T08:40:13.453Z
Answers.LastEditDate: 2023-08-01T08:40:13.453Z
Answers.UserDisplayName: Carl
Body: <p>I have two systems, a system <code>A</code> that generates some data <code>X</code> and a system <code>B</code> that calculates some <code>Y</code> given an <code>X</code>. System <code>B</code> is comparatively very slow to run on the available hardware. This is why I'm thinking of training a simple neural network to approximate <code>X</code> -&gt; <code>Y</code>.</p>
  <p>The question that arose from this is how to create the training, validation, and testing datasets. Since I could generate new training data each epoch using my existing systems, is there even a need for the validation and testing datasets? Are there any benefits to not using new training data each epoch?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T08:11:31.577Z
Comments.Text: If there is no validation and test set, how do you verify `X` can approximate `Y`?
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T09:09:33.010Z
Comments.Text: I was thinking that it could be enough to verify that the model can approximate `Y` over the training data if the model always trains on previously unseen data, i.e. that lower training-set loss implies lower validation- and test-set losses in the absence of validation- and test-sets.
Comments.UserDisplayName: n-l-i
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T09:47:09.773Z
Comments.Text: OK, say I generate 1000 samples with `A` and their corresponding targets `Y` with `B`. Now, I feed all 1000 (unseen) samples to train the model. It gives me 1000 predictions `P`. How shall verify that the `P`s are good approximation of `Y`?
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T11:22:58.227Z
Comments.Text: That could be verified for example by running the loss function over the `P`s and `Y`s. If the loss is low accross all samples then the predictions were quite good for that set. One could also look at other metrics such as accuracy etc.
Comments.UserDisplayName: n-l-i
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-31T01:17:45.363Z
Comments.Text: @Carl's answer basically says it. `P` is training loss, we cannot assume it to generalize. If you don't believe, try first generate and hold out a small test set (say 200 samples), put it away, and generate a lot more training samples to train for a long time until the loss saturates. Then, test the model against the test set. Very likely you will see severe overfitting.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-31T01:20:58.967Z
Comments.Text: The key takeaway is in @Carl's word, 'The fact you can create new data (X,Y) each epoch makes no difference'. I can see that you are tempted because of a slow-running `B`, but unfortunately this is how science works.
Comments.UserDisplayName: lpounng
Tags: <neural-networks><supervised-learning>
Title: Is validation data needed with generated training data?
UserDisplayName: n-l-i
ThreadText: # Is validation data needed with generated training data?
  
  (Asked by: username_0 on 2023-07-28)
  
  I have two systems, a system `A` that generates some data `X` and a system `B` that calculates some `Y` given an `X`. System `B` is comparatively very slow to run on the available hardware. This is why I'm thinking of training a simple neural network to approximate `X` -> `Y`.
  
  The question that arose from this is how to create the training, validation, and testing datasets. Since I could generate new training data each epoch using my existing systems, is there even a need for the validation and testing datasets? Are there any benefits to not using new training data each epoch?
  
  **Tags:** neural-networks, supervised-learning
  
  > **Comments:**
  > **username_1**: If there is no validation and test set, how do you verify `X` can approximate `Y`?  
  > **username_0**: I was thinking that it could be enough to verify that the model can approximate `Y` over the training data if the model always trains on previously unseen data, i.e. that lower training-set loss implies lower validation- and test-set losses in the absence of validation- and test-sets.  
  > **username_1**: OK, say I generate 1000 samples with `A` and their corresponding targets `Y` with `B`. Now, I feed all 1000 (unseen) samples to train the model. It gives me 1000 predictions `P`. How shall verify that the `P`s are good approximation of `Y`?  
  > **username_0**: That could be verified for example by running the loss function over the `P`s and `Y`s. If the loss is low accross all samples then the predictions were quite good for that set. One could also look at other metrics such as accuracy etc.  
  > **username_1**: @username_2's answer basically says it. `P` is training loss, we cannot assume it to generalize. If you don't believe, try first generate and hold out a small test set (say 200 samples), put it away, and generate a lot more training samples to train for a long time until the loss saturates. Then, test the model against the test set. Very likely you will see severe overfitting.  
  > **username_1**: The key takeaway is in @username_2's word, 'The fact you can create new data (X,Y) each epoch makes no difference'. I can see that you are tempted because of a slow-running `B`, but unfortunately this is how science works.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-30. Score: 3
  
  **TL;DR**: ***You always need held-out (unseen) data*** to know if a model 'generalises' to new data and to evaluate it (to compare models, hyper-parameters, assess performance over the training phase etc.)
  
  ---
  
  If the NN is trained on training data (X,Y), to predict Y from X, the system "sees" Y and it could simply learn to perfectly map each observed X to its observed Y, giving high training accuracy. But, then for unseen X it could give complete gibberish (low test accuracy) - you simply have no idea what it might be doing elsewhere. If all it does is get the training data right, it's effectively memorised it and is useless as a "model" for any other X. Typically, the whole point of the model, is to bake in some parametric rules such that if the parameters are tuned so the model fits *some* samples of X (the training set), then it should also fit other points in X-space (test set / whatever data comes in when deployed)
  
  The fact you can create new data (X,Y) each epoch makes no difference, e.g. the NN might simply memorise each new batch of data and forget the previous. It might be mildly reassuring if you then tested on a previous batch of data and found the model still performed well, but that doesn't rule out that the NN just memorises more data.
  
  You ultimately always need unseen test data to know if the model generalises, i.e. is any use in practice for new data.
  
  I don't think there is any harm "using new data each epoch". In effect that is not a "new epoch", it is a new batch within one very long epoch using a large (essentially unbounded) data set.
  
  * If you save all the data and retrain on it again, that is a second epoch.
  * Not saving the data would have implications for reproducibility (for comparison/debugging etc), unless process A can reproduce it again identically (i.e. system A effectively encodes the data set), so no harm in continuously generating new data. In the limit, you will see the entire distribution.
  
  > **Comments:**
  > **username_0**: I don't understand what you mean by memorisation, how does that play a role in this scenario? If the model doesn't see the data more than once, what effect would the potential memorisation have?  
  > **username_0**: Are you talking about the case of training on data X and then evaluating the model again on data X? I was thinking about the scenario of training on data X and monitoring the loss and other metrics the model got during the training for that batch, not by revisiting the same data once more.  
  > **username_2**: You want to learn a function from X-space to Y-space (could be infinite spaces) from a finite sample of pairs. Imagine you see those pairs.  - what do you predict for a seen X? Probably the Y seen with it (or the average if several)  - what do you predict for an unseen X? Who knows, depends on the "model" At the least you want to predict correctly for the training data but if that's *all* you do, it's essentially "memorisation", the NN acts as a look-up table. But typically it's a continuous fn so makes "predictions" for all X, it's those you want to evaluate, they could be arbitrary.  
  > **username_2**: response edited to hopefully clarify  
  > **username_0**: I still don't think we are talking about the same thing. "If all it does is get the training data right, it's effectively memorised it and is useless as a 'model' for any other X." If it memories its training data and gets low accuracy for other data, then it will get low accuracy on subsequent training data since it would be training exclusively on unseen data. So if it has poor generalisability then that would never be a surprise, even in the case of no validation dataset. Your conclusions might be fully correct but I don't think your reasoning supports them.  
  > **username_2**: If the model trains on some training data, it *might* get low accuracy on some new data (without training on it) or not. But if you train on that data and evaluate the model on it, it tells you nothing about how it the model will perform on the next batch. If when new data comes in, you evaluate the model on it immediately (with frozen parameters/without training on it) and the model performs well, then you are evaluating generalisability. But the minute you train the model on data and evaluate on that data, you are only testing its ability to fit (or worst case memorise) not generalise  
  
  ---
  
  