Answers.Body: <p>Let's quickly get out our copies of <a href="https://www.deeplearningbook.org/contents/optimization.html" rel="noreferrer"><strong>Deep Learning</strong></a> by <em>Goodfellow et al.</em> (2016). More specifically, I'm referring to <a href="https://www.deeplearningbook.org/contents/optimization.html" rel="noreferrer">page 276</a>.</p>
  <p>On this page, the authors argue for a relatively small minibatch size, since there are <em>less than linear returns</em> for estimating the gradient when increasing the minibatch size. <em>Returns</em> here refer to the reduction of the standard error of the mean (gradient per weight) computed over a minibatch.</p>
  <p>So, yes. In <em>theory</em>, having unlimited resources, you will get the best performance when averaging the loss over all samples in your dataset. In practice, however, the larger the size of minibatches, the slower the training procedure, and consequently the less the total number of weight updates that can be afforded. Reversely, in <em>practice</em>, the cheaper the weight updates, the quicker the training procedure can converge to a (subjectively) satisfactory result.</p>
  <p>Eventually, also Goodfellow et al. state that <em>rapidly</em> computing gradients leads to <em>much</em> faster convergence (in terms of total computations) for most optimization algorithms than when training them more slowly on exact gradients.</p>
  <p>So, to summarize: If the main concern is to get to a specific level of accuracy at all, go for rather low minibatch sizes, whereas you could go up to a few hundreds (as the Goodfellow et al. state as a reasonable upper bound on page 148) if you are interested in more accurate gradients for your weight updates.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-14T16:24:59.077Z
Answers.Text: Thank you for the reference to this book. I will read it
Answers.UserDisplayName: NewEndian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T21:21:58.563Z
Answers.LastActivityDate: 2020-07-14T00:21:23.303Z
Answers.LastEditDate: 2020-07-14T00:21:23.303Z
Answers.UserDisplayName: Daniel B.
Answers.Body: <p>This would be more suitable as a comment but I don't have enough points; but here's my opinion.</p>
  <p>Optimisation algorithms like gradient descent are iterative algorithms. So it is rarely possible that they arrive at the minima in 1 epoch. A single epoch means that all data points have been visited once or a certain number of data samples have been taken from a distribution. However more passes might be necessary.</p>
  <blockquote>
  <p>generated on the fly</p>
  </blockquote>
  <p>I am assuming that the data is being generated as a part of a fixed distribution. Hence multiple epochs of multiple samples is still the ideal scenario.</p>
  <p>1000 samples 1 epoch: Not enough training.<br />
  1 sample 1000 epoch: Overfitting or possibly not enough training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T23:32:41.347Z
Answers.Text: How does this answer the question?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T23:37:05.667Z
Answers.Text: My apologies @nbro. I think now the answer is more suitable.
Answers.UserDisplayName: kgkmeekg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T16:45:56.160Z
Answers.Text: I think you need to explain a bit more.... Answer seems off, although, seems correct to me.
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T23:27:34.000Z
Answers.LastActivityDate: 2020-07-14T00:24:06.287Z
Answers.LastEditDate: 2020-07-14T00:24:06.287Z
Answers.UserDisplayName: kgkmeekg
Body: <p>Should the training data be the same in each epoch?</p>
  <p>If the training data is generated on the fly, for example, is there a difference between training 1000 samples with 1 epoch or training 1000 epochs with 1 sample each?</p>
  <p>To elaborate further, samples do not need to be saved or stay in memory if they are never used again. However, if training performs best by training over the same samples repeatedly, then data would have to be stored to be reused in each epoch.</p>
  <p>More samples is generally considered advantageous. Is there a disadvantage to never seeing the same sample twice in training?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T00:27:02.337Z
Comments.Text: Can you please clarify what you mean by "data is generated on the fly"? Do you mean that at each epoch **new** data is sampled from the data-generating process?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T05:37:13.163Z
Comments.Text: An epoch refers to running over the **entire** training set. So for an epoch to actually be an epoch, the data must be the same. If the data changes each epoch, you aren't running epochs, but rather iterations. I'm confused as to why there are answers suggesting otherwise.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T16:28:18.723Z
Comments.Text: @nbro that is correct. Every sample could be unique for a virtually unlimited data stream. But if training requires that a specific set of samples must be seen repeatedly, then that data must be saved to be used again.
Comments.UserDisplayName: NewEndian
Tags: <neural-networks><machine-learning>
Title: Should the training data be the same in each epoch?
UserDisplayName: NewEndian
ThreadText: # Should the training data be the same in each epoch?
  
  (Asked by: username_0 on 2020-07-13)
  
  Should the training data be the same in each epoch?
  
  If the training data is generated on the fly, for example, is there a difference between training 1000 samples with 1 epoch or training 1000 epochs with 1 sample each?
  
  To elaborate further, samples do not need to be saved or stay in memory if they are never used again. However, if training performs best by training over the same samples repeatedly, then data would have to be stored to be reused in each epoch.
  
  More samples is generally considered advantageous. Is there a disadvantage to never seeing the same sample twice in training?
  
  **Tags:** neural-networks, machine-learning
  > **Comments:**
  > **username_1**: Can you please clarify what you mean by "data is generated on the fly"? Do you mean that at each epoch **new** data is sampled from the data-generating process?  
  > **username_2**: An epoch refers to running over the **entire** training set. So for an epoch to actually be an epoch, the data must be the same. If the data changes each epoch, you aren't running epochs, but rather iterations. I'm confused as to why there are answers suggesting otherwise.  
  > **username_0**: @nbro that is correct. Every sample could be unique for a virtually unlimited data stream. But if training requires that a specific set of samples must be seen repeatedly, then that data must be saved to be used again.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-07-13. Score: 5
  
  Let's quickly get out our copies of [**Deep Learning**](https://www.deeplearningbook.org/contents/optimization.html) by *Goodfellow et al.* (2016). More specifically, I'm referring to [page 276](https://www.deeplearningbook.org/contents/optimization.html).
  
  On this page, the authors argue for a relatively small minibatch size, since there are *less than linear returns* for estimating the gradient when increasing the minibatch size. *Returns* here refer to the reduction of the standard error of the mean (gradient per weight) computed over a minibatch.
  
  So, yes. In *theory*, having unlimited resources, you will get the best performance when averaging the loss over all samples in your dataset. In practice, however, the larger the size of minibatches, the slower the training procedure, and consequently the less the total number of weight updates that can be afforded. Reversely, in *practice*, the cheaper the weight updates, the quicker the training procedure can converge to a (subjectively) satisfactory result.
  
  Eventually, also Goodfellow et al. state that *rapidly* computing gradients leads to *much* faster convergence (in terms of total computations) for most optimization algorithms than when training them more slowly on exact gradients.
  
  So, to summarize: If the main concern is to get to a specific level of accuracy at all, go for rather low minibatch sizes, whereas you could go up to a few hundreds (as the Goodfellow et al. state as a reasonable upper bound on page 148) if you are interested in more accurate gradients for your weight updates.
  
  > **Comments:**
  > **username_0**: Thank you for the reference to this book. I will read it  
  
  ---
  
  ### Answer by username_4 on 2020-07-13. Score: 0
  
  This would be more suitable as a comment but I don't have enough points; but here's my opinion.
  
  Optimisation algorithms like gradient descent are iterative algorithms. So it is rarely possible that they arrive at the minima in 1 epoch. A single epoch means that all data points have been visited once or a certain number of data samples have been taken from a distribution. However more passes might be necessary.
  
  > generated on the fly
  
  I am assuming that the data is being generated as a part of a fixed distribution. Hence multiple epochs of multiple samples is still the ideal scenario.
  
  1000 samples 1 epoch: Not enough training.  
  1 sample 1000 epoch: Overfitting or possibly not enough training.
  
  > **Comments:**
  > **username_1**: How does this answer the question?  
  > **username_4**: My apologies @nbro. I think now the answer is more suitable.  
  > **username_5**: I think you need to explain a bit more.... Answer seems off, although, seems correct to me.  
  
  ---
  
  