Answers.Body: <p>There is a trade-off between the:</p>
  <ul>
  <li>Memory capacity of computation device</li>
  <li>Quality of gradient approximation</li>
  <li>Generalization ability of the network</li>
  </ul>
  <p><strong>Memory capacity</strong></p>
  <p>I would say, that it is possible to process the whole dataset at once only for small enough dataset and image resolution (or any other measure of the data sample size - text sequence length, number of points in point cloud, whatever). Indeed, one can compute the update by accumulation several gradients (this functionality is provided in <a href="https://pytorch-lightning.readthedocs.io/en/latest/advanced/training_tricks.html" rel="nofollow noreferrer">Pytorch Lightning</a>) and only then update the parameters, but this would be rather slow.</p>
  <p>For dataset of ImageNet size you will need to wait several hours (on single of few GPU's) to make a single update, which is prohibitive.</p>
  <p>Usually, one is tempted to take large batches in order to traverse the training dataset as fast as possible, due to the large parallelization ability of modern computation accelerators (GPUs, TPUs, e.t.c).
  Typical batch size is of order <strong>1k</strong>-<strong>2k</strong> on <a href="https://arxiv.org/abs/2110.00476" rel="nofollow noreferrer">ImageNet</a>.</p>
  <p><strong>Generalization</strong></p>
  <p>There is quite a lot of research devoted to the optimal choice of the <code>batch size</code>. A smaller batch size is said to be beneficial in the initial stage of training since it allows to find better and wider optima. Wider optima lead to more robust behavior of the training procedure and have less train-test error discrepancy, as mentioned, for instance, in <a href="https://arxiv.org/abs/1803.05407" rel="nofollow noreferrer">this paper</a>.</p>
  <p>In the consequent stage of training, when one is close to the optimum, the typical strategy is to decrease the amount of &quot;noise&quot; in SGD. This objective can be achieved in several ways:</p>
  <ul>
  <li>Decrease of the learning rate (making smaller steps)</li>
  <li><a href="https://arxiv.org/pdf/1711.00489.pdf" rel="nofollow noreferrer">Increase of the batch size</a></li>
  </ul>
  <p>Assuming that the individual gradients are uncorrelated (which may be a good or bad assumption), the standard deviation of the mini-batch gradients will scale roughly as:
  <span class="math-container">$$
  \mathcal{O}\left(\frac{1}{\sqrt{N}}\right)
  $$</span></p>
  <p><strong>The answer</strong></p>
  <p>Provided your computational resource allow, or you can allow for the accumulation of a large number of batches - you can try to update the weights, based on the gradients on the whole dataset. But the learning rate decay is a simpler strategy.</p>
  <p>One step on the dataset with <strong>1M</strong> samples with learning rate <span class="math-container">$\eta$</span> is roughly equivalent to <strong>1K</strong> steps with batch size <strong>1K</strong> and learning rate <span class="math-container">$\eta / 1000$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-03T10:03:31.020Z
Answers.LastActivityDate: 2021-11-03T10:03:31.020Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the general area where convergence happens and it kind of oscillates around that area.</p>
  <p>Is it possible to use stochastic gradient descent at the beginning and find the way to a general convergence and then use batch gradient descent on only a few training examples out of the huge dataset to get even closer to the exact point of convergence?</p>
  <p>I know that a model with a cost function that's a bit away from the lowest value for the cost function performs well in stochastic gradient descent, but assuming you want better results, will this work well?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-03T12:17:19.743Z
Comments.Text: I don't understand this sentence "_I know that a model with a cost function that's a bit away from the lowest value for the cost function performs well in stochastic gradient descent_". What do you mean by "_a model with a cost function that's a bit away from the lowest value for the cost function_". There's probably a typo there, which you should fix.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-06T16:05:01.463Z
Comments.Text: I meant that, I heard that in stochastic gradient descent, if the cost function is not *actually* at the global optimum but is somewhere close by, it won't matter a lot because you still get almost similar results.
Comments.UserDisplayName: Robo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-06T17:00:32.983Z
Comments.Text: Ok, can you please edit your post to clarify that?
Comments.UserDisplayName: nbro
Tags: <machine-learning><gradient-descent><stochastic-gradient-descent><mini-batch-gradient-descent>
Title: Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?
UserDisplayName: Robo
ThreadText: # Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?
  
  (Asked by: username_0 on 2021-11-03)
  
  Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the general area where convergence happens and it kind of oscillates around that area.
  
  Is it possible to use stochastic gradient descent at the beginning and find the way to a general convergence and then use batch gradient descent on only a few training examples out of the huge dataset to get even closer to the exact point of convergence?
  
  I know that a model with a cost function that's a bit away from the lowest value for the cost function performs well in stochastic gradient descent, but assuming you want better results, will this work well?
  
  **Tags:** machine-learning, gradient-descent, stochastic-gradient-descent, mini-batch-gradient-descent
  > **Comments:**
  > **username_1**: I don't understand this sentence "_I know that a model with a cost function that's a bit away from the lowest value for the cost function performs well in stochastic gradient descent_". What do you mean by "_a model with a cost function that's a bit away from the lowest value for the cost function_". There's probably a typo there, which you should fix.  
  > **username_0**: I meant that, I heard that in stochastic gradient descent, if the cost function is not *actually* at the global optimum but is somewhere close by, it won't matter a lot because you still get almost similar results.  
  > **username_1**: Ok, can you please edit your post to clarify that?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-03. Score: 2 | Accepted answer
  
  There is a trade-off between the:
  
  * Memory capacity of computation device
  * Quality of gradient approximation
  * Generalization ability of the network
  
  **Memory capacity**
  
  I would say, that it is possible to process the whole dataset at once only for small enough dataset and image resolution (or any other measure of the data sample size - text sequence length, number of points in point cloud, whatever). Indeed, one can compute the update by accumulation several gradients (this functionality is provided in [Pytorch Lightning](https://pytorch-lightning.readthedocs.io/en/latest/advanced/training_tricks.html)) and only then update the parameters, but this would be rather slow.
  
  For dataset of ImageNet size you will need to wait several hours (on single of few GPU's) to make a single update, which is prohibitive.
  
  Usually, one is tempted to take large batches in order to traverse the training dataset as fast as possible, due to the large parallelization ability of modern computation accelerators (GPUs, TPUs, e.t.c).
  Typical batch size is of order **1k**-**2k** on [ImageNet](https://arxiv.org/abs/2110.00476).
  
  **Generalization**
  
  There is quite a lot of research devoted to the optimal choice of the `batch size`. A smaller batch size is said to be beneficial in the initial stage of training since it allows to find better and wider optima. Wider optima lead to more robust behavior of the training procedure and have less train-test error discrepancy, as mentioned, for instance, in [this paper](https://arxiv.org/abs/1803.05407).
  
  In the consequent stage of training, when one is close to the optimum, the typical strategy is to decrease the amount of "noise" in SGD. This objective can be achieved in several ways:
  
  * Decrease of the learning rate (making smaller steps)
  * [Increase of the batch size](https://arxiv.org/pdf/1711.00489.pdf)
  
  Assuming that the individual gradients are uncorrelated (which may be a good or bad assumption), the standard deviation of the mini-batch gradients will scale roughly as:
  $$
  \mathcal{O}\left(\frac{1}{\sqrt{N}}\right)
  $$
  
  **The answer**
  
  Provided your computational resource allow, or you can allow for the accumulation of a large number of batches - you can try to update the weights, based on the gradients on the whole dataset. But the learning rate decay is a simpler strategy.
  
  One step on the dataset with **1M** samples with learning rate $\eta$ is roughly equivalent to **1K** steps with batch size **1K** and learning rate $\eta / 1000$.
  
  
  ---
  
  