Answers.Body: <p>Ideally, you need to update weights by going over all the samples in the dataset. This is called as <strong>Batch Gradient Descent</strong>. But, as the no. of training examples increases, the computation becomes huge and training will be very slow. With the advent of deep learning, training size is in millions and computation using all training examples is very impractical and very slow. </p>
  
  <p>This is where, two optimization techniques became prominent. </p>
  
  <ol>
  <li><strong>Mini-Batch Gradient Descent</strong></li>
  <li><strong>Stochastic Gradient Descent (SGD)</strong></li>
  </ol>
  
  <p>In mini-batch gradient descent, you use a batch size that is considerably less than total no. of training examples and update your weights after passing through these examples. </p>
  
  <p>In stochastic gradient descent, you update the weights after passing through each training example. </p>
  
  <p>Coming to advantages and disadvantages of the three methods we discussed.</p>
  
  <ul>
  <li><p>Batch gradient descent gradually converges to the global minimum but
  it is slow and requires huge computing power.</p></li>
  <li><p>Stochastic gradient descent converges fast but not to the global
  minimum, it     converges somewhere near to the global minimum and
  hovers    around that point, but doesn't converge ever to the global
  minimum.    But, the converged point in Stochastic gradient descent
  is good    enough for all practical purposes.</p></li>
  <li><p>Mini-Batch gradient is a trade-off the above two methods. But, if you
  have a vectorized implementation of the weights updation and you<br>
  are training with a multi-core setup or submitting the training to<br>
  multiple machines, this is the best method both in terms of time for 
  training and convergence to global minimum.</p></li>
  </ul>
  
  <p>You can plot the cost function, w.r.t the no. of iterations to understand the difference between convergence in all the 3 types of gradient descent. </p>
  
  <ul>
  <li><p>Batch gradient descent plot falls smoothly and slowly and gets stabilized and
  gets to global minimum. </p></li>
  <li><p>Stochastic gradient descent plot will have    oscillations, will fall
  fast but hovers around global minimum.</p></li>
  </ul>
  
  <p>These are some blogs where there is detailed explanation of advantages, disadvantages of each method and also graphs of how cost function changes for all the three methods with iterations. </p>
  
  <p><a href="https://adventuresinmachinelearning.com/stochastic-gradient-descent/" rel="nofollow noreferrer">https://adventuresinmachinelearning.com/stochastic-gradient-descent/</a></p>
  
  <p><a href="https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/" rel="nofollow noreferrer">https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-27T12:56:09.767Z
Answers.LastActivityDate: 2019-05-27T13:01:40.860Z
Answers.LastEditDate: 2019-05-27T13:01:40.860Z
Answers.UserDisplayName: Kartik Podugu
Answers.Body: <blockquote>
    <p>Should I be changing the weights/biases on every single sample before moving on to the next sample, </p>
  </blockquote>
  
  <p>You can do this, it is called <em>stochastic</em> gradient descent (SGD) and typically you will shuffle the dataset before working through it each time.</p>
  
  <blockquote>
    <p>or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?</p>
  </blockquote>
  
  <p>You can do this, it is called <em>batch</em> gradient descent, or in some cases (especially in older resources) just assumed as the normal approach and called gradient descent.</p>
  
  <p>Each approach offers advantages and disadvantages. In general:</p>
  
  <ul>
  <li><p>SGD makes each update sooner in terms of amount of data that has been processed. So you may need less epochs before converging on reasonable values.</p></li>
  <li><p>SGD does more processing per sample (because it updates more frequently), so is also slower in the sense that it will take longer to process each sample. </p></li>
  <li><p>SGD can take less advantage of parallelisation, as the update steps mean you have to run each data item serially (as the weights have changed and error/gradient results are calculated for a specific set of weights).</p></li>
  <li><p>SGD individual steps make typically only very rough guesses at the correct gradients to change weights in. This is both a disadvantage (performance of the NN against the objective on the training set can decrease as well as increase) and an advantage (there is less likelihood of getting stuck in a local stationary point due the "jitter" these random differences cause).</p></li>
  </ul>
  
  <p>What happens in practice is that most software allows you to compromise between batch processing and single-sample processing, to try and get the best performance and update characteristics. This is called mini-batch processing, which involves:</p>
  
  <ul>
  <li><p>Shuffling the dataset at the start of each epoch.</p></li>
  <li><p>Working through the shuffled data, N items per time where N might vary from maybe 10 to 1000, depending on the problem and any constraints on the hardware. A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.</p></li>
  <li><p>Calculate the update required for each small batch, then apply it.</p></li>
  </ul>
  
  <p>This is nowadays the most common update method that most neural network libraries assume, and they almost universally will accept a batch size parameter in the training API. Most of the libraries will still call simple optimisers that do that SGD; technically it is true, the gradients calculated are still somewhat randomised due to not using the full batch, but you may find this called mini-batch gradient descent in some older papers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T06:40:55.677Z
Answers.Text: @DuttaA e.g. https://stackoverflow.com/questions/46654424/how-to-calculate-optimal-batch-size and https://www.pugetsystems.com/labs/hpc/GPU-Memory-Size-and-Deep-Learning-Performance-batch-size-12GB-vs-32GB----1080Ti-vs-Titan-V-vs-GV100-1146/
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T16:27:42.557Z
Answers.Text: "A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel." -- You have it backwards! The first heuristic is to process the *smallest* mini-batch size that results in acceptable performance. With many models, however, you hit memory limits before you saturate hardware efficiency, so you end up running the largest model that will fit into RAM. Generally, though, smaller batch sizes find better minima since they exhibit more stochasticity. A caveat is that batch-norm breaks with very small batch sizes.
Answers.UserDisplayName: Aleksandr Dubinsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-26T07:41:37.107Z
Answers.Text: @AleksandrDubinsky: Yes, but I don't want to import so much detail and caveats into the answer on that topic. I'm not 100% sure of the heuristic you suggest being the most used/popular; there are 2 performance metrics (training time and cross-validation results), and more stochasticity is not *necessarily* a good thing for best generalised metrics. As you note, adding in batch norm (and maybe other things) makes the landscape even more complex, and a bit much for this answer to cover. Many people do seem to end up hitting GPU RAM limits with low training time being the primary goal.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T06:01:37.007Z
Answers.LastActivityDate: 2019-05-26T07:47:33.887Z
Answers.LastEditDate: 2019-05-26T07:47:33.887Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm trying to learn neural networks by watching <a href="https://www.youtube.com/watch?v=tIeHLnjs5U8&amp;t=17s" rel="noreferrer">this series of videos</a> and implementing a simple neural network in Python.</p>
  
  <p>Here's one of the things I'm wondering about: I'm training the neural network on sample data, and I've got 1,000 samples. The training consists of gradually changing the weights and biases to make the cost function result in a smaller cost.</p>
  
  <p><strong>My question:</strong> Should I be changing the weights/biases on every single sample before moving on to the next sample, or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?</p>
  
Tags: <neural-networks><deep-learning><gradient-descent><stochastic-gradient-descent><mini-batch-gradient-descent>
Title: Is neural networks training done one-by-one?
UserDisplayName: Ram Rachum
ThreadText: # Is neural networks training done one-by-one?
  
  (Asked by: username_0 on 2019-05-25)
  
  I'm trying to learn neural networks by watching [this series of videos](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=17s) and implementing a simple neural network in Python.
  
  Here's one of the things I'm wondering about: I'm training the neural network on sample data, and I've got 1,000 samples. The training consists of gradually changing the weights and biases to make the cost function result in a smaller cost.
  
  **My question:** Should I be changing the weights/biases on every single sample before moving on to the next sample, or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?
  
  **Tags:** neural-networks, deep-learning, gradient-descent, stochastic-gradient-descent, mini-batch-gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-05-25. Score: 11 | Accepted answer
  
  > Should I be changing the weights/biases on every single sample before moving on to the next sample,
  
  You can do this, it is called *stochastic* gradient descent (SGD) and typically you will shuffle the dataset before working through it each time.
  
  > or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?
  
  You can do this, it is called *batch* gradient descent, or in some cases (especially in older resources) just assumed as the normal approach and called gradient descent.
  
  Each approach offers advantages and disadvantages. In general:
  
  * SGD makes each update sooner in terms of amount of data that has been processed. So you may need less epochs before converging on reasonable values.
  * SGD does more processing per sample (because it updates more frequently), so is also slower in the sense that it will take longer to process each sample.
  * SGD can take less advantage of parallelisation, as the update steps mean you have to run each data item serially (as the weights have changed and error/gradient results are calculated for a specific set of weights).
  * SGD individual steps make typically only very rough guesses at the correct gradients to change weights in. This is both a disadvantage (performance of the NN against the objective on the training set can decrease as well as increase) and an advantage (there is less likelihood of getting stuck in a local stationary point due the "jitter" these random differences cause).
  
  What happens in practice is that most software allows you to compromise between batch processing and single-sample processing, to try and get the best performance and update characteristics. This is called mini-batch processing, which involves:
  
  * Shuffling the dataset at the start of each epoch.
  * Working through the shuffled data, N items per time where N might vary from maybe 10 to 1000, depending on the problem and any constraints on the hardware. A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.
  * Calculate the update required for each small batch, then apply it.
  
  This is nowadays the most common update method that most neural network libraries assume, and they almost universally will accept a batch size parameter in the training API. Most of the libraries will still call simple optimisers that do that SGD; technically it is true, the gradients calculated are still somewhat randomised due to not using the full batch, but you may find this called mini-batch gradient descent in some older papers.
  
  > **Comments:**
  > **username_2**: @DuttaA e.g. https://stackoverflow.com/questions/46654424/how-to-calculate-optimal-batch-size and https://www.pugetsystems.com/labs/hpc/GPU-Memory-Size-and-Deep-Learning-Performance-batch-size-12GB-vs-32GB----1080Ti-vs-Titan-V-vs-GV100-1146/  
  > **username_3**: "A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel." -- You have it backwards! The first heuristic is to process the *smallest* mini-batch size that results in acceptable performance. With many models, however, you hit memory limits before you saturate hardware efficiency, so you end up running the largest model that will fit into RAM. Generally, though, smaller batch sizes find better minima since they exhibit more stochasticity. A caveat is that batch-norm breaks with very small batch sizes.  
  > **username_2**: @AleksandrDubinsky: Yes, but I don't want to import so much detail and caveats into the answer on that topic. I'm not 100% sure of the heuristic you suggest being the most used/popular; there are 2 performance metrics (training time and cross-validation results), and more stochasticity is not *necessarily* a good thing for best generalised metrics. As you note, adding in batch norm (and maybe other things) makes the landscape even more complex, and a bit much for this answer to cover. Many people do seem to end up hitting GPU RAM limits with low training time being the primary goal.  
  
  ---
  
  ### Answer by username_1 on 2019-05-27. Score: 1
  
  Ideally, you need to update weights by going over all the samples in the dataset. This is called as **Batch Gradient Descent**. But, as the no. of training examples increases, the computation becomes huge and training will be very slow. With the advent of deep learning, training size is in millions and computation using all training examples is very impractical and very slow.
  
  This is where, two optimization techniques became prominent.
  
  1. **Mini-Batch Gradient Descent**
  2. **Stochastic Gradient Descent (SGD)**
  
  In mini-batch gradient descent, you use a batch size that is considerably less than total no. of training examples and update your weights after passing through these examples.
  
  In stochastic gradient descent, you update the weights after passing through each training example.
  
  Coming to advantages and disadvantages of the three methods we discussed.
  
  * Batch gradient descent gradually converges to the global minimum but
    it is slow and requires huge computing power.
  * Stochastic gradient descent converges fast but not to the global
    minimum, it converges somewhere near to the global minimum and
    hovers around that point, but doesn't converge ever to the global
    minimum. But, the converged point in Stochastic gradient descent
    is good enough for all practical purposes.
  * Mini-Batch gradient is a trade-off the above two methods. But, if you
    have a vectorized implementation of the weights updation and you  
    are training with a multi-core setup or submitting the training to  
    multiple machines, this is the best method both in terms of time for
    training and convergence to global minimum.
  
  You can plot the cost function, w.r.t the no. of iterations to understand the difference between convergence in all the 3 types of gradient descent.
  
  * Batch gradient descent plot falls smoothly and slowly and gets stabilized and
    gets to global minimum.
  * Stochastic gradient descent plot will have oscillations, will fall
    fast but hovers around global minimum.
  
  These are some blogs where there is detailed explanation of advantages, disadvantages of each method and also graphs of how cost function changes for all the three methods with iterations.
  
  <https://adventuresinmachinelearning.com/stochastic-gradient-descent/>
  
  <https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/>
  
  
  ---
  
  