Answers.Body: <p>No.  Different batch sizes mean different gradients (check stochastic gradient descent concept you will get how loss calculated) are calculated in each step, and thus the gradient descent will likely end up in different places in parameter space.</p>
  
  <p>In addition, how this is actually parallelized might make a difference, including the order of operations and converting between FP precision.</p>
  
  <p>Additional check resources:<a href="https://stackoverflow.com/questions/43845644/multi-gpu-architecture-gradient-averaging-less-accurate-model">issue of multi gpus</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T09:07:06.200Z
Answers.LastActivityDate: 2020-01-09T15:21:23.953Z
Answers.LastEditDate: 2020-01-09T15:21:23.953Z
Answers.UserDisplayName: Ketil Malde
Answers.Body: <p>This should make a difference, but how big is the difference heavily depends on your task. However, generally speaking, a smaller batch size will have a lower speed if counted in sample/minutes, but have a higher speed in batch/minutes. If the batch size is too small, the batch/minute will be very low and therefore decreasing training speed severely. However, a batch size too small (for example 1) will make the model hard to generalize and also slower to converge.</p>
  <p>This slide (<a href="https://cs230.stanford.edu/files/C2M2.pdf" rel="nofollow noreferrer">source</a>) is a great demonstration of how batch size affects training.</p>
  <p><a href="https://i.sstatic.net/cb2hu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cb2hu.png" alt="Batch size slide" /></a></p>
  <p>As you can see from the diagram, when you have a small batch size, the route to convergence will be ragged and not direct. This is because the model may train on an outlier and have its performance decrease before fitting again. Of course, this is an edge case and you would never train a model with 1 batch size.</p>
  <p>On the other hand, with a batch size too large, your model will take too long per iteration. With at least a decent batch size (like 16+) the number of iterations needed to train the model is similar, so a larger batch size is not going to help a lot. The performance is not going to vary a lot.</p>
  <p>In your case, the accuracy will make a difference but only minimally. Whilst writing this answer, I have run a few tests on batch size effect on performance and time, and here are the results. (Results to be added for 1 batch size)</p>
  <pre><code>Batch size 256 Time required 98.50849771499634s : 0.9414
  Batch size 128 Time required 108.53689193725586s : 0.9668
  Batch size 64 Time required 129.92272853851318s : 0.9776
  Batch size 32 Time required 162.13709354400635s : 0.9844
  Batch size 16 Time required 224.82269191741943s : 0.9854
  Batch size 8 Time required 351.2729814052582s : 0.9861
  Batch size 4 Time required 514.2667407989502s : 0.9862
  Batch size 2 Time required 829.1623721122742s : 0.9869
  </code></pre>
  <p>You can test out yourself in <a href="https://colab.research.google.com/drive/1xJEkyLd_02XVRYF6bmLgQ5MAXV2palCu" rel="nofollow noreferrer">this Google Colab</a>.</p>
  <p><a href="https://i.sstatic.net/dcD7R.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dcD7R.png" alt="enter image description here" /></a></p>
  <p>As you can see, the accuracy increases while the batch size decreases. This is because a higher batch size means it will be trained on fewer iterations. 2x batch size = half the iterations, so this is expected. The time required has risen exponentially, but the batch size of 32 or below doesn't seem to make a large difference in the time taken. The accuracy seems to be normal as half the iterations are trained with double the batch size.</p>
  <p>In your case, I would actually recommend you stick with 64 batch size even for 4 GPU. In the case of multiple GPUs, the rule of thumb will be using at least 16 (or so) batch size per GPU, given that, if you are using 4 or 8 batch size, the GPU cannot be completely utilized to train the model.</p>
  <p>For multiple GPU, there might be a slight difference due to precision error. Please, see <a href="https://stackoverflow.com/q/43845644/10675215">here</a>.</p>
  <h2>Conclusion</h2>
  <p>The batch size doesn't matter to performance too much, as long as you set a reasonable batch size (16+) and keep the <strong>iterations</strong> not epochs the same. However, training time will be affected. For multi-GPU, you should use the minimum batch size for each GPU that will utilize 100% of the GPU to train. 16 per GPU is quite good.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-10T00:31:14.850Z
Answers.Text: @ClementHui And each model saved from above three will have almost same inference accuracy, right?
Answers.UserDisplayName: bit_scientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-10T00:34:07.447Z
Answers.Text: They will have similar accuracy, give or take maybe 3%
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-10T00:34:54.147Z
Answers.Text: You don't need a batch size that large probably. @voo_doo
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T11:31:11.890Z
Answers.LastActivityDate: 2021-12-27T09:01:04.203Z
Answers.LastEditDate: 2021-12-27T09:01:04.203Z
Answers.UserDisplayName: Clement
Body: <p>I have a data set that was split using a fixed random seed and I am going to use 80% of the data for training and the rest for validation.</p>
  <p>Here are my GPU and batch size configurations</p>
  <ul>
  <li>use <code>64 batch size</code> with <code>one GTX 1080Ti</code></li>
  <li>use <code>128 batch size</code> with <code>two GTX 1080Ti</code></li>
  <li>use <code>256 batch size</code> with <code>four GTX 1080Ti</code></li>
  </ul>
  <p>All other hyper-parameters such as <code>lr</code>, <code>opt</code>, <code>loss</code>, etc., are fixed. Notice the linearity between the <code>batch size</code> and the <code>number of GPUs</code>.</p>
  <p>Will I get the same accuracy for those three experiments? Why and why not?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T15:39:57.440Z
Comments.Text: You might find the following somewhat useful to understand the effect of batch size: https://deeplearning.ai/ai-notes/optimization/?utm_campaign=BlogAINotesOptimizationNovember122019&utm_content=105388620&utm_medium=social&utm_source=linkedin&hss_channel=lcp-18246783
Comments.UserDisplayName: SpiderRico
Tags: <deep-learning><keras><accuracy><gpu><batch-size>
Title: Effect of batch size and number of GPUs on model accuracy
UserDisplayName: bit_scientist
ThreadText: # Effect of batch size and number of GPUs on model accuracy
  
  (Asked by: username_0 on 2020-01-09)
  
  I have a data set that was split using a fixed random seed and I am going to use 80% of the data for training and the rest for validation.
  
  Here are my GPU and batch size configurations
  
  * use `64 batch size` with `one GTX 1080Ti`
  * use `128 batch size` with `two GTX 1080Ti`
  * use `256 batch size` with `four GTX 1080Ti`
  
  All other hyper-parameters such as `lr`, `opt`, `loss`, etc., are fixed. Notice the linearity between the `batch size` and the `number of GPUs`.
  
  Will I get the same accuracy for those three experiments? Why and why not?
  
  **Tags:** deep-learning, keras, accuracy, gpu, batch-size
  
  > **Comments:**
  > **username_1**: You might find the following somewhat useful to understand the effect of batch size: https://deeplearning.ai/ai-notes/optimization/?utm_campaign=BlogAINotesOptimizationNovember122019&utm_content=105388620&utm_medium=social&utm_source=linkedin&hss_channel=lcp-18246783  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-01-09. Score: 11 | Accepted answer
  
  This should make a difference, but how big is the difference heavily depends on your task. However, generally speaking, a smaller batch size will have a lower speed if counted in sample/minutes, but have a higher speed in batch/minutes. If the batch size is too small, the batch/minute will be very low and therefore decreasing training speed severely. However, a batch size too small (for example 1) will make the model hard to generalize and also slower to converge.
  
  This slide ([source](https://cs230.stanford.edu/files/C2M2.pdf)) is a great demonstration of how batch size affects training.
  
  [![Batch size slide](https://i.sstatic.net/cb2hu.png)](https://i.sstatic.net/cb2hu.png)
  
  As you can see from the diagram, when you have a small batch size, the route to convergence will be ragged and not direct. This is because the model may train on an outlier and have its performance decrease before fitting again. Of course, this is an edge case and you would never train a model with 1 batch size.
  
  On the other hand, with a batch size too large, your model will take too long per iteration. With at least a decent batch size (like 16+) the number of iterations needed to train the model is similar, so a larger batch size is not going to help a lot. The performance is not going to vary a lot.
  
  In your case, the accuracy will make a difference but only minimally. Whilst writing this answer, I have run a few tests on batch size effect on performance and time, and here are the results. (Results to be added for 1 batch size)
  
  ```
  Batch size 256 Time required 98.50849771499634s : 0.9414
  Batch size 128 Time required 108.53689193725586s : 0.9668
  Batch size 64 Time required 129.92272853851318s : 0.9776
  Batch size 32 Time required 162.13709354400635s : 0.9844
  Batch size 16 Time required 224.82269191741943s : 0.9854
  Batch size 8 Time required 351.2729814052582s : 0.9861
  Batch size 4 Time required 514.2667407989502s : 0.9862
  Batch size 2 Time required 829.1623721122742s : 0.9869
  
  ```
  
  You can test out yourself in [this Google Colab](https://colab.research.google.com/drive/1xJEkyLd_02XVRYF6bmLgQ5MAXV2palCu).
  
  [![enter image description here](https://i.sstatic.net/dcD7R.png)](https://i.sstatic.net/dcD7R.png)
  
  As you can see, the accuracy increases while the batch size decreases. This is because a higher batch size means it will be trained on fewer iterations. 2x batch size = half the iterations, so this is expected. The time required has risen exponentially, but the batch size of 32 or below doesn't seem to make a large difference in the time taken. The accuracy seems to be normal as half the iterations are trained with double the batch size.
  
  In your case, I would actually recommend you stick with 64 batch size even for 4 GPU. In the case of multiple GPUs, the rule of thumb will be using at least 16 (or so) batch size per GPU, given that, if you are using 4 or 8 batch size, the GPU cannot be completely utilized to train the model.
  
  For multiple GPU, there might be a slight difference due to precision error. Please, see [here](https://stackoverflow.com/q/43845644/10675215).
  
  ## Conclusion
  
  The batch size doesn't matter to performance too much, as long as you set a reasonable batch size (16+) and keep the **iterations** not epochs the same. However, training time will be affected. For multi-GPU, you should use the minimum batch size for each GPU that will utilize 100% of the GPU to train. 16 per GPU is quite good.
  
  > **Comments:**
  > **username_0**: @ClementHui And each model saved from above three will have almost same inference accuracy, right?  
  > **username_3**: They will have similar accuracy, give or take maybe 3%  
  > **username_3**: You don't need a batch size that large probably. @voo_doo  
  
  ---
  
  ### Answer by username_2 on 2020-01-09. Score: 3
  
  No. Different batch sizes mean different gradients (check stochastic gradient descent concept you will get how loss calculated) are calculated in each step, and thus the gradient descent will likely end up in different places in parameter space.
  
  In addition, how this is actually parallelized might make a difference, including the order of operations and converting between FP precision.
  
  Additional check resources:[issue of multi gpus](https://stackoverflow.com/questions/43845644/multi-gpu-architecture-gradient-averaging-less-accurate-model)
  
  
  ---
  
  