Answers.Body: <p>I copied raw numbers from the graph and did some plotting.</p>
  <pre><code>   2048     83
     1024     84
      512     86
      256     90
      128    104
       64    135
       32    204
       16    360
        8    656
  </code></pre>
  <p>I don't know the actual number of training samples, so I used a place-holder value of 10k here. As we can see, the hardware is under-utilized when the batch size is too small. This is caused by a constant(?) overhead related to calculating losses and gradients in regards to each batch. Ah, I forgot to divide training time by the number of epochs, but only relative numbers matter here.</p>
  <p><a href="https://i.sstatic.net/RnD50.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RnD50.png" alt="batch_size vs. performance" /></a></p>
  <p>In an other point of view, the data fits well on a function <span class="math-container">$y = a/x + b$</span> where <span class="math-container">$x$</span> is the batch size and <span class="math-container">$y$</span> is the training time. In the limit, training time approaches the theoretical minimum when the hardware is best utilized, but each separate batch causes a fixed overhead on setting things up.</p>
  <p><a href="https://i.sstatic.net/0L9CW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0L9CW.png" alt="batch_size vs. time" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-07T16:54:33.563Z
Answers.Text: I did not include this in the question but the reason why I had this question was because I wanted perform regression on some data I obtained from a network I was training. I hypothesize that the reason behind the data having an inverse relationship is the fact that the number of backwards passes is inversely proportional to batch size. And the vertical shift is due to the (near) constant training time associated with forwards propagation if differences in GPU utilization across batch sizes are ignored. I am not entirely sure whether my reasoning is correct, but thanks for your answer.
Answers.UserDisplayName: Alpy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-06T18:24:57.197Z
Answers.LastActivityDate: 2023-02-06T18:24:57.197Z
Answers.UserDisplayName: NikoNyrh
Answers.Body: <p>This has a very simple hardware explanation.</p>
  <p>GPUs have several thousand cores. If you are not making use of enough of these cores, eventually the cost of moving data over to the gpu comes to dominate the parallelism. Every time you reduce batch, you reduce parallelism and increase data transfers.</p>
  <p>For example, if you halve batch size, you have increased the amount of data transfers by 2, and decreased the parallelism by 2. Now, this can be expected to be more than twice as slow, since both data transfer and computation have doubled.</p>
  <p>However, I do not think this relationship is actually exponential. The article you linked does not seem to support that conclusion either. I know the article uses the world exponential, but if you look at the chart this is simply not the case. The chart in that article is presented on a logarithmic scale, so when you actually look at the numbers on a linear scale, the relationship is close, but not quite, linear, for reasons I outlined.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-06T17:40:56.833Z
Answers.LastActivityDate: 2023-02-06T17:40:56.833Z
Answers.UserDisplayName: chessprogrammer
Body: <p>I'm quite new to machine learning and wanted to ask a question regarding why reducing batch sizes cause exponentially increasing training times. An example of this relationship between batch size and training time can be seen in <a href="https://wandb.ai/ayush-thakur/dl-question-bank/reports/What-s-the-Optimal-Batch-Size-to-Train-a-Neural-Network---VmlldzoyMDkyNDU" rel="nofollow noreferrer">this article</a>. I had a hard time finding any answers to this question with my googling skills and any possible hypotheses that I have as to why are most likely incorrect. I'd greatly appreciate if somebody could explain the reasoning behind this relationship.</p>
  
Tags: <neural-networks><training-datasets><batch-size>
Title: Why does linearly decreasing batch sizes result in exponentially increasing training times?
UserDisplayName: Alpy
ThreadText: # Why does linearly decreasing batch sizes result in exponentially increasing training times?
  
  (Asked by: username_0 on 2023-02-06)
  
  I'm quite new to machine learning and wanted to ask a question regarding why reducing batch sizes cause exponentially increasing training times. An example of this relationship between batch size and training time can be seen in [this article](https://wandb.ai/ayush-thakur/dl-question-bank/reports/What-s-the-Optimal-Batch-Size-to-Train-a-Neural-Network---VmlldzoyMDkyNDU). I had a hard time finding any answers to this question with my googling skills and any possible hypotheses that I have as to why are most likely incorrect. I'd greatly appreciate if somebody could explain the reasoning behind this relationship.
  
  **Tags:** neural-networks, training-datasets, batch-size
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-02-06. Score: 0 | Accepted answer
  
  I copied raw numbers from the graph and did some plotting.
  
  ```
     2048     83
     1024     84
      512     86
      256     90
      128    104
       64    135
       32    204
       16    360
        8    656
  
  ```
  
  I don't know the actual number of training samples, so I used a place-holder value of 10k here. As we can see, the hardware is under-utilized when the batch size is too small. This is caused by a constant(?) overhead related to calculating losses and gradients in regards to each batch. Ah, I forgot to divide training time by the number of epochs, but only relative numbers matter here.
  
  [![batch_size vs. performance](https://i.sstatic.net/RnD50.png)](https://i.sstatic.net/RnD50.png)
  
  In an other point of view, the data fits well on a function $y = a/x + b$ where $x$ is the batch size and $y$ is the training time. In the limit, training time approaches the theoretical minimum when the hardware is best utilized, but each separate batch causes a fixed overhead on setting things up.
  
  [![batch_size vs. time](https://i.sstatic.net/0L9CW.png)](https://i.sstatic.net/0L9CW.png)
  
  > **Comments:**
  > **username_0**: I did not include this in the question but the reason why I had this question was because I wanted perform regression on some data I obtained from a network I was training. I hypothesize that the reason behind the data having an inverse relationship is the fact that the number of backwards passes is inversely proportional to batch size. And the vertical shift is due to the (near) constant training time associated with forwards propagation if differences in GPU utilization across batch sizes are ignored. I am not entirely sure whether my reasoning is correct, but thanks for your answer.  
  
  ---
  
  ### Answer by username_2 on 2023-02-06. Score: 2
  
  This has a very simple hardware explanation.
  
  GPUs have several thousand cores. If you are not making use of enough of these cores, eventually the cost of moving data over to the gpu comes to dominate the parallelism. Every time you reduce batch, you reduce parallelism and increase data transfers.
  
  For example, if you halve batch size, you have increased the amount of data transfers by 2, and decreased the parallelism by 2. Now, this can be expected to be more than twice as slow, since both data transfer and computation have doubled.
  
  However, I do not think this relationship is actually exponential. The article you linked does not seem to support that conclusion either. I know the article uses the world exponential, but if you look at the chart this is simply not the case. The chart in that article is presented on a logarithmic scale, so when you actually look at the numbers on a linear scale, the relationship is close, but not quite, linear, for reasons I outlined.
  
  
  ---
  
  