Answers.Body: <p>The choice of the batch size to be a power of 2 is not due the quality of predictions .</p>
  <p>The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for <code>batch_size=61</code> and <code>batch_size=64</code>, since the amount of stochasticity would be of the same order of magnitude.</p>
  <p>The actual reason, as explained <a href="https://datascience.stackexchange.com/questions/20179/what-is-the-advantage-of-keeping-batch-size-a-power-of-2">here</a>, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of <code>CUDA</code> streaming multiprocessors).</p>
  <p>If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.</p>
  <p>So the reason for batches to be powers of 2 is about efficient utilization of GPU.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T09:22:38.860Z
Answers.Text: When you write "However, there is not much different say in the behavior for training with", I would write "For the purposes of optimization with SGD, there might not be much different in the behavior for training with say .... However, this is in theory. In practice, GPUs might have memory whose size is a power of 2..." etc
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T08:46:10.590Z
Answers.LastActivityDate: 2021-12-27T19:54:55.493Z
Answers.LastEditDate: 2021-12-27T19:54:55.493Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>The main reason to use powers of 2 is in the way existing hardware and software are made, there isn't any purely mathematical reason. CPUs, GPUs, memories, and internal buses all use a size that's the power of 2 since that's the most efficient way to address them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T19:03:31.533Z
Answers.LastActivityDate: 2021-12-28T16:48:53.833Z
Answers.LastEditDate: 2021-12-28T16:48:53.833Z
Answers.UserDisplayName: cactusresistance
Body: <p>I am confused among the following in selecting the batch size for my model.</p>
  <p><strong>#1: powers of 2</strong></p>
  <p>I generally see that batch sizes are in powers of two: 32, 64, 128, 256.</p>
  <p><strong>#2: maximum GPU</strong></p>
  <p>Suppose my GPU allows a maximum batch size of 61. And it is not a power of two.</p>
  <p>Which one should I apt? Is there anything like the powers of 2 will give relatively good results?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T09:11:19.197Z
Comments.Text: It seems that this question is a duplicate of [this](https://ai.stackexchange.com/q/18381/2444), which, however, seems to be more than one question, so it might be closed as too broad. I would recommend that you search a little bit on the site before asking a question. Many questions have already been asked, but, of course, the answers may not be good and even the questions may be poor, and can be deleted. If you think they are poor, you can ask the question again and state that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T09:59:58.483Z
Comments.Text: @nbro Yeah, I missed the question because of tags assigned for that question. I think it is related but not duplicate. The intention of this question is to select which one: either #1 or #2, if can't both.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:02:39.950Z
Comments.Text: What do you mean by select #1 or #2? #1 is the batch size, #2 is the GPU, so what does that mean? I think they are exact duplicates. Here, it seems that you're asking why do we use powers of 2. If that's not the question, then the answer below doesn't answer your question. Your question in the title is "Is it true that batch size of form $2^k$ gives better results?", which seems to be the same as "why do we use powers of 2?"
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:04:17.467Z
Comments.Text: I mean, suppose my GPU allows at most 61 as batch-size. Then should I opt for 32 or 61 is the question. @nbro
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:06:51.423Z
Comments.Text: Here, you're actually asking multiple questions. One of your questions is: "Is there anything like the powers of 2 will give relatively good results?", which is a duplicate of that one, which the answer below answers. Your other question is "Which one should I apt?", which is unclear, but your second formulation in the comment above is better, so I would recommend that you edit your post to leae that formulation "Suppose my GPU allows at most 61 as batch size. Then should I opt for 32 or 61?"
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:06:55.367Z
Comments.Text: "Which one among the both should I consider for the batch size" is the first title I kept. But, for readability, I changed the title. @nbro
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:07:37.940Z
Comments.Text: Actually, I would recommend you ask that question in a separate post. I think that question "Suppose my GPU allows at most 61 as batch size. Then should I opt for 32 or 61?" is distinct enough to be asked in a separate post, taking into account the explanation in the answer below.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:08:46.833Z
Comments.Text: Okay @nbro.  I will do accordingly.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T10:11:08.160Z
Comments.Text: But maybe it's not worth it, if you ask the OP below to address that question too.
Comments.UserDisplayName: nbro
Tags: <deep-learning><hyper-parameters><gpu><batch-size>
Title: Is it true that batch size of form $2^k$ gives better results?
UserDisplayName: hanugm
ThreadText: # Is it true that batch size of form $2^k$ gives better results?
  
  (Asked by: username_0 on 2021-12-27)
  
  I am confused among the following in selecting the batch size for my model.
  
  **#1: powers of 2**
  
  I generally see that batch sizes are in powers of two: 32, 64, 128, 256.
  
  **#2: maximum GPU**
  
  Suppose my GPU allows a maximum batch size of 61. And it is not a power of two.
  
  Which one should I apt? Is there anything like the powers of 2 will give relatively good results?
  
  **Tags:** deep-learning, hyper-parameters, gpu, batch-size
  
  > **Comments:**
  > **username_1**: It seems that this question is a duplicate of [this](https://ai.stackexchange.com/q/18381/2444), which, however, seems to be more than one question, so it might be closed as too broad. I would recommend that you search a little bit on the site before asking a question. Many questions have already been asked, but, of course, the answers may not be good and even the questions may be poor, and can be deleted. If you think they are poor, you can ask the question again and state that.  
  > **username_0**: @username_1 Yeah, I missed the question because of tags assigned for that question. I think it is related but not duplicate. The intention of this question is to select which one: either #1 or #2, if can't both.  
  > **username_1**: What do you mean by select #1 or #2? #1 is the batch size, #2 is the GPU, so what does that mean? I think they are exact duplicates. Here, it seems that you're asking why do we use powers of 2. If that's not the question, then the answer below doesn't answer your question. Your question in the title is "Is it true that batch size of form $2^k$ gives better results?", which seems to be the same as "why do we use powers of 2?"  
  > **username_0**: I mean, suppose my GPU allows at most 61 as batch-size. Then should I opt for 32 or 61 is the question. @username_1  
  > **username_1**: Here, you're actually asking multiple questions. One of your questions is: "Is there anything like the powers of 2 will give relatively good results?", which is a duplicate of that one, which the answer below answers. Your other question is "Which one should I apt?", which is unclear, but your second formulation in the comment above is better, so I would recommend that you edit your post to leae that formulation "Suppose my GPU allows at most 61 as batch size. Then should I opt for 32 or 61?"  
  > **username_0**: "Which one among the both should I consider for the batch size" is the first title I kept. But, for readability, I changed the title. @username_1  
  > **username_1**: Actually, I would recommend you ask that question in a separate post. I think that question "Suppose my GPU allows at most 61 as batch size. Then should I opt for 32 or 61?" is distinct enough to be asked in a separate post, taking into account the explanation in the answer below.  
  > **username_0**: Okay @username_1.  I will do accordingly.  
  > **username_1**: But maybe it's not worth it, if you ask the OP below to address that question too.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-12-27. Score: 4
  
  The choice of the batch size to be a power of 2 is not due the quality of predictions .
  
  The larger the batch\_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for `batch_size=61` and `batch_size=64`, since the amount of stochasticity would be of the same order of magnitude.
  
  The actual reason, as explained [here](https://datascience.stackexchange.com/questions/20179/what-is-the-advantage-of-keeping-batch-size-a-power-of-2), is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of `CUDA` streaming multiprocessors).
  
  If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.
  
  So the reason for batches to be powers of 2 is about efficient utilization of GPU.
  
  > **Comments:**
  > **username_1**: When you write "However, there is not much different say in the behavior for training with", I would write "For the purposes of optimization with SGD, there might not be much different in the behavior for training with say .... However, this is in theory. In practice, GPUs might have memory whose size is a power of 2..." etc  
  
  ---
  
  ### Answer by username_3 on 2021-12-27. Score: 1
  
  The main reason to use powers of 2 is in the way existing hardware and software are made, there isn't any purely mathematical reason. CPUs, GPUs, memories, and internal buses all use a size that's the power of 2 since that's the most efficient way to address them.
  
  
  ---
  
  