Answers.Body: <p>There is a hardware based reasoning. Matrix multiplication is one of the central computations in deep learning. SIMD operations in CPUs happen in batch sizes, which are powers of 2.</p>
  <p>Here is a good reference about speeding up neural networks on CPUs by leveraging SIMD instructions:</p>
  <p><a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf" rel="nofollow noreferrer">Improving the speed of neural networks on CPUs</a></p>
  <p>You will notice batch sizes that are powers of 2. This is a good paper to read about implementing neural networks using SIMD instructions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-20T23:45:45.657Z
Answers.Text: Do all sizes for the batches really need to be a power of 2 in the case of SIMD operations? It would be nice if you can provide a link to a reliable reference that supports this claim or just for us to read more about the topic.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-01T11:19:24.927Z
Answers.LastActivityDate: 2021-05-07T15:56:04.170Z
Answers.LastEditDate: 2021-05-07T15:56:04.170Z
Answers.UserDisplayName: Hari
Answers.Body: <blockquote>
  <p>I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.</p>
  </blockquote>
  <p>I would quite like to see a reference to this suggestion, in case it has been misunderstood.</p>
  <p>As far as I know, there is no such effect in normal neural networks. In convolutional neural networks, it might potentially be true in a minor way because some FFT approaches work better with <span class="math-container">$2^n$</span> items.</p>
  <blockquote>
  <p>Is this a fact? If it is, why is this true? Does it have something to do with how the memory is laid down?</p>
  </blockquote>
  <p>I would say that this is not a general fact. Instead, it seems like misunderstood advice to <em>search</em> some hyperparameters such as number of neurons in each layer, by increasing or decreasing by a factor of 2. Doing this and trying layer sizes of 32, 64, 128 etc should increase the speed of finding a good layer size compared to trying sizes 32, 33, 34 etc.</p>
  <p>The main reason to pick powers of 2 is tradition in computer science. Provided there is no driver to pick other specific numbers, may as well pick a power of 2 . . . but equally you will see researchers picking multiples of 10, 100 or 1000 as &quot;round numbers&quot;, for a similar reason.</p>
  <p>One related factor: If a researcher presents a result for some new technique where the hidden layer sizes were tuned to e.g. 531, 779, 282 etc, then someone reviewing the work would ask the obvious question &quot;Why?&quot; - such numbers might imply the new technique is not generic or requires large amounts of hyperparameter tuning, neither of which would be seen as positive traits. Much better to be seen using an obvious &quot;simple&quot; number . . .</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-02-22T18:09:34.890Z
Answers.LastActivityDate: 2021-01-20T23:42:58.787Z
Answers.LastEditDate: 2021-01-20T23:42:58.787Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.</p>
  <p>Is this a fact? If it is, why is this true? Does it have something to do with how the memory is laid down?</p>
  
Tags: <deep-learning><optimization><hyperparameter-optimization><hyper-parameters><hidden-layers>
Title: Why should the number of neurons in a hidden layer be a power of 2?
UserDisplayName: dsfx3d
ThreadText: # Why should the number of neurons in a hidden layer be a power of 2?
  
  (Asked by: username_0 on 2018-02-22)
  
  I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
  
  Is this a fact? If it is, why is this true? Does it have something to do with how the memory is laid down?
  
  **Tags:** deep-learning, optimization, hyperparameter-optimization, hyper-parameters, hidden-layers
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2018-02-22. Score: 14 | Accepted answer
  
  > I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
  
  I would quite like to see a reference to this suggestion, in case it has been misunderstood.
  
  As far as I know, there is no such effect in normal neural networks. In convolutional neural networks, it might potentially be true in a minor way because some FFT approaches work better with $2^n$ items.
  
  > Is this a fact? If it is, why is this true? Does it have something to do with how the memory is laid down?
  
  I would say that this is not a general fact. Instead, it seems like misunderstood advice to *search* some hyperparameters such as number of neurons in each layer, by increasing or decreasing by a factor of 2. Doing this and trying layer sizes of 32, 64, 128 etc should increase the speed of finding a good layer size compared to trying sizes 32, 33, 34 etc.
  
  The main reason to pick powers of 2 is tradition in computer science. Provided there is no driver to pick other specific numbers, may as well pick a power of 2 . . . but equally you will see researchers picking multiples of 10, 100 or 1000 as "round numbers", for a similar reason.
  
  One related factor: If a researcher presents a result for some new technique where the hidden layer sizes were tuned to e.g. 531, 779, 282 etc, then someone reviewing the work would ask the obvious question "Why?" - such numbers might imply the new technique is not generic or requires large amounts of hyperparameter tuning, neither of which would be seen as positive traits. Much better to be seen using an obvious "simple" number . . .
  
  
  ---
  
  ### Answer by username_1 on 2020-09-01. Score: 1
  
  There is a hardware based reasoning. Matrix multiplication is one of the central computations in deep learning. SIMD operations in CPUs happen in batch sizes, which are powers of 2.
  
  Here is a good reference about speeding up neural networks on CPUs by leveraging SIMD instructions:
  
  [Improving the speed of neural networks on CPUs](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf)
  
  You will notice batch sizes that are powers of 2. This is a good paper to read about implementing neural networks using SIMD instructions.
  
  > **Comments:**
  > **username_2**: Do all sizes for the batches really need to be a power of 2 in the case of SIMD operations? It would be nice if you can provide a link to a reliable reference that supports this claim or just for us to read more about the topic.  
  
  ---
  
  