Answers.Body: <p>First, I have not read and do not have that book.</p>
  <p>That said, I would interpret that statement in the context of the intractability of guaranteeing that the optimization function will find global minima in the loss surface.</p>
  <p>In other words, higher precision values will do nothing to improve whether we have descended into a global or local minimum.  On the other hand, using a half precision value could result in a decrease in accuracy since the learning rate and weights may not have the precision to &quot;settle&quot; into the lowest point of a local minimum.  If you consider it in this context, a 64 bit value would not greatly improve the accuracy over a 32 bit value, while a 32 bit value would potentially have a noticeable improvement over 16 bit values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-19T11:16:51.277Z
Answers.LastActivityDate: 2022-01-19T23:47:59.157Z
Answers.LastEditDate: 2022-01-19T23:47:59.157Z
Answers.UserDisplayName: David Hoelzer
Body: <p>Consider the following paragraph from the subsubsection <a href="https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=51" rel="nofollow noreferrer">3.5.2: A dtype for every occasion</a> chapter named <strong>It starts with a tensor</strong> from the textbook titled <strong>Deep Learning with PyTorch</strong> by <em>Eli Stevens et al.</em></p>
  <blockquote>
  <p>As we will see in future chapters, computations happening in neural
  networks are typically executed with 32-bit floating-point precision.
  <strong>Higher precision, like 64-bit, will not buy improvements in the accuracy of a model and will require more memory and computing time.</strong>
  The 16-bit floating-point, half-precision data type is not present
  natively in standard CPUs, but it is offered on modern GPUs. <strong>It is
  possible to switch to half-precision to decrease the footprint of a
  neural network model if needed, with a minor impact on accuracy.</strong></p>
  </blockquote>
  <p>The paragraph is discussing the precision of weights of a neural network. Three types of floating-point numbers are discussed: 64-bit, 32-bit, and 16-bit (half precision). It has been said that it is recommended to use 32-bit numbers. If we decrease to half-precision then it will demote accuracy but if we increase to 64-bit then there will be no impact on accuracy.</p>
  <p>I am confused about how the improvement in precision from 32-bit to 64-bit does not improve the accuracy but the improvement from 16-bit to 32-bit can improve accuracy?</p>
  
Tags: <neural-networks><pytorch><weights><performance><precision>
Title: Why doesn't the high precision of neural network weights improve accuracy?
UserDisplayName: hanugm
ThreadText: # Why doesn't the high precision of neural network weights improve accuracy?
  
  (Asked by: username_0 on 2022-01-19)
  
  Consider the following paragraph from the subsubsection [3.5.2: A dtype for every occasion](https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=51) chapter named **It starts with a tensor** from the textbook titled **Deep Learning with PyTorch** by *Eli Stevens et al.*
  
  > As we will see in future chapters, computations happening in neural
  > networks are typically executed with 32-bit floating-point precision.
  > **Higher precision, like 64-bit, will not buy improvements in the accuracy of a model and will require more memory and computing time.**
  > The 16-bit floating-point, half-precision data type is not present
  > natively in standard CPUs, but it is offered on modern GPUs. **It is
  > possible to switch to half-precision to decrease the footprint of a
  > neural network model if needed, with a minor impact on accuracy.**
  
  The paragraph is discussing the precision of weights of a neural network. Three types of floating-point numbers are discussed: 64-bit, 32-bit, and 16-bit (half precision). It has been said that it is recommended to use 32-bit numbers. If we decrease to half-precision then it will demote accuracy but if we increase to 64-bit then there will be no impact on accuracy.
  
  I am confused about how the improvement in precision from 32-bit to 64-bit does not improve the accuracy but the improvement from 16-bit to 32-bit can improve accuracy?
  
  **Tags:** neural-networks, pytorch, weights, performance, precision
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-19. Score: 2
  
  First, I have not read and do not have that book.
  
  That said, I would interpret that statement in the context of the intractability of guaranteeing that the optimization function will find global minima in the loss surface.
  
  In other words, higher precision values will do nothing to improve whether we have descended into a global or local minimum. On the other hand, using a half precision value could result in a decrease in accuracy since the learning rate and weights may not have the precision to "settle" into the lowest point of a local minimum. If you consider it in this context, a 64 bit value would not greatly improve the accuracy over a 32 bit value, while a 32 bit value would potentially have a noticeable improvement over 16 bit values.
  
  
  ---
  
  