Answers.Body: <p>Regarding post-training quantization, how well you quantize your model depends on the range in which you model weights are.</p>
  <p>if you have the following weights:<br />
  <span class="math-container">$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53]$</span>,<br />
  you quantize and de-quantize them you get:<br />
  <span class="math-container">$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.28, -2.11, 0.33, -0.53]$</span>.<br />
  Two minor errors on the 0.01 precision level.</p>
  <p>However if you have the following weights with an outlier:<br />
  <span class="math-container">$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]$</span>,<br />
  and you again quantize and de-quantize, then you get:<br />
  <span class="math-container">$[-0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.0]$</span>.</p>
  <p>Currently, one of the famous quantization algorithms for large language models is <a href="https://arxiv.org/pdf/2208.07339.pdf" rel="nofollow noreferrer">LLM.int8()</a>. This algorithm exploits a very interesting phenomenon. It turns out that such kind of outliers are very systematic with language models with &gt; 6.7B parameters, so they can be separated from the rest of the parameters. They quantize the regular values (which are around 99.9% of the model weights) and the outliers are not quantized. You can read more <a href="https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/#Mixed_precision_decomposition" rel="nofollow noreferrer">here</a>, this is the blog of the lead author of the LLM.int8() paper.</p>
  <p>Now, if you want to quantize your own model you have to check if you have such outliers, and combine and quantize the weights accordingly. What is so important about the LLM.int8() paper is that they provide an algorithm for <strong>any</strong> large language model. They claim that the irregularities that they found are present in every large language model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-05T09:21:50.873Z
Answers.Text: So you're saying that although some outliers may spoil quantization, there're indeed solutions; you mention PTQ may fix this, but for sure weight calibration alone will avoid that altogether. So I don't see how this answer my questions.
Answers.UserDisplayName: edmz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T09:57:33.807Z
Answers.LastActivityDate: 2023-08-02T11:58:14.083Z
Answers.LastEditDate: 2023-08-02T11:58:14.083Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>The rationale is that simpler tasks involve less complex decision boundaries, and sparse networks have more zero-valued weights that can be efficiently represented in lower precision. Larger networks with more parameters may face challenges during quantization, as there is more information to compress into fewer bits.</p>
  <p>Ultimately, the quantization suitability depends on a combination of factors, and experimenting with different architectures</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-05T09:16:58.627Z
Answers.Text: For sure shallower nn will have more ease 'compressing', however also large DNNs (VGG et al) and ResNet do quantize down quite nicely. There're quite a number of strategies you might take to do so, yet AFAIK even this for some NN won't apply. So what you say does not scale, and also does not ansswer my questions.
Answers.UserDisplayName: edmz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T08:36:41.757Z
Answers.LastActivityDate: 2023-08-02T08:36:41.757Z
Answers.UserDisplayName: Anay
Body: <p>I know this might be a bit general question and concerning a rather active research field, much beyond my expertise, but I do believe there're some answers.</p>
  <p>The use of NN parameters quantization can span from post-training static/dynamic quantization (PTQ) to quantization-aware training (QAT). Generally the target is cutting down FP-32 weights to UINT-8 whilst retaining overall accuracy; the benefits in performance are often sensible, yielding zero to negligible (depending on application) to few percentage drop in accuracy.</p>
  <p>However, I hear these statements are true depending on <em>which</em> NN is being quantized (<a href="https://youtu.be/c3MT2qV5f9w?t=606" rel="nofollow noreferrer">source</a>). Some models indeed are very forgiving, whereas others are not even using more aggressive strategies.</p>
  <blockquote>
  <p><em>(extracted from PyTorch source)</em></p>
  </blockquote>
  <blockquote>
  <p>There're some networks that are very forgiving: you can do PTQ and
  the end result is as accurate as the FP value. [...] In <strong>some</strong> NN
  they're slightly more demanding, they're slightly less forgiving on
  the approximation [...]</p>
  </blockquote>
  <p>My question are:</p>
  <ul>
  <li>Which categories of DNN are more suited for quantization?</li>
  <li>What could be one rationale why this would be? Does the size of the network (i.e. number of parameters) has any role, for instance?</li>
  <li>What is introducing such demands (this will depend on actual model indeed) in the forward path? Is it to be found in the activation/actual MVP or what?</li>
  </ul>
  <p>My <strong>guess</strong> would be depending on achievable weights sparsity -- however, this in general (AFAIK) can be tuned forcing training constraints, so it wouldn't really answer the question.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T18:15:05.377Z
Comments.Text: IMHO, depends more on the task than on the architecture, even though I can imagine that architectures that tend to overfit might have a hard time being quantized
Comments.UserDisplayName: Alberto
Tags: <neural-networks><deep-learning>
Title: Why are some Neural Networks more forgiving on Quantization?
UserDisplayName: edmz
ThreadText: # Why are some Neural Networks more forgiving on Quantization?
  
  (Asked by: username_0 on 2023-07-29)
  
  I know this might be a bit general question and concerning a rather active research field, much beyond my expertise, but I do believe there're some answers.
  
  The use of NN parameters quantization can span from post-training static/dynamic quantization (PTQ) to quantization-aware training (QAT). Generally the target is cutting down FP-32 weights to UINT-8 whilst retaining overall accuracy; the benefits in performance are often sensible, yielding zero to negligible (depending on application) to few percentage drop in accuracy.
  
  However, I hear these statements are true depending on *which* NN is being quantized ([source](https://youtu.be/c3MT2qV5f9w?t=606)). Some models indeed are very forgiving, whereas others are not even using more aggressive strategies.
  
  > *(extracted from PyTorch source)*
  
  > There're some networks that are very forgiving: you can do PTQ and
  > the end result is as accurate as the FP value. [...] In **some** NN
  > they're slightly more demanding, they're slightly less forgiving on
  > the approximation [...]
  
  My question are:
  
  * Which categories of DNN are more suited for quantization?
  * What could be one rationale why this would be? Does the size of the network (i.e. number of parameters) has any role, for instance?
  * What is introducing such demands (this will depend on actual model indeed) in the forward path? Is it to be found in the activation/actual MVP or what?
  
  My **guess** would be depending on achievable weights sparsity -- however, this in general (AFAIK) can be tuned forcing training constraints, so it wouldn't really answer the question.
  
  **Tags:** neural-networks, deep-learning
  
  > **Comments:**
  > **username_1**: IMHO, depends more on the task than on the architecture, even though I can imagine that architectures that tend to overfit might have a hard time being quantized  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-08-02. Score: 1
  
  Regarding post-training quantization, how well you quantize your model depends on the range in which you model weights are.
  
  if you have the following weights:  
  $[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53]$,  
  you quantize and de-quantize them you get:  
  $[-0.10, -0.23, 0.08, -0.38, -0.28, -0.28, -2.11, 0.33, -0.53]$.  
  Two minor errors on the 0.01 precision level.
  
  However if you have the following weights with an outlier:  
  $[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]$,  
  and you again quantize and de-quantize, then you get:  
  $[-0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.0]$.
  
  Currently, one of the famous quantization algorithms for large language models is [LLM.int8()](https://arxiv.org/pdf/2208.07339.pdf). This algorithm exploits a very interesting phenomenon. It turns out that such kind of outliers are very systematic with language models with > 6.7B parameters, so they can be separated from the rest of the parameters. They quantize the regular values (which are around 99.9% of the model weights) and the outliers are not quantized. You can read more [here](https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/#Mixed_precision_decomposition), this is the blog of the lead author of the LLM.int8() paper.
  
  Now, if you want to quantize your own model you have to check if you have such outliers, and combine and quantize the weights accordingly. What is so important about the LLM.int8() paper is that they provide an algorithm for **any** large language model. They claim that the irregularities that they found are present in every large language model.
  
  > **Comments:**
  > **username_0**: So you're saying that although some outliers may spoil quantization, there're indeed solutions; you mention PTQ may fix this, but for sure weight calibration alone will avoid that altogether. So I don't see how this answer my questions.  
  
  ---
  
  ### Answer by username_3 on 2023-08-02. Score: 0
  
  The rationale is that simpler tasks involve less complex decision boundaries, and sparse networks have more zero-valued weights that can be efficiently represented in lower precision. Larger networks with more parameters may face challenges during quantization, as there is more information to compress into fewer bits.
  
  Ultimately, the quantization suitability depends on a combination of factors, and experimenting with different architectures
  
  > **Comments:**
  > **username_0**: For sure shallower nn will have more ease 'compressing', however also large DNNs (VGG et al) and ResNet do quantize down quite nicely. There're quite a number of strategies you might take to do so, yet AFAIK even this for some NN won't apply. So what you say does not scale, and also does not ansswer my questions.  
  
  ---
  
  