Answers.Body: <p>Then work with quantized network it's good idea to normalize block of them them with  floating point scaling factor (16 bit float for example).
  The weight channel and input channel will have represenation S_f * w_n where S_f - scalar float and w_n - low-bit fixed or integer tensor or vector. Then calculating dot product put it into high-bit fixed/integer variable or float and recalculate scaling factor and low-bit representation for output. That is only if you can implement floating point (or high bit fixed) operations of cause. For low-bit representation you should clamp values to S_f*[-2^n:2^n] and that could be non-trivial. Good idea to use statistics of weights/inputs for it.
  There are dozen of papers on quantization of neural networks with different approaches, you may want to read some.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-04T06:56:59.633Z
Answers.LastActivityDate: 2019-03-04T07:02:15.520Z
Answers.LastEditDate: 2019-03-04T07:02:15.520Z
Answers.UserDisplayName: mirror2image
Body: <p>I am working on a project where the Neural Network weights must be quantized on 8 or 16 bits for an embedded platform, thus I will lose some precision.</p>
  
  <p>Since our platform does not have floating point arithmetic we need to quantize the weights. By quantizing i mean taking the max absolute value of the weights and divide it by the maximum signed number representable on 8 or 16 bits. this operation will give us a quantization factor <span class="math-container">$(qf)$</span>.
  the final quantized weights will be integer<span class="math-container">$(value * qf)$</span>.</p>
  
  <p>If my weights are very sparse and have a very bad distribution, I lose more precision.</p>
  
  <p>For example, to the left here is the distribution of weights for one layer, and to the right is the distribution of weights after I added to the loss function the Kurtosis and skew measures of the weights, and it improved a bit the shape of the distribution while keeping the same accuracy, even a bit higher.<a href="https://i.sstatic.net/XPQOG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XPQOG.png" alt="enter image description here"></a></p>
  
  <p>Does anybody have any other suggestions? Has anyone tackled this problem before?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-01T09:30:02.550Z
Comments.Text: Not sure if I understand your question correctly, but could you take the _log_ of the values to compress the range?
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-01T11:36:08.320Z
Comments.Text: I edited my description. Not sure if it would help me because I work with integer arithmetic. And another downside is that the log is an expensive operation that would have to be applied also to the input...I think
Comments.UserDisplayName: Florentin Alexandru Iftimie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T14:36:47.077Z
Comments.Text: @DuttaA 
  1. For a convolutional filter, we find the absolute max value of that tensor, name it Vmax, then if we want to have 8 bit precision, we divide 127 / Vmax, or if we want 16 bit precision we divide 32767 / Vmax both of these values will be named the quantization factor (qf). The resulting quantized tensor will have each value computed as v * qf. 
  2. The weights are initialized by Pytorch which is either He or Xavier (can't remember exactly)
  3. I am trying to achieve a minimum quality loss of the quantized version vs floating point version
Comments.UserDisplayName: Florentin Alexandru Iftimie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T15:05:56.220Z
Comments.Text: @DuttaA
  sorry for my bad explanation. I was referring to the newly quantized weight tensor of the convolutional filter. V is just a single weight in that tensor. I should have named it as w. Sometimes I look over badly written code and write the same bad naming in forums
Comments.UserDisplayName: Florentin Alexandru Iftimie
Tags: <neural-networks><hardware>
Title: Tips for keeping the distribution of weights normal
UserDisplayName: Florentin Alexandru Iftimie
ThreadText: # Tips for keeping the distribution of weights normal
  
  (Asked by: username_0 on 2019-02-01)
  
  I am working on a project where the Neural Network weights must be quantized on 8 or 16 bits for an embedded platform, thus I will lose some precision.
  
  Since our platform does not have floating point arithmetic we need to quantize the weights. By quantizing i mean taking the max absolute value of the weights and divide it by the maximum signed number representable on 8 or 16 bits. this operation will give us a quantization factor $(qf)$.
  the final quantized weights will be integer$(value \* qf)$.
  
  If my weights are very sparse and have a very bad distribution, I lose more precision.
  
  For example, to the left here is the distribution of weights for one layer, and to the right is the distribution of weights after I added to the loss function the Kurtosis and skew measures of the weights, and it improved a bit the shape of the distribution while keeping the same accuracy, even a bit higher.[![enter image description here](https://i.sstatic.net/XPQOG.png)](https://i.sstatic.net/XPQOG.png)
  
  Does anybody have any other suggestions? Has anyone tackled this problem before?
  
  **Tags:** neural-networks, hardware
  > **Comments:**
  > **username_1**: Not sure if I understand your question correctly, but could you take the _log_ of the values to compress the range?  
  > **username_0**: I edited my description. Not sure if it would help me because I work with integer arithmetic. And another downside is that the log is an expensive operation that would have to be applied also to the input...I think  
  > **username_0**: @DuttaA  1. For a convolutional filter, we find the absolute max value of that tensor, name it Vmax, then if we want to have 8 bit precision, we divide 127 / Vmax, or if we want 16 bit precision we divide 32767 / Vmax both of these values will be named the quantization factor (qf). The resulting quantized tensor will have each value computed as v * qf.  2. The weights are initialized by Pytorch which is either He or Xavier (can't remember exactly) 3. I am trying to achieve a minimum quality loss of the quantized version vs floating point version  
  > **username_0**: @DuttaA sorry for my bad explanation. I was referring to the newly quantized weight tensor of the convolutional filter. V is just a single weight in that tensor. I should have named it as w. Sometimes I look over badly written code and write the same bad naming in forums  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-03-04. Score: 1
  
  Then work with quantized network it's good idea to normalize block of them them with floating point scaling factor (16 bit float for example).
  The weight channel and input channel will have represenation S\_f \* w\_n where S\_f - scalar float and w\_n - low-bit fixed or integer tensor or vector. Then calculating dot product put it into high-bit fixed/integer variable or float and recalculate scaling factor and low-bit representation for output. That is only if you can implement floating point (or high bit fixed) operations of cause. For low-bit representation you should clamp values to S\_f\*[-2^n:2^n] and that could be non-trivial. Good idea to use statistics of weights/inputs for it.
  There are dozen of papers on quantization of neural networks with different approaches, you may want to read some.
  
  
  ---
  
  