Answers.Body: <p>Their <a href="https://github.com/tensorflow/tflite-micro/blob/main/tensorflow/lite/kernels/internal/common.h#L569C14-L569C49" rel="nofollow noreferrer">lookup-table</a> is declared as <code>uint16_t sigmoid_table_uint16[256]</code>. At the bottom is a plot.</p>
  <h2>Input Scaling</h2>
  <p>Their <code>Tanh()</code> function gets 16bit inputs. Those are presumably the accumulated (and biased) results of several 8bit multiplications. So at this point there is no zero-point to consider any more, only a scaling factor.</p>
  <p>(My understanding is that zero-point this is only relevant during the 8bit matrix-multiplication, but I'm not 100% sure to be honest. The results of the 8bit multiplications are of course 16bit, and you don't go back to 8bit until you have to.)</p>
  <p>According to the code, the scaling is <span class="math-container">$\frac{\text{input_multiplier}}{2^{\text{input_left_shift}}}$</span>. But the point is really to scale the input such that <span class="math-container">$2^{8+8}$</span> corresponds to a floating-point input of 10.7 (the chosen end of the table). We are in 32bit integer-land by now, and the input can be larger than <span class="math-container">$2^{8+8}$</span>.</p>
  <h2>Look-Up</h2>
  <p>The number <span class="math-container">$2^{8+8}$</span> contains two engineering choices. First, the size of the look-up table. They have chosen 256 entries, so the top 8 bits will be the index. Second, the precision for linear interpolation (fraction part of the index). They have chosen 8 bits. An alternative would be zero, simply returning the nearest value from the table.</p>
  <p>Now they divide by <span class="math-container">$2^8$</span> (the expression <code>abs_input_data &gt;&gt; 8</code>) to get the index, and use the remainder of the division (<code>abs_input_data &amp; 0xFF</code>) for linear interpolation between the actual and the next value from the look-up table.</p>
  <h2>Interpolation</h2>
  <pre><code>result = (ua &lt;&lt; 8) + ut * (ub - ua);
  </code></pre>
  <p>Here they interpolate between <code>ua</code> and <code>ub</code>, which are adjacent 16bit values from the table, using <code>ut</code> (the 8-bit remainder).</p>
  <p>To get a float <code>sigmoid</code> result (between 0.5 and 1.0) you'd have to divide by <span class="math-container">$2^{16+8}$</span>.</p>
  <h2>Scaling back</h2>
  <p>If you check the plot below, the table has the right half of a 16bit unsigned <code>sigmoid</code>. But they want <code>tanh</code>. So they add back the sign and subtract the zero-point. The zero-point is at <span class="math-container">$2^{16+8-1}$</span> and written as <code>1&lt;&lt;(14 + 9)</code> in their the code.</p>
  <p>We still have to divide by <span class="math-container">$2^8$</span> from the interpolation, and shift one bit right to make room for the sign that we added, but we also have to divide by two to go from <code>sigmoid</code> to <code>tanh</code>. Those cancel out, and we divide by <span class="math-container">$2^8$</span> with rounding.</p>
  <p>Rounding is done by adding 128 (or <code>1 &lt;&lt; (9 - 2)</code> in the code). But you also have to be concerned about the asymmetric range of signed integers, and whether C is rounding towards zero or towards negative infinity. Which is probably why they saturate at <code>0xFFFF00</code> instead of <code>0xFFFFFF</code> but I'm not certain. With that, we're knee-deep in the engineering swamp and I've probably lost all the machine learning people, so I'll leave it at that.</p>
  <p>TL;DR: Fixed-point math is not too hard to grasp, just impossible to read and validate. That's the price you pay to avoid costly conversions to/from floats.</p>
  <hr />
  <p><a href="https://i.sstatic.net/hIWWY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hIWWY.png" alt="tanh look-up table" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-01T16:04:59.023Z
Answers.Text: Thank you very much! Why is net necessary to scale the input so that 2^(8+8) corresponds to the end of the LUT? Because that expression always matches the maximum value. Also, do you understand what the shifting of the result does? I understand it's fixed point, but I don't see where the exact values are derived from.
Answers.UserDisplayName: Necrotos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-01T20:57:39.340Z
Answers.Text: I've made a major edit which should answer both questions, I think.
Answers.UserDisplayName: maxy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-04T16:31:55.567Z
Answers.Text: Thanks! There are a few minor things I wonder after reading the update. Do you see any indication where the scaling factor for the input would come from? Also, you mention that they subtract the zero-point, but what zero point does this refer to? Finally, why do they add 128 to round? That number seems very random. Is it just an engineering choice, or is there some deeper meaning?
Answers.UserDisplayName: Necrotos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-04T18:17:04.783Z
Answers.Text: The first value in the table (see plot) is used for input 0. Because `tanh(0)=0` this is the zero-point. They multiply it by 256 with `(ua << 8)` and later divide by 256 with an arithmetic right shift (line 108, `result >>= 8`). This division truncates the result (rounds towards -oo). If you add 128 first, you get normal rounding.
Answers.UserDisplayName: maxy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-04T18:22:01.237Z
Answers.Text: For the scaling factor, you'd have to read the rest of their code. I've only read the part that interested me personally, which was the LUT.
Answers.UserDisplayName: maxy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T13:13:58.407Z
Answers.Text: Thank you so much! This clears up a lot of my confusion. For the actual inputs: Are those just regular int values that are interpreted as fixed point here for the sake of the interpolation? Also, can you recommend any resources on fixed point operations? I feel like I am lacking quite a bit here.
Answers.UserDisplayName: Necrotos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-01T14:37:53.373Z
Answers.LastActivityDate: 2023-07-01T21:01:38.517Z
Answers.LastEditDate: 2023-07-01T21:01:38.517Z
Answers.UserDisplayName: maxy
Answers.Body: <p>Please refer to following document link, we convert any Q15 integer to Q4.12 fixed point format with help of input scaling with factor of 3/4. Q4.12 {3 bits - Integer, 12 bits of fraction}. Then ua uses upper 8 bits {Q4(INT).4(precision)} while ut uses lower fixed point precision.</p>
  <p>Link - <a href="https://github.com/tensorflow/tflite-micro/pull/1253" rel="nofollow noreferrer">https://github.com/tensorflow/tflite-micro/pull/1253</a></p>
  <p>Regards,
  Chaitanya</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-27T19:55:00.943Z
Answers.Text: Thanks! But I don't understand what scaling with the factor 3/4 does, or where in the code it is applied. Could you expand on that?
Answers.UserDisplayName: Necrotos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-23T03:59:22.610Z
Answers.LastActivityDate: 2023-08-23T05:50:59.220Z
Answers.LastEditDate: 2023-08-23T05:50:59.220Z
Answers.UserDisplayName: Chaitanya Naik
Body: <p>I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach.</p>
  <p>My problem is that the relation of data width and size of the LUT, scaling factor and zero point will affect which index of the LUT I have to use for the calculation. E.g. TFLite-Micro has an <a href="https://github.com/tensorflow/tflite-micro/blob/d2c4ef56f7a67db2da97f42de1f67e404f13eaad/tensorflow/lite/kernels/internal/reference/integer_ops/tanh.h#L63" rel="nofollow noreferrer">implementation of such a LUT for tanh and sigmoid</a>. They have an implementation, but I do not understand how they calculate the indices. Is there some guidance on how to design such a function available somewhere? I haven't managed to find anything more specific.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-30T01:00:48.240Z
Comments.Text: Can you please reformulate the title as your _specific question_?
Comments.UserDisplayName: nbro
Tags: <deep-learning><long-short-term-memory><activation-functions><hardware>
Title: LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks
UserDisplayName: Necrotos
ThreadText: # LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks
  
  (Asked by: username_0 on 2023-06-27)
  
  I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach.
  
  My problem is that the relation of data width and size of the LUT, scaling factor and zero point will affect which index of the LUT I have to use for the calculation. E.g. TFLite-Micro has an [implementation of such a LUT for tanh and sigmoid](https://github.com/tensorflow/tflite-micro/blob/d2c4ef56f7a67db2da97f42de1f67e404f13eaad/tensorflow/lite/kernels/internal/reference/integer_ops/tanh.h#L63). They have an implementation, but I do not understand how they calculate the indices. Is there some guidance on how to design such a function available somewhere? I haven't managed to find anything more specific.
  
  **Tags:** deep-learning, long-short-term-memory, activation-functions, hardware
  > **Comments:**
  > **username_1**: Can you please reformulate the title as your _specific question_?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-01. Score: 0 | Accepted answer
  
  Their lookup-table is declared as uint16_t sigmoid_table_uint16[256]. At the bottom is a plot.
  Input Scaling
  Their Tanh() function gets 16bit inputs. Those are presumably the accumulated (and biased) results of several 8bit multiplications. So at this point there is no zero-point to consider any more, only a scaling factor.
  (My understanding is that zero-point this is only relevant during the 8bit matrix-multiplication, but I'm not 100% sure to be honest. The results of the 8bit multiplications are of course 16bit, and you don't go back to 8bit until you have to.)
  According to the code, the scaling is $\frac{\text{input_multiplier}}{2^{\text{input_left_shift}}}$. But the point is really to scale the input such that $2^{8+8}$ corresponds to a floating-point input of 10.7 (the chosen end of the table). We are in 32bit integer-land by now, and the input can be larger than $2^{8+8}$.
  Look-Up
  The number $2^{8+8}$ contains two engineering choices. First, the size of the look-up table. They have chosen 256 entries, so the top 8 bits will be the index. Second, the precision for linear interpolation (fraction part of the index). They have chosen 8 bits. An alternative would be zero, simply returning the nearest value from the table.
  Now they divide by $2^8$ (the expression abs_input_data >> 8) to get the index, and use the remainder of the division (abs_input_data & 0xFF) for linear interpolation between the actual and the next value from the look-up table.
  Interpolation
  result = (ua << 8) + ut * (ub - ua);
  
  Here they interpolate between ua and ub, which are adjacent 16bit values from the table, using ut (the 8-bit remainder).
  To get a float sigmoid result (between 0.5 and 1.0) you'd have to divide by $2^{16+8}$.
  Scaling back
  If you check the plot below, the table has the right half of a 16bit unsigned sigmoid. But they want tanh. So they add back the sign and subtract the zero-point. The zero-point is at $2^{16+8-1}$ and written as 1<<(14 + 9) in their the code.
  We still have to divide by $2^8$ from the interpolation, and shift one bit right to make room for the sign that we added, but we also have to divide by two to go from sigmoid to tanh. Those cancel out, and we divide by $2^8$ with rounding.
  Rounding is done by adding 128 (or 1 << (9 - 2) in the code). But you also have to be concerned about the asymmetric range of signed integers, and whether C is rounding towards zero or towards negative infinity. Which is probably why they saturate at 0xFFFF00 instead of 0xFFFFFF but I'm not certain. With that, we're knee-deep in the engineering swamp and I've probably lost all the machine learning people, so I'll leave it at that.
  TL;DR: Fixed-point math is not too hard to grasp, just impossible to read and validate. That's the price you pay to avoid costly conversions to/from floats.
  
  > **Comments:**
  > **username_0**: Thank you very much! Why is net necessary to scale the input so that 2^(8+8) corresponds to the end of the LUT? Because that expression always matches the maximum value. Also, do you understand what the shifting of the result does? I understand it's fixed point, but I don't see where the exact values are derived from.  
  > **username_2**: I've made a major edit which should answer both questions, I think.  
  > **username_0**: Thanks! There are a few minor things I wonder after reading the update. Do you see any indication where the scaling factor for the input would come from? Also, you mention that they subtract the zero-point, but what zero point does this refer to? Finally, why do they add 128 to round? That number seems very random. Is it just an engineering choice, or is there some deeper meaning?  
  > **username_2**: The first value in the table (see plot) is used for input 0. Because `tanh(0)=0` this is the zero-point. They multiply it by 256 with `(ua << 8)` and later divide by 256 with an arithmetic right shift (line 108, `result >>= 8`). This division truncates the result (rounds towards -oo). If you add 128 first, you get normal rounding.  
  > **username_2**: For the scaling factor, you'd have to read the rest of their code. I've only read the part that interested me personally, which was the LUT.  
  > **username_0**: Thank you so much! This clears up a lot of my confusion. For the actual inputs: Are those just regular int values that are interpreted as fixed point here for the sake of the interpolation? Also, can you recommend any resources on fixed point operations? I feel like I am lacking quite a bit here.  
  
  ---
  
  ### Answer by username_3 on 2023-08-23. Score: 0
  
  Please refer to following document link, we convert any Q15 integer to Q4.12 fixed point format with help of input scaling with factor of 3/4. Q4.12 {3 bits - Integer, 12 bits of fraction}. Then ua uses upper 8 bits {Q4(INT).4(precision)} while ut uses lower fixed point precision.
  
  Link - <https://github.com/tensorflow/tflite-micro/pull/1253>
  
  Regards,
  Chaitanya
  
  > **Comments:**
  > **username_0**: Thanks! But I don't understand what scaling with the factor 3/4 does, or where in the code it is applied. Could you expand on that?  
  
  ---
  
  