Answers.Body: <p><strong>Floating Point Hardware</strong></p>
  
  <p>There are three common floating point formats used to approximate real numbers used in digital arithmetic circuitry.  These are defined in IEEE 754, a standard that was adopted in 1985 with a revision in 2008.  These mappings of bit-wise layouts real number representations are designed into CPUs, FPUs, DSPs, and GPUs either in gate level hardware, firmware, or libraries<sup>1</sup>.</p>
  
  <ul>
  <li>binary 32 has a 24 bit mantissa and an 8 bit exponent</li>
  <li>binary 64 has a 53 bit mantissa and an 11 bit exponent</li>
  <li>binary 128 has a 113 bit mantissa and a 15 bit exponent<sup>2</sup></li>
  </ul>
  
  <p><strong>Factors in Choosing Numerical Representations</strong></p>
  
  <p>Any of these can represent signals in signal processing, and all have been experimented with in AI for various purposes related to three things:</p>
  
  <ul>
  <li>Value range &mdash; not a concern in ML applications where the signal is properly normalized</li>
  <li>Averting saturation of the signal with rounding noise &mdash; a key issue in parameter adjustment</li>
  <li>Time required to execute an algorithm on a given target architecture</li>
  </ul>
  
  <p>The balance in the best designed AI is between these last two items.  In the case of back-propagation in neural nets, the gradient-based signal that approximates the desired corrective action to apply to the attenuation parameters of each layer must not become saturated with rounding noise.<sup>3</sup></p>
  
  <p><strong>Hardware Trends Favor Floating-point</strong></p>
  
  <p>Because of the demand of certain markets and common uses, scalar, vector, or matrix operations using these standards may, in certain cases, be faster than integer arithmetic.  These markets include ...</p>
  
  <ul>
  <li>Closed loop control (piloting, targeting, countermeasures)</li>
  <li>Code breaking (Fourier, finite, convergence, fractal)</li>
  <li>Video rendering (movie watching, animation, gaming, VR)</li>
  <li>Scientific computing (particle physics, astrodynamics)</li>
  </ul>
  
  <p><strong>First Degree Transforms to Integers</strong></p>
  
  <p>On the opposing end of numerical range, one can represent signals as integers (signed) or non-negative integers (unsigned).</p>
  
  <p>In this case, the transformation between the set of real numbers, vectors, matrices, cubes, and hyper-cubes in the world of calculus<sup>4</sup> and the integers that approximate them is a first degree polynomial.</p>
  
  <p>The polynomial can be represented as $n = r(as + b)$, where $a \ne 0$, $n$ is the binary number approximation, $s$ is the scalar real, and $r$ is the function that rounds real numbers to the nearest integer.  This defines a super-set of the concept of fixed point arithmetic because of $b$.</p>
  
  <p>Integer based calculations have also been examined experimentally for many AI applications.  This gives more options:</p>
  
  <ul>
  <li>two's complement 16 bit integer</li>
  <li>16 bit non-negative integer</li>
  <li>two's complement 32 bit integer</li>
  <li>32 bit non-negative integer</li>
  <li>two's complement 64 bit integer</li>
  <li>64 bit non-negative integer</li>
  <li>two's complement 128 bit integer</li>
  <li>128 bit non-negative integer</li>
  </ul>
  
  <p><strong>Example Case</strong></p>
  
  <p>For instance, if your theory indicates the need to represent the real numbers in the range $[-\pi, \pi]$ in some algorithm, then you might represent this range as a 64 bit non-negative integer (if that works to the advantage of speed optimization for some reason that is algorithm and possibly hardware specific).</p>
  
  <p>You know that $[-\pi, \pi]$ in the closed form (algebraic relation) developed from the calculus needs to be represented in the range $[0, 2^64 - 1]$, so in $n = r(as + b)$, $a = 2^61$ and $b = 2^60$. Choosing $a = \frac {2^64} {\pi}$ would likely create the need for more lost cycles in multiplication when a simple manipulation of the base two exponent is much more efficient.</p>
  
  <p>The range of values for that real number would then be [0, 1100,1001,0000,1111,1101,1010,1010,0010,0010,0001,0110,1000,1100,0010,0011,0101] and the number of bits wasted by keeping the relationship based on powers of two will be $\log_2 4 - log_2 \pi$, which is approximately 0.3485 bits.  That's better than 99% conservation of information.</p>
  
  <p><strong>Back to the Question</strong></p>
  
  <p>The question is a good one, and is hardware and domain dependent.</p>
  
  <p>As mentioned above hardware is continuously developing in the direction to work with IEEE floating point vector and matrix arithmetic, particularly in 32 and 64 bit floating point multiplication.  For some domains and execution targets (hardware, bus architecture, firmware, and kernel), the floating point arithmetic may grossly out perform what performance gains can be obtained in 20th century CPUs by applying the above first degree polynomial transformation.</p>
  
  <p><strong>Why the Question is Relevant</strong></p>
  
  <p>In contract, if the product manufacturing price, the power consumption, and PC board size and weight must be kept low to enter certain consumer, aeronautic, and industrial markets, the low cost CPUs may be demanded. By design, these smaller CPU architectures do not have DSPs and the FPU capabilities don't usually have hardware realization of 64 bit floating point multiplication.<sup>5</sup></p>
  
  <p><strong>Handling Number Ranges</strong></p>
  
  <p>Care in normalizing signals and picking the right values for <em>a</em> and <em>b</em> are essential, as mentioned, more so than with floating point, where the diminution of the exponent can eliminate many cases where saturation would be an issue with integers<sup>6</sup>.  Augmentation of the exponent can avert overflow automatically too, up to a point, of course.</p>
  
  <p>In either type of numeric representation, normalizing is part of what improves convergence rate and reliability anyway, so it should always be addressed.</p>
  
  <p>The only way to deal with saturation in applications requiring small signals (such as with gradient descent in back-propagation, especially when the data set ranges are over an order of magnitude) is to carefully work out the mathematics to avoid it.</p>
  
  <p>This is a science by itself, and only a few people have the scope of knowledge to handle hardware manipulation at the circuitry and assembly language level along with the linear algebra, calculus, and machine learning comprehension.  The interdisciplinary skill set is rare and valuable.</p>
  
  <hr>
  
  <p><strong>Notes</strong></p>
  
  <p>[1] Libraries for low level hardware operations such as 128 bit multiplication are written in cross assembly language or in C with the -S option turned on so the developer can check the machine instructions.</p>
  
  <p>[2] Unless you are calculating the number of atoms in the universe, the number of permutations in possible game-play for the game Go, the course to a landing pad in a crater on Mars, or the distance in meters to reach a potentially habitable planet revolving around Proxima Centauri, you will likely not need larger representations than the three common IEEE floating point representations.</p>
  
  <p>[3] Rounding noise naturally arises when digital signals approach zero and the rounding of the least significant bit in the digital representation begins to produce chaotic noise of a magnitude that approaches that of the signal.  When this happens, the signal is saturated in that noise and cannot be used to reliably communicate the signal.</p>
  
  <p>[4] The closed forms (algebraic formulae) to be realized in software driven algorithms arise out of the solution of equations, usually involving differentials.</p>
  
  <p>[5] Daughter boards with GPUs are often too pricey, power hungry, hot, heavy, and/or packaging unfriendly in non-terrestrial and consumer markets.</p>
  
  <p>[6] The zeroing of feedback is skipped in this answer because it points to either one of two things: (A) Perfect convergence or (B) Poorly resolved mathematics.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-11T20:58:28.993Z
Answers.Text: There are a few typos in your post, such as `[0, 2^64 - 1]` which renders as $[0, 2^64 - 1]$. From context, I think you mean $[0, 2^{64} - 1]$.
Answers.UserDisplayName: Sycorax
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-29T01:26:28.747Z
Answers.LastActivityDate: 2018-08-16T13:25:04.813Z
Answers.LastEditDate: 2018-08-16T13:25:04.813Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>It is possible in principle, but you will end up emulating floating point arithmetic using integers in multiple places, so it is unlikely to be efficient for general use. Training is likely to be an issue.</p>
  
  <p>If the output of a layer is scaled to [-INT_MAX, INT_MAX], then you need to multiply those values by weights (which you will also need to be integers with a large enough range that learning can progress smoothly) and sum them up, and then feed them into a non-linear function. </p>
  
  <p>If you are restricting yourself to integer operations only, this will involve handling multiple integers to represent high/low words in a larger int type, that you then must scale (which introduces a multi-word integer division). By the time you have done this, it is unlikely there will be much benefit to using integer arithmetic. Although perhaps it is still possible, depending on the problem, network architecture and your target CPU/GPU etc.</p>
  
  <p>There are instances of working neural networks used in computer vision <a href="https://petewarden.com/2015/05/23/why-are-eight-bits-enough-for-deep-neural-networks/" rel="nofollow noreferrer">with only 8-bit floating point (downsampled after training)</a>. So it is definitely possible to simplify and approximate some NNs. An integer-based NN derived from a pre-trained 32-bit fp one could potentially offer good performance in certain embedded environments. I found <a href="http://sharpneat.sourceforge.net/research/integer-neuralnet/integer-neuralnet.html" rel="nofollow noreferrer">this experiment on a PC AMD chip</a> which showed a marginal improvement even on PC architecture. On devices without dedicated fp processing, the relative improvement should be even better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T14:02:55.173Z
Answers.Text: Hello Neil! Do you have time to give an answer to this question https://ai.stackexchange.com/questions/20871/what-is-the-advantage-of-using-more-than-one-environment-with-the-advantage-acto ?
Answers.UserDisplayName: jgauth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-28T22:44:03.413Z
Answers.Text: A multi-word integer division would surely kill performance, but nobody should ever thing about it. Such a scaling gets simply done by bit shifting, which is maybe two orders of magnitude faster.
Answers.UserDisplayName: maaartinus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-22T16:41:10.763Z
Answers.LastActivityDate: 2018-07-22T16:41:10.763Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <h1>TL;DR</h1>
  
  <p>Not only it's possible, it even gets done and is <a href="https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu" rel="noreferrer">commercially available</a>. It's just impractical on a commodity HW which is pretty good at FP arithmetic.</p>
  
  <h1>Details</h1>
  
  <p>It is definitely possible, and you <em>might</em> get some speed for a lot of troubles.</p>
  
  <p>The nice thing about floating point is that it works without you knowing the exact range. You can scale your inputs to the range <code>(-1, +1)</code> (such a scaling is pretty commonplace as it speeds up the convergence) and multiply them by <code>2**31</code>, so they used the range of signed 32-bit integers. That's fine.</p>
  
  <p>You can't do the same to your weights as there's no limit on them. You can assume them to lie in the interval <code>(-128, +128)</code> and scale them accordingly.</p>
  
  <p>If your assumption was wrong, you get an overflow and a huge negative weight where a huge positive weight should be or the other way round. In any case, a disaster.</p>
  
  <p>You could check for overflow, but this is too expensive. Your arithmetic gets slower than FP.</p>
  
  <p>You could <em>check for possible overflow from time to time</em> and take a corrective action. The details may get complicated.</p>
  
  <p>You could use <a href="https://en.wikipedia.org/wiki/Saturation_arithmetic" rel="noreferrer">saturation arithmetic</a>, but it's implemented only in some specialized CPUs, not in your PC.</p>
  
  <p>Now, there's a multiplication. With use of 64-bit integers, it goes well, but you need to compute a sum (with a possible overflow) and scale the output back to same sane range (another problem).</p>
  
  <p><em>All in all, with fast FP arithmetic available, it's not worth the hassle.</em></p>
  
  <p>It might be a good idea for a custom chip, which could do saturation integer arithmetic with much less hardware and much faster then FP.</p>
  
  <hr>
  
  <p>Depending on what integer types you use, there may be a precision loss when compared to the floating point, which may or may not matter. Note that TPU (used in AlphaZero) has 8-bit precision only.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-28T22:42:14.073Z
Answers.LastActivityDate: 2018-07-29T04:14:13.680Z
Answers.LastEditDate: 2018-07-29T04:14:13.680Z
Answers.UserDisplayName: maaartinus
Body: <p>Is it possible to make a neural network that uses only integers by scaling input and output of each function to [-INT_MAX, INT_MAX]? Is there any drawbacks?</p>
  
Tags: <neural-networks><machine-learning>
Title: Why do we need floats for using neural networks?
UserDisplayName: elimohl
ThreadText: # Why do we need floats for using neural networks?
  
  (Asked by: username_0 on 2018-07-22)
  
  Is it possible to make a neural network that uses only integers by scaling input and output of each function to [-INT\_MAX, INT\_MAX]? Is there any drawbacks?
  
  **Tags:** neural-networks, machine-learning
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2018-07-28. Score: 10 | Accepted answer
  
  # TL;DR
  
  Not only it's possible, it even gets done and is [commercially available](https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu). It's just impractical on a commodity HW which is pretty good at FP arithmetic.
  
  # Details
  
  It is definitely possible, and you *might* get some speed for a lot of troubles.
  
  The nice thing about floating point is that it works without you knowing the exact range. You can scale your inputs to the range `(-1, +1)` (such a scaling is pretty commonplace as it speeds up the convergence) and multiply them by `2**31`, so they used the range of signed 32-bit integers. That's fine.
  
  You can't do the same to your weights as there's no limit on them. You can assume them to lie in the interval `(-128, +128)` and scale them accordingly.
  
  If your assumption was wrong, you get an overflow and a huge negative weight where a huge positive weight should be or the other way round. In any case, a disaster.
  
  You could check for overflow, but this is too expensive. Your arithmetic gets slower than FP.
  
  You could *check for possible overflow from time to time* and take a corrective action. The details may get complicated.
  
  You could use [saturation arithmetic](https://en.wikipedia.org/wiki/Saturation_arithmetic), but it's implemented only in some specialized CPUs, not in your PC.
  
  Now, there's a multiplication. With use of 64-bit integers, it goes well, but you need to compute a sum (with a possible overflow) and scale the output back to same sane range (another problem).
  
  *All in all, with fast FP arithmetic available, it's not worth the hassle.*
  
  It might be a good idea for a custom chip, which could do saturation integer arithmetic with much less hardware and much faster then FP.
  
  ---
  
  Depending on what integer types you use, there may be a precision loss when compared to the floating point, which may or may not matter. Note that TPU (used in AlphaZero) has 8-bit precision only.
  
  
  ---
  
  ### Answer by username_1 on 2018-07-29. Score: 3
  
  **Floating Point Hardware**
  
  There are three common floating point formats used to approximate real numbers used in digital arithmetic circuitry. These are defined in IEEE 754, a standard that was adopted in 1985 with a revision in 2008. These mappings of bit-wise layouts real number representations are designed into CPUs, FPUs, DSPs, and GPUs either in gate level hardware, firmware, or libraries1.
  
  * binary 32 has a 24 bit mantissa and an 8 bit exponent
  * binary 64 has a 53 bit mantissa and an 11 bit exponent
  * binary 128 has a 113 bit mantissa and a 15 bit exponent2
  
  **Factors in Choosing Numerical Representations**
  
  Any of these can represent signals in signal processing, and all have been experimented with in AI for various purposes related to three things:
  
  * Value range — not a concern in ML applications where the signal is properly normalized
  * Averting saturation of the signal with rounding noise — a key issue in parameter adjustment
  * Time required to execute an algorithm on a given target architecture
  
  The balance in the best designed AI is between these last two items. In the case of back-propagation in neural nets, the gradient-based signal that approximates the desired corrective action to apply to the attenuation parameters of each layer must not become saturated with rounding noise.3
  
  **Hardware Trends Favor Floating-point**
  
  Because of the demand of certain markets and common uses, scalar, vector, or matrix operations using these standards may, in certain cases, be faster than integer arithmetic. These markets include ...
  
  * Closed loop control (piloting, targeting, countermeasures)
  * Code breaking (Fourier, finite, convergence, fractal)
  * Video rendering (movie watching, animation, gaming, VR)
  * Scientific computing (particle physics, astrodynamics)
  
  **First Degree Transforms to Integers**
  
  On the opposing end of numerical range, one can represent signals as integers (signed) or non-negative integers (unsigned).
  
  In this case, the transformation between the set of real numbers, vectors, matrices, cubes, and hyper-cubes in the world of calculus4 and the integers that approximate them is a first degree polynomial.
  
  The polynomial can be represented as $n = r(as + b)$, where $a \ne 0$, $n$ is the binary number approximation, $s$ is the scalar real, and $r$ is the function that rounds real numbers to the nearest integer. This defines a super-set of the concept of fixed point arithmetic because of $b$.
  
  Integer based calculations have also been examined experimentally for many AI applications. This gives more options:
  
  * two's complement 16 bit integer
  * 16 bit non-negative integer
  * two's complement 32 bit integer
  * 32 bit non-negative integer
  * two's complement 64 bit integer
  * 64 bit non-negative integer
  * two's complement 128 bit integer
  * 128 bit non-negative integer
  
  **Example Case**
  
  For instance, if your theory indicates the need to represent the real numbers in the range $[-\pi, \pi]$ in some algorithm, then you might represent this range as a 64 bit non-negative integer (if that works to the advantage of speed optimization for some reason that is algorithm and possibly hardware specific).
  
  You know that $[-\pi, \pi]$ in the closed form (algebraic relation) developed from the calculus needs to be represented in the range $[0, 2^64 - 1]$, so in $n = r(as + b)$, $a = 2^61$ and $b = 2^60$. Choosing $a = \frac {2^64} {\pi}$ would likely create the need for more lost cycles in multiplication when a simple manipulation of the base two exponent is much more efficient.
  
  The range of values for that real number would then be [0, 1100,1001,0000,1111,1101,1010,1010,0010,0010,0001,0110,1000,1100,0010,0011,0101] and the number of bits wasted by keeping the relationship based on powers of two will be $\log\_2 4 - log\_2 \pi$, which is approximately 0.3485 bits. That's better than 99% conservation of information.
  
  **Back to the Question**
  
  The question is a good one, and is hardware and domain dependent.
  
  As mentioned above hardware is continuously developing in the direction to work with IEEE floating point vector and matrix arithmetic, particularly in 32 and 64 bit floating point multiplication. For some domains and execution targets (hardware, bus architecture, firmware, and kernel), the floating point arithmetic may grossly out perform what performance gains can be obtained in 20th century CPUs by applying the above first degree polynomial transformation.
  
  **Why the Question is Relevant**
  
  In contract, if the product manufacturing price, the power consumption, and PC board size and weight must be kept low to enter certain consumer, aeronautic, and industrial markets, the low cost CPUs may be demanded. By design, these smaller CPU architectures do not have DSPs and the FPU capabilities don't usually have hardware realization of 64 bit floating point multiplication.5
  
  **Handling Number Ranges**
  
  Care in normalizing signals and picking the right values for *a* and *b* are essential, as mentioned, more so than with floating point, where the diminution of the exponent can eliminate many cases where saturation would be an issue with integers6. Augmentation of the exponent can avert overflow automatically too, up to a point, of course.
  
  In either type of numeric representation, normalizing is part of what improves convergence rate and reliability anyway, so it should always be addressed.
  
  The only way to deal with saturation in applications requiring small signals (such as with gradient descent in back-propagation, especially when the data set ranges are over an order of magnitude) is to carefully work out the mathematics to avoid it.
  
  This is a science by itself, and only a few people have the scope of knowledge to handle hardware manipulation at the circuitry and assembly language level along with the linear algebra, calculus, and machine learning comprehension. The interdisciplinary skill set is rare and valuable.
  
  ---
  
  **Notes**
  
  [1] Libraries for low level hardware operations such as 128 bit multiplication are written in cross assembly language or in C with the -S option turned on so the developer can check the machine instructions.
  
  [2] Unless you are calculating the number of atoms in the universe, the number of permutations in possible game-play for the game Go, the course to a landing pad in a crater on Mars, or the distance in meters to reach a potentially habitable planet revolving around Proxima Centauri, you will likely not need larger representations than the three common IEEE floating point representations.
  
  [3] Rounding noise naturally arises when digital signals approach zero and the rounding of the least significant bit in the digital representation begins to produce chaotic noise of a magnitude that approaches that of the signal. When this happens, the signal is saturated in that noise and cannot be used to reliably communicate the signal.
  
  [4] The closed forms (algebraic formulae) to be realized in software driven algorithms arise out of the solution of equations, usually involving differentials.
  
  [5] Daughter boards with GPUs are often too pricey, power hungry, hot, heavy, and/or packaging unfriendly in non-terrestrial and consumer markets.
  
  [6] The zeroing of feedback is skipped in this answer because it points to either one of two things: (A) Perfect convergence or (B) Poorly resolved mathematics.
  
  > **Comments:**
  > **username_2**: There are a few typos in your post, such as `[0, 2^64 - 1]` which renders as $[0, 2^64 - 1]$. From context, I think you mean $[0, 2^{64} - 1]$.  
  
  ---
  
  ### Answer by username_3 on 2018-07-22. Score: 0
  
  It is possible in principle, but you will end up emulating floating point arithmetic using integers in multiple places, so it is unlikely to be efficient for general use. Training is likely to be an issue.
  
  If the output of a layer is scaled to [-INT\_MAX, INT\_MAX], then you need to multiply those values by weights (which you will also need to be integers with a large enough range that learning can progress smoothly) and sum them up, and then feed them into a non-linear function.
  
  If you are restricting yourself to integer operations only, this will involve handling multiple integers to represent high/low words in a larger int type, that you then must scale (which introduces a multi-word integer division). By the time you have done this, it is unlikely there will be much benefit to using integer arithmetic. Although perhaps it is still possible, depending on the problem, network architecture and your target CPU/GPU etc.
  
  There are instances of working neural networks used in computer vision [with only 8-bit floating point (downsampled after training)](https://petewarden.com/2015/05/23/why-are-eight-bits-enough-for-deep-neural-networks/). So it is definitely possible to simplify and approximate some NNs. An integer-based NN derived from a pre-trained 32-bit fp one could potentially offer good performance in certain embedded environments. I found [this experiment on a PC AMD chip](http://sharpneat.sourceforge.net/research/integer-neuralnet/integer-neuralnet.html) which showed a marginal improvement even on PC architecture. On devices without dedicated fp processing, the relative improvement should be even better.
  
  > **Comments:**
  > **username_5**: A multi-word integer division would surely kill performance, but nobody should ever thing about it. Such a scaling gets simply done by bit shifting, which is maybe two orders of magnitude faster.  
  > **username_4**: Hello username_3! Do you have time to give an answer to this question https://ai.stackexchange.com/questions/20871/what-is-the-advantage-of-using-more-than-one-environment-with-the-advantage-acto ?  
  
  ---
  
  