Java’s floating-point hurts everyone everywhere (1998) [pdf]
The problems were not specific to Java, they also exist in most modern programming environments, including FORTRAN and ANSI C. Java just happened to be the hottest language in the news in the late 90s, so the author decided to specifically target Java - &quot;hurts everyone everywhere&quot; was a wordplay of Java&#x27;s advertisement.<p>The main author of this report, W. Kahan, was also the original author of IEEE-754. He was (and is) strongly unsatisfied on the state of floating-point in practical systems, and it was not his first time to complain about these problems. One can find his most recent critique from a few years ago.<p>My understanding is that, Kahan&#x27;s IEEE-754 is meant to:<p>1. The use of double and extended precision should be encouraged to safeguard non-experts from floating-point errors. Everything should be at least double precision by default, and extended precision serves as an additional safeguard. In fact, when IEEE-754 was being drafted, Kahan believed 128-bit floating point should also be supported as computers become more powerful in the future. The computational cost was too high at that time, so he settled on 80-bit as seen on the 8087. He criticized Java for not supporting it.<p>2. Floating-point exceptions should be used and turned on everywhere to safeguard programmers from making mistakes, and possibly to allow programs to handle them as special cases in the logic at runtime. He criticized Java for not supporting it.<p>3. Unsafe optimizations should not be done, such as automatically using FMA, or using algebraic identity in compiler optimization.  He criticized Java for allowing it in some cases.<p>Unfortunately, as far as I can see, these ideals of IEEE-754 have all but largely disappeared in real-world applications since then, for various practical reasons.<p>The industry did not move to 128-bit floating point because its performance overhead is too large, the original assessment of IEEE-754 in the 1980s was too optimistic. Similarly, the industry did not accept 80-bit extended precision as a standard but saw it as an oddity of the Intel 8087, even Intel has abandoned it - x86_64&#x27;s SSE or AVX has removed all support of that, making it an exclusive feature of obsolete i386&#x2F;i686 machines. My impression is that anything above double precision is no longer used in the industry (IBM POWER does have native quad-precision support, to their credit).<p>A secondary overhead of higher-precision floats is the memory wall, arguably a more serious problem today. Memory bandwidth has become the most serious overhead for many numerical programs. Modern computers have a machine balance of 100:1, it means you need to do as many as 100 floating-point operations after loading a single value from RAM to reach the machine&#x27;s peak performance. But this is not compatible with many algorithms with an inherently low arithmetic intensity, including important physics simulations. The use of of FP80 or FP128 will make them unacceptably slower. As a result, today&#x27;s trend is moving from FP64 to FP32, and even to FP16 or a custom 16-bit format if possible, not vice versa.<p>The use of floating-point exceptions similarly became unpopular because of performance problems. I&#x27;m not an expert on this, but it was my impression that signaling a floating-point exception was so expensive on both hardware and operation-system level that it was never seriously used in most practical programs. So in contrary to IEEE-754, these exceptions never became an integral part of the programming environment. In addition to an expensive Unix signal, potential exceptions also inhibit the efficient pipelining of modern out-of-order, superscalar CPUs.<p>Finally, the use of unsafe optimizations is prevalent in many applications when rigor is sacrificed for speed.<p>So overall, the original spirit of IEEE-754 was long gone - for better or worse, unfortunately.