Floating Point in the Browser, Part 2: Bad Epsilon
I think the real problem is that IEEE 754 floating point almost always gets treated as black box that magically provides the behavior of real numbers, which is impossible. Instead of recognizing that the abstraction leaks in cases like these because the floating point representation only has so many bits, people just ignore the mathematical reality. The right thing to do is descending to the level of the representation (bits) and comparing ULP differences when necessary.<p>The most annoying thing is how the bit-level representation of floating point is ignored even when relative error is significant instead of absolute error, because ULP differences are just that - relative error.<p>EDIT: the epsilon fix in the test code in the blog post actually converts the relative error to absolute error so both could be done with the absolute error check. It would be more clear to instead do both checks explicitly. I think it would also be faster, which might matter in some situations.