Hi Amity!



 amity137:

So are you saying that if we do a series of flops in the same order, we’d always get the same result (till 20 digits). But if we change the order of flops, while keeping the process identical logically, the final result would be different?


Yes, this is true (for any reasonable definition of “do a series
of flops in the same order,” and excluding perverse hardware
implementations).
Consider this:
Mathematically, addition is commutative, i.e. x + y = y + x.
It is also associative: x + (y + z) = (x + y) + z.
Floating-point addition can be (and should be) commutative.  But
there is no reasonable way to make floating-point addition
associative.
1.0 + ((-1.0) + 1.e-20) = 0.0 with floating-point arithmetic,
while (1.0 + (-1.0)) + 1.e-20 = 1.e-20.  This is because
(-1.0) + 1.e-20 = (-1.0), as neither floats nor doubles have
enough precision to represent (-1.0) + 1.e-20 as a number
different from -1.0.  There’s really no way around this.
So if you change the order of operations in x + y + z, you
get the same answer mathematically, but not necessarily  with
floating-point arithmetic.
Hello Yaroslav !



 Yaroslav_Bulatov:

You lose precision in matmul and could get different results even if you run instructions in the exact same order, here’s an explanation – https://www.nag.com/content/wandering-precision


This isn’t correct (at least for what one would normally mean by
“run instructions in the exact same order”), and your nag reference
doesn’t support this.  Quoting from your nag reference:

The problem is that by messing with the order of the accumulations, you are quite possibly changing the final result, simply due to rounding differences when working with finite precision computer arithmetic.

They talk about logic at a higher level (in this case the compiler)
choosing to change the order of the floating-point operations,
causing the result to change (at the level of floating-point precision).
Best.
K. Frank