Hi Tobias!



 Tobias_Kalb:

Is this just some float precision error,


As you suspect, this is very likely floating-point round-off error.
As the size of a tensor changes, pytorch can change the details of
how it pipelines tensor operations, both on the cpu and gpu.  This
can cause operations to be reordered in a way that is mathematically
equivalent, but that produces differing round-off errors.
You could check this by repeating your experiment in double-precision.
I would then expect the discrepancy to be reduced by several orders of
magnitude.
Note that the details of when and how the pipelining changes with
tensor size will differ between the single-precision and double-precision
cases, so you might have to play around with the tensor sizes to see
a similar effect.
Best.
K. Frank