Hi Zebulun!



 Zebulun:

Compute comparison tests using the same data：
sum：0.0179971529
numpy： 0.017990112.  (RE=0.04%)
torch cuda：0.0179862976.  (RE=0.06%)
torch cpu：0.0181007385  （RE=0.57%）


As Utku suggested with his link, the imprecision you see is expected
as it is consistent with 32-bit floating-point round-off error.
You are summing over about 15,000 normally distributed values, so
the expected size of your sum is about 124.  Let’s treat your numpy
result as the exact result.  Then your cpu result is off in the fourth
place after the decimal point, so relative to the expected value, this
is fully consistent with round-off error.
However, there is something fishy going on with your numbers.  The
value you get for your sum is improbably small.  As I mentioned
above, the expected size of your sum is about 124, so a value of
0.018 is very unlikely – about one chance in ten thousand.  I can’t
think of any good reason this should happen, and it would be pushing
your luck to suggest that luck explains it.
Here is a script that goes through the calculations relevant to these
two points:
import numpy as np

tdim = (2, 128, 10, 6)
sumcpu = 0.0181007385
sumnpy = 0.017990112

nSum = np.product (tdim)
expected_size = np.sqrt (nSum)

relerr = (sumcpu - sumnpy) / expected_size

print ('nSum =', nSum, ', expected_size =', expected_size)
print ('relerr =', relerr)

import math

prob_small = math.erf (abs (sumnpy) / (math.sqrt (2.0) * expected_size))
print ('prob_small =',  prob_small)

And here is its output:
>>> import numpy as np
>>>
>>> tdim = (2, 128, 10, 6)
>>> sumcpu = 0.0181007385
>>> sumnpy = 0.017990112
>>>
>>> nSum = np.product (tdim)
>>> expected_size = np.sqrt (nSum)
>>>
>>> relerr = (sumcpu - sumnpy) / expected_size
>>>
>>> print ('nSum =', nSum, ', expected_size =', expected_size)
nSum = 15360 , expected_size = 123.93546707863734
>>> print ('relerr =', relerr)
relerr = 8.92613733644216e-07
>>>
>>> import math
>>>
>>> prob_small = math.erf (abs (sumnpy) / (math.sqrt (2.0) * expected_size))
>>> print ('prob_small =',  prob_small)
prob_small = 0.00011581860221173588

Two things:
relerr = 8.92613733644216e-07
32-bit round-off error is about 1.e-7.  You are summing over about
15,000 values of order one, so the round-off error could easily
accumulate to this level.
prob_small = 0.00011581860221173588
When you sum over 15,000 (independent) normally distributed values
(so each value has standard deviation one), the sum comes from a
gaussian distribution that has a standard deviation of about 124 (and
a mean of zero).  You can calculate the probability of such a sum
having a magnitude smaller that 0.018 using the so-called error
function, and that probability is very small – about 1% of 1%.
Best.
K. Frank