Hi Zvant!



 zvant:

with an RTX30 GPU
…
But for 32-bits floats, CUDA tensors give quite different results
…
I have another machine with GTX1080Ti


Depending on your  GPU, nvidia might be switching you over by
default to the misleading (dishonestly?) named “tf32” floating-point
arithmetic.  (tf32 is essentially half-precision floating-point.)
You can try turning tf32 off with:
torch.backends.cuda.matmul.allow_tf32 = False

See the following thread and the github issue @tom references in it:

  
    
    
    Bug? matmul seems to cast to float16 internally 
  
  
    What’s going on here? Comments will be appreciated. 
[[1.0]] multiplied by [[1.0001]] incorrectly results in 1., whereas [1.0] multiplied by [1.0001] correctly results in 1.0001 on Win10. 
It seems as if the internal accumulator is only float16. It works with float64, or without using CUDA. 
Cannot reproduce on Ubuntu machine. 
Code 
import torch 
dtype = torch.float32 
A = torch.tensor([[1.]], dtype=dtype).cuda() 
B = torch.tensor([[1.0001]], dtype=dtype).cuda() 
test1 = torch.matmul(A, B) 
A =…
  


Best.
K. Frank