Solved by ptrblck in post #2 
              
              
                Thanks for the code snippet! 
I debugged it a bit and think you are seeing some non-determinism due to some atomic operations. 
From the reproducibility docs: 

There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atom…