Solved by ratishsp in post #2 
              
              
                I think the reason for this is explained in https://pytorch.org/docs/stable/notes/randomness.html. 
There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular atomicAdd, where the order of pa…