What’s the number of workers for your dataloader? The following post might be helpful for deterministic results.

  
    
    
    Deterministic/non-deterministic results with PyTorch 
  
  
    Hi, 
I’ve had some trouble in reproducing results, and after reading a few posts there seem to be multiple causes for non-determinism, some which are expected. I’d like to check if I got it right. 
Issue 1 
Dataloaders with multiple threads: They seem to be problematic, even if randoms seeds are set beforehand. [ref]. 
Issue 2 
cuDNN: Apparently cuDNN seems to have non-deterministic kernels [ref1][ref2] 
Issue 3 
GPU: Apparently, some reductions are non-deterministic in GPU, even without cuDNN. …