Solved by lupus83 in post #10 
              
              
                Yeah, that’s it! Thanks a lot for checking it out. 
I experimented with different settings of cudnn and found that calling 
torch.backends.cudnn.deterministic = True

was sufficient to solve the issue. 
Some additional info with respect to runtime per batch for future readers (ii and iii solve the i…