Solved by vmirly1 in post #3 
              
              
                I think you would also need to set the deterministic flag to True: 
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

From docs: https://pytorch.org/docs/stable/notes/randomness.html