I think you would also need to set the deterministic flag to True:
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

From docs: https://pytorch.org/docs/stable/notes/randomness.html 2