Are you using the GPU with cuDNN?
If so, you could set the cuDNN behavior to be deterministic, which would unfortunately trade performance for determinism.
torch.backends.cudnn.deterministic = True

Also, are you using some other libraries sampling random numbers?
If so, you should also seed them.