Are you using multiple GPUs? If yes, you need to use torch.cuda.manual_seed_all.
This should be enough and is enough for getting deterministic results in all examples we have. The problem probably lies somewhere in your code.