You still have cudnn random algorithms (if using cudnn) and data loader worker multiprocessing randomness (if using num_workers > 0).
For the former, you can set torch.backends.cudnn.deterministic=True. For the later, set num_workers=0 or build from a commit after https://github.com/pytorch/pytorch/pull/4640 36.