Thanks!
num_workers = 0 and torch.backends.cudnn.enabled = False are the real thing that works! And I also see that if you train one step 10 times,  only  using num_workers = 0 we can get exactly same output 8 times and different output 2 times.