I tried to upgrade pytorch to 1.1.0 and re-run my code. But the problem is still happening. I don’t change any hyperparameters, only change running from a single GPU to multiple GPU by using torch.DataParallel(). Is there any reason for this?