Hi, this seems like it might be a possible bug with DataParallel, but a repro with some example model and training code would be needed to confirm. If you’re able to get that could you please post an issue to Issues · pytorch/pytorch · GitHub?