Thanks for posting @tymons, I noticed that you are using nn.DataParallel to do mutigpu training, since we don’t recommend using nn.DataParallel right now, could you try using distributed data parallel (nn.DistributedDataParallel) and see if the issue still exist? You can see the tutorial here Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.10.0+cu102 documentation