Related issue: Distributed: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048]] is at version 4; expected version 3 instead · Issue #62474 · pytorch/pytorch · GitHub 1
May be related to the BatchNorm layers and replacing those with SyncBatchNorm — PyTorch 1.9.1 documentation 1 would help