Solved by H-Huang in post #3 
              
              
                Related issue: Distributed: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048]] is at version 4; expected version 3 instead · Issue #62474 · pytorch/pytorch · GitHub 
May be related to the BatchNorm layers and …