I am using DDP, and I am usiong BatchNorm in my network. If I do not set track_running_stats=False, then I get the following error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)

I tried to switch it to SyncBatchNorm, and I still get the same error.