You can specify the dimensionality of your input into SyncBatchNorm, so as long as the dimensions are aligned then it will be fine. Here is the follow up issue to track this: BatchNorm runtimeError: one of the variables needed for gradient computation has been modified by an inplace operation · Issue #66504 · pytorch/pytorch · GitHub 7 as well as some additional workarounds.