DistributedDataParallel can be used in two different setups as given in the docs 34.

Single-Process Multi-GPU and
Multi-Process Single-GPU, which is the fastest and recommended way.

SyncBatchNorm will only work in the second approach.
I’m not sure, if you would need SyncBatchNorm, since FrozenBatchNorm seems to fix all buffers:

BatchNorm2d where the batch statistics and the affine parameters are fixed.
It contains non-trainable buffers called
“weight” and “bias”, “running_mean”, “running_var”,
initialized to perform identity transformation.