nn.SynchBatchNorm 318 was added into the PyTorch Master branch 9 days ago via this PR 46. Alternatively, you could use the NVIDIA/apex implementation from here 67, if you don’t want to build PyTorch from source.