Hey all,
Why don’t you take a look at SyncBatchNorm 20.
From the link: For a network, already initialized, that has any BatchNormalization layers you can do :
sync_bn_network = torch.nn.utils.convert_sync_batchnorm(network, process_group)