Hi,
my understanding is:
batchnorm could be synced during distribued training by API torch.nn.SyncBatchNorm.convert_sync_batchnorm().
Could layer_norm be synced during distributed training by API torch.nn.SyncBatchNorm.convert_sync_batchnorm()?
model_sync = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)