Unfortunately, I don’t think that this is easily doable in the current nn.DataParallel approach, as to the best of my knowledge, buffers won’t be synced (such as the running stats in batchnorm layers) by default.
You could try to either use the functional parallel calls and scatter and gather the buffer manually or alternatively, you could try to use DistributedDataParallel and reuse the code of SyncBatchNorm.