How do you do validation with a model wrapped with DistributedDataParallel? I imagine you call .eval() everywhere and have every worker validate a subset of your validation set, followed by allreduce of the accuracies. Is this correct?
Another piece of information that might be useful here: if you construct DistributedDataParallel with broadcast_buffers=True then all ranks will receive a copy of the registered buffers prior to executing a forward pass. With respect to batch normalization this means they all use the running stats from rank 0.