I was looking at def train in DistributedDataParallel, more specifically these lines 8 and am worried the slicing of replicas may be causing this. Looking at def train in nn.Module I don’t see how the train mode would be set on self.module. Can you try removing the slicing to ensure the train mode is set on every module replica (even if there aren’t any) and see what happens? Since the train mode controls whether or not you’re accumulating running stats in batch norm layers, this could explain both the regression and the recovery after a couple of iterations.