When torch.nn.parallel.DistributedDataParallel is initialized with the right distributed context then every iteration should happen in lock step between all processes. If a single process starts going by itself, I think there is something missing in initialization.
Can you share a code snippet how you initialize all of them?