Hi @DzReal

From pytorch source code, it seems like during forward pass, all model replicas will be synced with the one running in subprocess with rank 0.

If you are using DistributedDataParallel, above is actually not true. The distributed sync occurs during the backward pass, and it averages all gradients instead of parameters (check torch/csrc/distributed/c10d/reducer.cpp). So that when the optimizer consumes those gradients, they are already global gradients.
The sync you mentioned in the forward pass might be this 13. This only does intra-node sync when you use one DDP process to work on multiple GPUs.

Does that mean I could just initialise one optimiser for subprocess 0 and only update the parameters of the first model replica?

No. Each DDP process should have its own local optimizer.