DDP instances need to all participate in the backward, otherwise it would hang. But there are work around. If you know that master would run say 100 epochs, and other nodes would run 80 epochs, you can call forward-backward on the DDP instance for 80 epochs. After that, you can delete the DDP instance, which will remove the DDP grad hooks accordingly. Then, you can run forward-backward on DDP.module (as DDP is deleted, you won’t be able to call DDP.module, but the program can still have a reference to the original local module separately) on master, and it will no longer trigger communications.