Hi @mrshenli,
I wonder if I can ask some questions on the implementation details of the communication traffic for both DDP data-parallel and RPC model-parallel using multiple machines.
In the documentation, https://pytorch.org/docs/stable/notes/ddp.html?highlight=ddp, if I understand correctly the .backforward() will work on the local gradients and the DDP will help to synchronize the local gradients with the gradients in other nodes (not sure, perhaps ring-all reduce?). For the SGD optimizer as an example, the local gradients are first averaged over the batch size, and then the locally averaged gradients will be sent to other nodes in buckets by DDP. The traffic size would be roughly the size of the NN’s gradient size of a batch size of 1. Do I understand this correctly? Or some optimizers or networks do need to send some size (gradients * batch size)?
https://pytorch.org/docs/stable/rpc.html For the RPC, the Distributed Autograd will take care of the communication in the backward direction at the RPC boundaries of each machine. So the communication happens when the user code calls .to_here for the RREF and the Distributed Autograd passes the gradients in backward? The traffic does relate to the batch size * the size of feature maps at the boundaries?
Please let me know if I understand the mechanisms of the communication behind DDP and RPC correctly. Thank you very much for your time!