Dear @mrshenli,
I have briefly gone through the PipeDream 1 paper. Now I understand better how this parameter RPC example can be implicitly pipelined which speeds up the training phase for model parallelism. However, I still have several questions below and hope you could help to answer:


This rpc_parameter_example seems to me not a strict parameter server strategy for data parallelism training. Multiple trainers can update the “parameter server” however that is done separately by the distributed  Autograd which means there is no averaging for the gradients of each trainer, IIUC? Is a typical parameter server strategy looking more like

from Scaling Distributed paper instead? If I want to do this parameter server strategy (averaging the gradients of each trainer ) for data-parallel , is it possible to do that with RPC?


I remember last time you mentioned that by default the Distributed Data Parallel (DDP) uses ring allreduce for averaging the gradients. Is ring allreduce better than parameter server all the time? But can the DDP use parameter server strategy instead?


In the PipeDream 1 paper, the workload can be fine-grained and controlled as below:

Would RPC be able to do the same? Or what would be the order/priority for the RPC to choose running which job (1,2,3,4 forward or 1,2,3,4 backward) on the GPU?


Thank you very much!
Best,