mrshenli:


kicked off whenever a gradient bucket is ready.

This  is a brief description of how DDP works. You


Hi Shen,
Thanks a lot for your reply! The scope of my work is similar to this paper TicTac: Accelerating Distributed Deep Learning with Communication Scheduling, https://arxiv.org/abs/1803.03288 3 that will use heuristics to pipelining communication and computation. Specifically, I want to design a new parameter server algorithm that will use a set of distributed servers and pipeline the communication/computation to improve the bottleneck.