Hey @lkp411, in the context of DDP, there are two different types of synchronizations, intra-process (CUDA stream) and inter-process (collective comm). The gradient averaging (AllReduce) is inter-process sync and the CPU-to-GPU copy is a intra-process sync. Which ones are you referring to in the following question?

What are the other sync points in DDP?