From what I understand, in Distributed Data Parallel, the gradients are averaged and sent to all other rank1,rank2, … devices from rank0 process. Consider a case that 3 GPUs are used: rank0 process completes its 1st iteration at time T=10ms from the start, rank1 process completes its 1st iteration at time T=13ms  from the start and rank2 completes its 1st iteration at T=12ms. Then will rank0 and rank2 processes wait for rank1 process till T=13ms so that rank0 can do all_reduce and broadcast the gradients to rank1 and rank2 processes? Does it happen synchronously/asynchronously? Is there any detailed explanation? Thanks in advance.