I was referring to interprocess syncs. Are there any other sync points besides the ddp constructor, forward, backward and collective communication calls? And does calling torch.cuda.synchronize(rank) in a specific process after issuing an async collective communication call (for example an all_reduce) block till the result of the collective communication call is available?
Also since we’re on the topic, are there plans to add sparse all_reduce capabilities to the NCCL backend?