hey @Ohm, if you are using DistributedDataParallel, you can try the no_sync 3 context manager. For example, you can wrap local training iterations with no_sync. When you need to do gradient averaging, just run one fw-bw out of the no_sync context, and DDP should be able to take care of the gradient synchronization.
Another option would be building your application using torch.distributed.rpc 1 and then use a parameter server to sync models. See this tutorial 3.
If all parameters are dense, the DDP solution should be more efficient. If there are sparse parameters, the parameter server solution might be faster.