Hey @stclaireva

Is that possible to do that with torch.distributed package?

This is possible. You can

Run forward-backward locally.
Use all_gather 1 or all_gather_coalesced 1 to collect all gradients into rank 0.
Manually add/average those gradients into param.grad on rank 0.
Run optimizer.step() on rank 0 to update parameters.
Let rank 0 broadcast the updated parameters to other ranks.
go to step 1 to start a new iteration

Curious, why do you need the above algorithm instead of letting DistributedDataParallel handle it for you?