Hey @ankahira, usually, there are 4 steps in distributed data parallel training:

local forward to compute loss
local backward to compute local gradients
allreduce (communication) to compute global gradients. This would be allreduce with SUM + divide by world size to calculate average
optimizer step to use global gradients to update parameters

Both examples you mentioned above conduct the same four steps and are mathematically equivalent. The difference is that DDP would allow step 2 (backward computation) and 3 (allreduce communication) to overlap and therefore DDP is expected to be faster than the average_gradients approach.
More specifically, in the first example with average_gradients, there is a hard barrier between backward and allreduce, i.e., no comm can start before computation finishes. In the second example, DDP organizes gradients into buckets, and will launch comm as soon as a bucket of gradients are ready, so that computation and communication can run in parallel. This 72 would help explain that.
I would recommend DDP.