Hi,
The version of DP in master should already start doing the all_reduce of each parameter as soon as it’s gradient is computed. Effectively overlapping the rest of the backward and the communication.