Thank you!
BTW, does the version 1.0rc1 aleady have the effective overlapping features?
Could I just use the code below instead of DP module to do an overlapping operation?

loss.backward()
dist.all_reduce(flatten_grads)