shivammehta007:

I think this is the default behaviour of DDP no?


Yep, it’s the default behavior. Just wanna make sure this is how DDP is used in your application so that my comment can be relevant.
Could you please elaborate more on your requirements for gradient clipping? Do you need to do that before DDP gradient averaging comm or after?
If you need to before DDP comm, then you probably can use the DDP communication hook 9. More specifically, you can wrap the gradient bucket clipping with the allreduce communication in the hook.
If it is OK to do clipping after DDP comm, then you can run the clipping ops after DDP backward() and before optimizer step.