On the master branch, there is a prototype DDP communication hook feature, which is built for this purpose: https://github.com/pytorch/pytorch/issues/39272 10
In prior releases (<= v1.6), there is no way to turn gradient averaging off without modifying C++ code and recompile.
Update
Synced with Sinan (author of this comm hook feature), this will be reverted due to perf regression. We are investigating.