We have landed gradient compression communication hooks to be used with pytorch DDP in 1.8: DDP Communication Hooks — PyTorch 1.8.1 documentation 10. This seems like it should support your use case quite nicely, as you would be able to run a custom UDF for the gradient reduction instead of the allreduce.