I went through paper quickly so correct me if I am wrong. They are not implementing a network communication process, they are not implementing a new communication semantic, they are pre-processing gradients before reducing them. If you are using DistributedDataParallel, you can pre-process gradients before doing backward propagation. If you want to also change distribution behavior like skipping gradient sharing every two epoch etc., you can implement DistributedDataParallel yourself by following distributed tutorial 3 with operations like  torch.distributed.reduce_op.