You can try ProcessGroupRoundRobin and see if it helps in reducing the communication time. Example usage: https://github.com/pytorch/pytorch/blob/master/test/distributed/test_c10d.py#L1511 55. Note that this API is not officially supported yet.