I’m currently implementing a heterogeneity-aware distributed model.
My basic idea is to do all-reduce on a subset of the fast workers in the world group, however I noticed that in torch.distributed. new_group, it says:

This function requires that all processes in the main group (i.e. all processes that are part of the distributed job) enter this function, even if they are not going to be members of the group.

Does that mean that the fast workers should wait for the slow workers before they can move forward (i.e. the slower one will sync the faster one)?
If so, is there any other way to implement the model?