Is it a good practice to create a new group on every training iteration?
Using dist.new_group requires every process to pass through the function even if they are not a part of the distributed training process. Sometimes it hangs up on this function for a reason that I am not aware of. I was wondering has anybody come up with a better solution for this?
Thanks!