@mrshenli
I think what  https://github.com/open-mpi/ompi/issues/7733#issuecomment-629806195 3
Suggests is what should be implemented inside Pytorch in cpp, if we want to use MPI process group correctly.
Maybe add optional event argument to torch.dist calls (cuda_event_to_sync_with or something).
As far as I know, callbacks on cuda events are not exposed to python API.
(Too bad they aren’t, actually)