I see. Thanks for sharing!

Maybe add optional event argument to torch.dist calls ( cuda_event_to_sync_with  or something).

I am not sure whether we should add this to c10d Python API if this is only required by the MPI backend. Could you please add an issue on GH to kick off the discussion on the pitch? Let’s discuss there to see what are the options.

As far as I know, callbacks on cuda events are not exposed to python API.
(Too bad they aren’t, actually)

We are actually exploring CUDA event callback for RPC, and also considering using it to handle CUDA errors. Let me create an issue to track this.