I get some bug during torch.distributed.isend and irecv: I think that there is a race condition there, but not sure how to debug this.
I’m using MPI, so the buffers have to stay untouched until the send/recv is over.
I see crazy “spikes” in error, which I get only with a certain degree of parallelism.
I wonder if pytorch/python garbage collector touches my buffers.
I saved them in a list, just in case.
Where can I check this in code? can I “guard” the buffers somehow?
I see pytorch tests barely check the Isend/Irecv,  and want to verify that the bug is not internal…