In your snippet dist.send and dist.recv will race. Since you’re using blocking calls, either path will not continue until it has finished. There must be some interaction between MPI and CUDA that means these calls are now really blocking. You could try explicitly sequencing them and see if that changes anything. In general, it is not encouraged to use Python threading in combination with a single distributed context. It is not written to be thread safe.