Tried this locally, it hangs with a non-deterministic number of iterations (4, 11, etc.), and it hangs at del g_tensor. I suspect it is due to CUDACachingAllocator, where it might need to keep the memory block alive until all other processes finish using it? But it is a per-process data structure, so it does not have the global view.  I am not sure if this is the reason for this hang.
Call for help cc @ptrblck @colesbury