Thanks for your reply - that does seem like a good addition to the code base. By the way: if I would use the torch.distributed primitives, then since nccl is the backend, wouldn’t that transfer through the GPU memory (since nccl does not support CPU ipc)? That might be unconvenient, also for my use case.