I also noticed that when using 2 processes communicating through a Queue,
The sender process (e.g sending from device0 to device1, with copy_) thholds this cuda contex on both devices. I created the buffer at the sender and sent it  through a queue (reusing it) as recommended  in the docs.
(push communication model)
Another option is to have a thread in the receiver waiting on that queue and pulling from it, I guess that won’t cause this extra memory?
So, I wonder, maybe it could be (theoretically, and very partially) solved by creating all tensors in a single process (single owner), and sending them to all processes sharing the device.