I didn’t say it, but I meant sharing cuda tensors.
(e.g: doing something like Hogwild example- but having the model on GPU instead of CPU and share its parameters with 2 MPI process on the same node)
(For CPU tensors, your answer is correct, I read once a blog post demonstrating it, and all we need to modify is using shared MPI mem instead).
However, my 2 cents on this: it works out of the box only for host shared memory. Somehow couldn’t create shared+pinned host memory with pytorch easily. (I managed to do it with ugly hack: os.fork. I didn’t bother to make it work for MPI too)