@wayi Thanks for answering! I guess your first option is what I need but I have several questions about it.


Documentation on RemoteModule says RemoteModule is not currently supported when using CUDA tensors, but you said tensors will be automatically placed to the same cuda device. Am I missing something? If CUDA tensors are not supported now, where can I track progress on this?


I still need to spawn a remote pytorch process manually every time I start my local process, right? Is there a solution to create a long-living remote process that will consume messages from different local processes?


If not, I can automate my local build process to do something like
ssh remote-host 'cd proj-dir; python remote-worker.py';
python train.py

It’s not very elegant but should work.