theoryofjake:

I have read through your tutorials for the parameter server and the DDP example. I have an application that just needs to share model parameters from a GPU process to a different CPU process. Which makes the most sense: a parameter-server type application using the remote calls, DDP, or sending tensors with recv/send?


Not PS: Since the goal is just to pass parameters cross two processes, parameter-server (PS) might be an overkill, as PS usually serves multiple parallel trainers.
Not DDP: Since you need to synchronize parameters, DDP might not be a good fit either, as DDP synchronizes model gradients.
send/recv vs RPC: this depends on how the problem is written. In general, senc/recv is a better fit for single-program multi-data (SPMD) applications, while with RPC, there is usually one driver/master that coordinates all computations in a cluster.

send/recv: The main difference between send/recv and RPC is that when using send/recv both processes need to proceed in the same pace, i.e., when one process calls send, the other one must call recv. If this is how your program is designed, then send/recv should be sufficient (though you still need to convert your model parameters into one tensor and then call send/recv, or calling one send/recv for each parameter).
RPC: When using RPC, you can program the entire logic on the master, and all other processes just block on the rpc.shutdown() call. You don’t need to worry about things like serializing a model, or coordinate multiple processes, etc.