matrix:

Furthermore, node with 1 GPU acts as a master, it encompasses these 2 parts of the model into one. Calls them (using rpc_sync) sequentially to run a full forward pass. Also, this dividing work (copying modules to GPUs) is being done in __init__ (think of Net1 and Net2 as 2 parts of the model) methods, could this be a problem when using mp.spawn?


Sorry about the delay. If rpc_sync succeeded in your environment, then it mean at least the comm layer is working. So if the backward hangs, I would assume it’s sth wrong with the backward instead of mp.spawn. But let me try Process instead of spawn anyway.