iffiX:

It’s really sad to hear that currently pytorch rpc cannot handle the 1st and 2nd condition, since that’s what my application code is designed to do.


Could you please elaborate more on the requirement? We understand it is important to provide failure-recovery + elasticity, but haven’t seen people explicitly requesting this yet, so that didn’t get into our top-priority in past releases. If this is a commonly required feature that might block many use cases, we will reprioritize work items and try to get this done sooner.

Anyway, thanks for your response! I still want to ask: what kind of P2P mechanism are you going to provide in the TensorPipe backend? could you please clarify your and your collegues’ plans a little bit more?

This will be released in v1.6 very soon. TensorPipe no longer requires rendezvous or full participation from all processes. So technically, crashed processes do not prevent the rest processes to function correctly (not yet so in v1.6). And it should also be faster than ProcessGroup-based RPC backend. Eventually, we will retire the ProcessGroup-based RPC backend, and make TensorPipe RPC backend as the default option.
cc the main contributor of TensorPipe @lcw