@iffiX A lot of interesting stuff here, let me get to it in order.
Your points #1 (failure of a node) and #2 (failure of a link) are indeed handled poorly by the ProcessGroup backend as they bring all other nodes and links down with them. The TensorPipe backend is more resilient and only fails the affected requests. At this stage, a failed link will remain failed forever (although we’re discussing how to improve this in https://github.com/pytorch/pytorch/issues/40936, please chime in there). A failed node is harder to deal with, as nodes can be stateful and thus restarting it doesn’t mean that the other nodes can resume talking to it as if nothing had happened. The current approach chosen by Elastic is to restart all nodes when one fails. That’s drastic but safe. Also, even in the TensorPipe backend, we currently have nodes exchange their addresses at the beginning, so if a node (re-)joins later it won’t be able to make its address known to the other ones.
The other point where the TensorPipe backend performs a collective operation among all nodes is during graceful shutdown. It is an intrinsic requirement of the API that this method operates like a barrier. Currently that is implemented using process groups and thus suffers from their problems, but even if we reimplemented it on top of TensorPipe it wouldn’t be much different: a failed link/node would cause all others to fail too. I don’t see a way around it while still upholding that requirement. However, if you have your own logic on the nodes’ lifetimes, you probably can do an ungraceful shutdown and thus avoid this whole problem entirely: in that case the TensorPipe backend never uses its process group so it doesn’t matter if it fails.
As for your point #3, to me it looks like we’re raising an exception in a function marked as noexcept. If that’s the case it should be an easy fix, but it would greatly help if you could provide a C++ stack trace. Just attach GDB to the process, set up exception catching (catch throw) and then let the process resume as usual (continue). When GDB then catches an exception you can get the stack trace through backtrace. There may be harmless exceptions that are fired before that last fatal one, so please make sure you get the right one. It would also help if your build has debug symbols, I don’t know if the default PyTorch one does.
Point #4 is particularly tricky, and also relates to the ones above. Dealing with exceptions is hard because, in principle, the RPC module is available to both C++ and Python, which have different exception types and thus it’s hard to preserve and convert between them. Moreover, the future objects have an error state which only contains a string message (see here), so there is no way to store a “error type” in there. The way the RPC module currently propagates the type from the remote UDF to the client is a bit hacky: it involves marking the future as successful, with a result value that contains the exception type and message, and then use an “unwrap” function that fires when accessing that value that raises it instead of returning it (see here). So it means that an errored UDF will return a successful future, but from Python there’s no way to distinguish them. A real failed future (which you get from example in case of an I/O error) will instead always raise a RuntimeError just because it has no way of specifying anything else.
All these problems are in principle solvable (storing a Python exception in C++ as pybind11::error_already_set, converting a C++ exception to Python using pybind11), but this requires changes to the future class which is a top-level util of PyTorch used also by other modules (like JIT) and that makes it hard to change it.