Hey @iffiX


during execution, the target process crashes and exits, also closing down all rpc execution threads.
during execution, connection to the target process is closed


Prior to v1.6, ProcessGroup is the only available backend, which requires all processes to be alive. So RPC gang cannot survive these failures. Even if you can catch it in application code, it will leave subsequent RPC behaviors in an undefined state, unless there is a global recovery process (we do have plans to providing this).
We will introduce TensorPipe 1 backend for RPC in v1.6, which is a P2P comm library. But in the first experimental version, it still has some part depends on ProcessGroup, so I think it still wouldn’t tolerate such failures in v1.6. There is also an ongoing project to provide elasticity to RPC.
cc @lcw @agolynski


during execution, the timeout limit is reached


This should thrown a RuntimeError for the timeout. And try-except on RuntimeError type should be able to catch it. See the code below:

  
      github.com
  
  
    pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L2501-L2502 1

with self.assertRaisesRegex(RuntimeError, error_str):
    fut.wait()



  
  
    
    
  
  


If this is not so in your application, then it is a bug that we need to fix. Please let us know how we can reproduce it. Or if you have other suggestions on how we should report timeout error, please also let us know your suggestions.
cc @rvarm1


during execution, an exception is raised in the executed function


If there is an exception during remote execution, this should be thrown on the caller side. See the code below:

  
      github.com
  
  
    pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L1625-L1631

@dist_init
def test_py_raise_in_user_func(self):
    n = self.rank + 1
    dst_rank = n % self.world_size
    fut = rpc.rpc_async(worker_name(dst_rank), raise_func)
    with self.assertRaises(ValueError):
        fut.wait()



  
  
    
    
  
  


This is how the Python error is captured:

  
      github.com
  
  
    pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/distributed/rpc/internal.py#L152-L171

def _run_function(python_udf):
    r"""
    This function is exclusively called from C++.
    See ``torch/csrc/distributed/rpc/python_rpc_handler.cpp``.

    Runs a Python UDF and returns its return value.
    Wraps any exception in ``RemoteException`` if the function raises.
    """
    try:
        if isinstance(python_udf, AttributeError):
            raise python_udf
        result = python_udf.func(*python_udf.args, **python_udf.kwargs)
    except Exception as e:
        # except str = exception info + traceback string
        except_str = (
            f"On {_get_current_rpc_agent().get_worker_info()}:\n"
            f"{repr(e)}\n{traceback.format_exc()}"
        )
        result = RemoteException(except_str, type(e))
    return result



  
  
    
    
  
  


This is how non-user error is captured:

  
      github.com
  
  
    pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/csrc/distributed/rpc/request_callback_impl.cpp#L907-L921

try {
  processRpc(*rpc, messageType, id, retFuture);
} catch (py::error_already_set& e) {
  retFuture->markCompleted(handleError(e, messageType, id));
  // There are request callback impls in Python, where Python
  // exceptions could be thrown. For releasing Python exception
  // py::objects, GIL must be held.
  py::gil_scoped_acquire acquire;
  e.restore(); // Release ownership on py::objects and also restore
               // Python Error Indicator.
  PyErr_Clear(); // Clear the Python Error Indicator as we has
                 // recorded the exception in the response message.
} catch (std::exception& e) {
  retFuture->markCompleted(handleError(e, messageType, id));
}



  
  
    
    
  
  


Please let us know if we missed any. Thanks!