iffiX:

I think your workers are stateful , so what will happen if during an execution of the rpc context, some worker fail?


This is true, they are stateful, and states including RRef context, distributed autograd context and application states. In the current version, an RPC gang cannot survive any node crash. We are actively collaborating with torchelastic to provide fault-tolerance and elasticity support.

what kind of exeptions will be thrown on the if the failure occurs in forward & backward pass?

If the error is recoverable, e.g., just an exception instead of hard node crash, RPC should throw the same exception (or a RuntimeError, this is not great, still working on improvements) on the caller. Below are some example tests:

  
      github.com
  
  
    pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/dist_autograd_test.py#L1204-L1210

def test_backward_autograd_engine_error(self):
    with dist_autograd.context() as context_id:
        t1 = torch.rand((3, 3), requires_grad=True)
        t2 = torch.rand((3, 3), requires_grad=True)
        # Perform some ops before error simulation.
        tmp = (t1 + t2) * (t1 + t2)
        t3 = SimulateBackwardError.apply(tmp)



  
  
    
    
  
  



  
      github.com
  
  
    pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/rpc_test.py#L1284-L1290 2

@dist_init
def test_py_raise_in_user_func(self):
    n = self.rank + 1
    dst_rank = n % self.world_size
    fut = rpc.rpc_async(worker_name(dst_rank), raise_func)
    with self.assertRaises(ValueError):
        fut.wait()