The second question: since your implementation of rpc is using the FAST & SMART mode algorithm 1, I have also read the detailed PR 2 on your github. I think your workers are stateful, so what will happen if during an execution of the rpc context, some worker fail? like the below scenario in your rpc parameter server example:
       with dist_autograd.context() as cid:
            model_output = net(data)
            target = target.to(model_output.device)
            loss = F.nll_loss(model_output, target)
            if i % 5 == 0:
                print(f"Rank {rank} training batch {i} loss {loss.item()}")
            dist_autograd.backward(cid, [loss])
            # Ensure that dist autograd ran successfully and gradients were
            # returned.
            assert remote_method(
                ParameterServer.get_dist_gradients,
                net.param_server_rref,
                cid) != {}
            opt.step(cid)

what kind of exeptions will be thrown on the if the failure occurs in forward & backward pass?