Thanks for sharing more details.

I have one more question, how do you handle errors like a worker disconnects from the group?

We are working on elasticity support to allow workers to dynamically join and leave.

As far as I’m concerned, there is just a timeout argument in the rpc api and no explicit error handling descriptions for rpc or collective in the document.

For timeout, there are per-RPC timeout on master branch for rpc_sync and rpc_async. For timeout in remote, the following PR is add it.

  
      github.com/pytorch/pytorch
  
  
    
  
    
  

  
    
      Implement timeout support for RRefs
    

    
      pytorch:gh/rohan-varma/129/base ← pytorch:gh/rohan-varma/129/head
    

    
      
        opened 
        
          
        
        May 16, 2020
      
      

      
        
          
          rohan-varma
        
      

      
        
          +394
          -37
        
      
    

  


  
  
    
    
  
  



no explicit error handling descriptions for rpc or collective in the document

Right, currently, there is only a wait API on the returned Future object. Currently, you will need to try-except the wait. In the C++ implementation, there are error handling APIs on the Future, let us expose it to Python as well. Besides this, do you also need things like error listener for errors occurs in background RPC threads or something else?

Maybe ZeroMQ is a better alternative if we are concerned about robustness?

It depends on what do you need in the application. If you just need P2P communication without tensor/autograd support, then any matured RPC-like system can do. If you need autograd, or would like to do direct device-2-device copy (coming to v1.6), or would like to integrate with TorchScript to speed up training, then torch.distributed.rpc might be better. The latter is a relatively new project and there are still many gaps in the features, but we’d love to drive it together with the community into a better shape.