I see, sorry that I misread the original question.

In summary, is it normal to see the error message I posted if I call init_rpc and don’t use it for 30 minutes?

This is indeed a bug, and I think it is due to the following code, where the ProcessGroup RPC agent’s recvAnysource timed out. We should have passed rpc timeout to process group or set it to infinity in the listen loop.

  
      github.com
  
  
    pytorch/pytorch/blob/1f87f15ba3cd46fe5474f7783ff80ef06950e157/torch/csrc/distributed/rpc/process_group_agent.cpp#L710-L715 2

void ProcessGroupAgent::listenLoopInternal() {
  while (rpcAgentRunning_.load()) {
    // rank, tensor size, message type
    std::vector<torch::Tensor> preamble = {torch::empty({4}, {torch::kInt64})};
    auto work = pg_->recvAnysource(preamble, pg_->getRank());
    {



  
  
    
    
  
  


Thanks for flagging this, do you want to create an issue on github to track this? We will fix. Thanks!