Hey @kmaeng, there are actually RPC activity for graceful RPC shutdown. See the code below. It’s basically using an RPC to prevent the idle process from exiting too early.

  
      github.com
  
  
    pytorch/pytorch/blob/9d0e935b489a33b87711b9d5d3525a7282ab89c1/torch/distributed/rpc/api.py#L133-L192 3

@_require_initialized
def _wait_all_workers():
    r"""
    Block until all local and remote RPC processes reach this method and wait
    for all outstanding work to complete. Every RPC process must call this
    method before exit to perform a graceful shutdown. This should be used to
    terminate the RPC framework, and there is no guarantee that the RPC
    framework will work after this method returns.
    """
    assert (
        _ALL_WORKER_NAMES is not None
    ), "`_ALL_WORKER_NAMES` is not initialized for `def _wait_all_workers`."
    leader_worker_name = sorted(_ALL_WORKER_NAMES)[0]

    self_worker_name = _get_current_rpc_agent().get_worker_info().name

    global _wait_all_workers_sequence_id
    with _wait_all_workers_dict_lock:
        sequence_id = _wait_all_workers_sequence_id
        _wait_all_workers_sequence_id += 1


  This file has been truncated. show original

  
  
    
    
  
  


To get around this, you can increase the default RPC timeout. Depending on the version you are using, you can either provide the timeout value in init_rpc 8 (with v1.5), or directly call rpc._set_rpc_timeout 7 (with v1.4).
Or if you know for sure when a process can safely exit, you can use shutdown(graceful=False) and do the termination detection in application code.
Per op (rpc_sync/rpc_async/remote) timeout are coming soon.