Why does larger num_worker_threads cause the memory leak problem?
When do we need a larger num_worker_threads?


Good question!  cc: @lcw (TensorPipe expert).
On the other hand, can you try ProcessGroupRpcBackendOptions to see if there is memory leak? This backend will be deprecated soon, as it is much slower on GPUs, but you should still be able to use it now.
Example code:
rpc.init_rpc(
    "worker1",
     rank=0,
     world_size=2,
     backend=rpc.BackendType.PROCESS_GROUP,
     rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(
         num_send_recv_threads=16,
         rpc_timeout=20 # 20 second timeout
     )
)