I tried the following two implementations. The first one uses mp.Process to spawn processes, and with the second one, I ran python test.py 0/1/2 in three different terminal tabs. Both work for me. distributed autograd
if __name__=="__main__":
    world_size = 3
    processes = []

    for rank in range(3):
        p = mp.Process(target=workers_init_rpc, args=(rank, world_size))
        p.start()
        processes.append(p)

    [p.join() for p in processes]

if __name__=="__main__":
    world_size = 3

    rank = int(sys.argv[1])
    workers_init_rpc(rank, world_size)

Is the code you shared above exactly the same where you hit the hang problem? There is a known gap in distributed autograd. We currently only support fast mode distributed autograd, which means all RPC comm operations (rpc_sync, rpc_async, remote) must participate in the backward, otherwise the backward would hang. See more details in the doc below.
https://pytorch.org/docs/stable/rpc/distributed_autograd.html 1