RuntimeError: Default process group must not be initialized before init_rpc.

As the error suggested, init_process_group cannot be called before init_rpc, as currently DDP and RPC do not work together. We are working on dropping this requirement: https://github.com/pytorch/pytorch/issues/33583 4
For the code snippet above, removing dist.init_process_group(backend, rank=rank, world_size=size) should work.