Thanks Can!
OK. looks like it works now. To run on multiple machines, i assume we dont’ need to start an etcd instance first. Just simply set rdzv_endpoint=“IP_ADDRESS” rdzv_backend=“c10d”. And it works. Looks like c10d backend takes care of everything for us.
Looking at the output, I am not sure if parallel part actually happened
(pid=15988, ip=10.231.21.63) {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "95509630884468593764425877023036505025", **"global_rank": 0**, "group_rank": 0, "worker_id": "16109", "role": "default_role", "hostname": "n231-021-063.novalocal", "state": "SUCCEEDED", "total_run_time": 75, "rdzv_backend": "c10d", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"sage_main_routine\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [1]}", "agent_restarts": 0}}
(pid=952664) {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "176607332379713185119918774283739049986", **"global_rank": 0**, "group_rank": 0, "worker_id": "952994", "role": "default_role", "hostname": "n231-013-071.novalocal", "state": "SUCCEEDED", "total_run_time": 76, "rdzv_backend": "c10d", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"sage_main_routine\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [1]}", "agent_restarts": 0}}

After it finishes, I got the output above. Both says it used group_rank as 0, my setup for the run is like this, each node has 0 or 1 as rank respectively:
    os.environ["MASTER_ADDR"] = "10.231.13.71"
    os.environ["MASTER_PORT"] = "2345"
    os.environ["WORLD_SIZE"] = "2"
    os.environ["RANK"] = str(rank)

Maybe I still missed something in the os environment var setting?  Are these four enough?