10.0.87.143: 10.0.92.168: terminate called after throwing an instance of ‘gloo::EnforceNotMet’
10.0.87.143: 10.0.92.168:   what():  [enforce fail at /tmp/pytorch/third_party/gloo/gloo/transport/tcp/address.cc:88] rv != -1. -1 vs -1. getpeername: Transport endpoint is not connected
10.0.87.143: 10.0.87.143: 151 2021-08-25,08:02:49.070 - {distributed_c10d.py (194)} - _store_based_barrier(): Added key: store_based_barrier_key:3 to store for rank: 7
10.0.87.143: 10.0.92.168:     main()
10.0.87.143: 10.0.92.168:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.92.168:     main()
10.0.87.143: 10.0.92.168:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.92.168:     dp_ep.init_ddp()
10.0.87.143: 10.0.92.168:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 68, in init_ddp
10.0.87.143: 10.0.92.168:     self.create_inter_node_all_to_all_process_group()
10.0.87.143: 10.0.92.168:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 150, in create_inter_node_all_to_all_process_group
10.0.87.143: 10.0.92.168:     backend=Backend.GLOO
10.0.87.143: 10.0.92.168:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 2700, in new_group
10.0.87.143: 10.0.92.168:     timeout=timeout)
10.0.87.143: 10.0.92.168:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 620, in _new_process_group_helper
10.0.87.143: 10.0.92.168:     timeout=timeout)
10.0.87.143: 10.0.95.41:     main()
10.0.87.143: 10.0.95.41:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.95.41:     main()
10.0.87.143: 10.0.95.41:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.95.41:     dp_ep.init_ddp()
10.0.87.143: 10.0.95.41:     main()
10.0.87.143: 10.0.95.41:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 68, in init_ddp
10.0.87.143: 10.0.92.168: RuntimeError: [/tmp/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [10.0.87.143]:57754: Connection refused
10.0.87.143: 10.0.95.41:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.95.41:     main()
10.0.87.143: 10.0.95.41:     main()
10.0.87.143: 10.0.95.41:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.95.41:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.95.41:     self.create_inter_node_all_to_all_process_group()
10.0.87.143: 10.0.95.41:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 150, in create_inter_node_all_to_all_process_group
10.0.87.143: 10.0.95.41:     main()
10.0.87.143: 10.0.95.41:   File “main_moe.py”, line 108, in main
10.0.87.143: 10.0.95.41:     backend=Backend.GLOO
10.0.87.143: 10.0.95.41:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 2700, in new_group
10.0.87.143: 10.0.92.168:     dp_ep.init_ddp()
10.0.87.143: 10.0.92.168:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 68, in init_ddp
10.0.87.143: 10.0.92.168:     self.create_inter_node_all_to_all_process_group()
10.0.87.143: 10.0.92.168:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 150, in create_inter_node_all_to_all_process_group
10.0.87.143: 10.0.92.168:     backend=Backend.GLOO
10.0.87.143: 10.0.92.168:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 2700, in new_group
10.0.87.143: 10.0.92.168:     timeout=timeout)
10.0.87.143: 10.0.92.168:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 620, in _new_process_group_helper
10.0.87.143: 10.0.95.41:     timeout=timeout)
10.0.87.143: 10.0.95.41:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 620, in _new_process_group_helper
10.0.87.143: 10.0.95.41:     timeout=timeout)
10.0.87.143: 10.0.95.41: RuntimeError: [/tmp/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [10.0.87.143]:31117: Connection reset by peer
10.0.87.143: 10.0.92.168:     timeout=timeout)
10.0.87.143: 10.0.92.168: RuntimeError: [/tmp/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [10.0.87.143]:59302: Connection reset by peer
10.0.87.143: 10.0.95.41:     dp_ep.init_ddp()
10.0.87.143: 10.0.95.41:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 68, in init_ddp
10.0.87.143: 10.0.95.41:     self.create_inter_node_all_to_all_process_group()
10.0.87.143: 10.0.95.41:   File “/home/deepspeed/.local/lib/python3.6/site-packages/moe_layer/dp/dp_manager.py”, line 150, in create_inter_node_all_to_all_process_group
10.0.87.143: 10.0.95.41:     backend=Backend.GLOO
10.0.87.143: 10.0.95.41:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 2700, in new_group
10.0.87.143: 10.0.95.41:     timeout=timeout)
10.0.87.143: 10.0.95.41:   File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 620, in _new_process_group_helper
10.0.87.143: 10.0.95.41:     timeout=timeout)
10.0.87.143: 10.0.95.41: RuntimeError: [/tmp/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [10.0.87.143]:31117: Connection refused