pytorch : 0.4.1
python : 3.6
conda env
I install from source, and use gloo mode, 4 GPUs in one node, but get error, but use 1 GPU is ok, and nccl mode with 4 gpus is ok too.
error:
terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at /home/zhoushengkai/code/pytorch_v0.4.1_for_whl/third_party/gloo/gloo/transport/ibverbs/pair.cc:462] wc->status == IBV_WC_SUCCESS. 12 vs 0. Memory region send for slot 0: transport retry counter exceeded
terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at /home/zhoushengkai/code/pytorch_v0.4.1_for_whl/third_party/gloo/gloo/transport/ibverbs/pair.cc:462] wc->status == IBV_WC_SUCCESS. 12 vs 0. Memory region send for slot 0: transport retry counter exceeded
terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at /home/zhoushengkai/code/pytorch_v0.4.1_for_whl/third_party/gloo/gloo/transport/ibverbs/pair.cc:462] wc->status == IBV_WC_SUCCESS. 12 vs 0. Memory region send for slot 0: transport retry counter exceeded
terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at /home/zhoushengkai/code/pytorch_v0.4.1_for_whl/third_party/gloo/gloo/transport/ibverbs/pair.cc:462] wc->status == IBV_WC_SUCCESS. 12 vs 0. Memory region send for slot 0: transport retry counter exceeded
done