text: Title: Allredue hang in yolo3 train using  multi machine 
  Body: **Environment:**
  1. Framework: MXNet
  2. Framework version:1.6.0.post0
  3. Horovod version:0.22.0
  4. MPI version:3.1.0
  5. CUDA version:10.2
  6. NCCL version:2.9.6
  7. Python version:3.6
  8. Spark / PySpark version:
  9. Ray version:
  10. OS and version: Ubuntu 18.04.5 LTS \n \l
  11. GCC version:7.5.0
  12. CMake version:3.20.2
  
  **Checklist:**
  1. Did you search issues to find if somebody asked this question before?
  2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
  3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
  4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?
  
  **Bug report:**
  Please describe erroneous behavior you're observing and steps to reproduce it.
  
  
  Hi, all
  
  I got warning like this. 
  When I am using 2 node and 2 GPUs for distributed training, there are some infos following:
  
  envs:
  ```
  docker image : gluonai/gluon-cv:gpu-latest
  python:3.6
  mxnet-cu102:1.6.0.post0
  horovod:0.22.0
  ```
  
  run_multi_node.sh in gluon-cv dirs is:
  
  ```
  export CUDA_VISIBLE_DEVICES=2
  export NCCL_DEBUG=info
  export NCCL_SOCKET_IFNAME=eth0
  export NCCL_IB_DISABLE=1
  
  rm -rf log
  mkdir log
  export MXNET_CUDNN_AUTOTUNE_DEFAULT=0
  export MXNET_EXEC_ENABLE_ADDTO=1
  python3 ./scripts/detection/yolo/train_yolo3.py \
        --network darknet53 \
        --dataset=coco \
        --batch-size=4 \
        --horovod \
        --num-workers 8 \
        --log-interval 10 \
        --lr-decay-epoch 220,250 \
        --epochs 280 \
        --warmup-epochs 2 \
        --mixup \
        --no-mixup-epochs 20 \
        --label-smooth --no-wd \
        --save-interval 1 \
        --val-interval 1 \
        --syncbn \
        --save-prefix log/
  ```
  and logs is:
  ```
  root@yq01-sys-hic-v100-box-a223-0155:/home/users/liuyuhui/workspace/gluon-cv# /usr/bin/mpirun bash run_multi_node.sh
  Mon May 24 06:42:09 2021[1,1]<stdout>:loading annotations into memory...
  Mon May 24 06:42:09 2021[1,0]<stdout>:loading annotations into memory...
  Mon May 24 06:42:25 2021[1,0]<stdout>:Done (t=16.53s)
  Mon May 24 06:42:25 2021[1,0]<stdout>:creating index...
  Mon May 24 06:42:26 2021[1,0]<stdout>:index created!
  Mon May 24 06:42:27 2021[1,1]<stdout>:Done (t=17.78s)
  Mon May 24 06:42:27 2021[1,1]<stdout>:creating index...
  Mon May 24 06:42:27 2021[1,1]<stdout>:index created!
  Mon May 24 06:42:51 2021[1,0]<stdout>:loading annotations into memory...
  Mon May 24 06:42:52 2021[1,0]<stdout>:Done (t=0.44s)
  Mon May 24 06:42:52 2021[1,0]<stdout>:creating index...
  Mon May 24 06:42:52 2021[1,0]<stdout>:index created!
  Mon May 24 06:42:52 2021[1,1]<stdout>:loading annotations into memory...
  Mon May 24 06:42:52 2021[1,1]<stdout>:Done (t=0.44s)
  Mon May 24 06:42:52 2021[1,1]<stdout>:creating index...
  Mon May 24 06:42:52 2021[1,1]<stdout>:index created!
  Mon May 24 06:43:03 2021[1,0]<stderr>:INFO:root:Namespace(amp=False, batch_size=4, data_shape=416, dataset='coco', epochs=280, gpus='0', horovod=True, label_smooth=True, log_interval=10, lr=0.001, lr_decay=0.1, lr_decay_epoch='220,250', lr_decay_period=0, lr_mode='step', mixup=True, momentum=0.9, network='darknet53', no_mixup_epochs=20, no_random_shape=False, no_wd=True, num_samples=117266, num_workers=8, resume='', save_interval=1, save_prefix='log/yolo3_darknet53_coco', seed=233, start_epoch=0, syncbn=True, val_interval=1, warmup_epochs=2, warmup_lr=0.0, wd=0.0005)
  Mon May 24 06:43:03 2021[1,0]<stderr>:INFO:root:Start training from [Epoch 0]
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Bootstrap : Using eth0:10.255.100.13<0>
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO NET/Socket : Using [0]eth0:10.255.100.13<0>
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Using network Socket
  Mon May 24 06:43:04 2021[1,0]<stdout>:NCCL version 2.9.6+cuda10.2
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Bootstrap : Using xgbe0:10.127.28.15<0>
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO NET/Socket : Using [0]xgbe0:10.127.28.15<0>
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Using network Socket
  Mon May 24 06:43:04 2021[1,1]<stderr>:INFO:root:Namespace(amp=False, batch_size=4, data_shape=416, dataset='coco', epochs=280, gpus='0', horovod=True, label_smooth=True, log_interval=10, lr=0.001, lr_decay=0.1, lr_decay_epoch='220,250', lr_decay_period=0, lr_mode='step', mixup=True, momentum=0.9, network='darknet53', no_mixup_epochs=20, no_random_shape=False, no_wd=True, num_samples=117266, num_workers=8, resume='', save_interval=1, save_prefix='log/yolo3_darknet53_coco', seed=233, start_epoch=0, syncbn=True, val_interval=1, warmup_epochs=2, warmup_lr=0.0, wd=0.0005)
  Mon May 24 06:43:04 2021[1,1]<stderr>:INFO:root:Start training from [Epoch 0]
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Channel 00/02 :    0   1
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Channel 01/02 :    0   1
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Setting affinity for GPU 2 to 0fffff
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Setting affinity for GPU 2 to ffffff
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Channel 00 : 1[41000] -> 0[42000] [receive] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Channel 00 : 0[42000] -> 1[41000] [receive] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Channel 01 : 1[41000] -> 0[42000] [receive] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Channel 01 : 0[42000] -> 1[41000] [receive] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Channel 00 : 0[42000] -> 1[41000] [send] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Channel 00 : 1[41000] -> 0[42000] [send] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Channel 01 : 0[42000] -> 1[41000] [send] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Channel 01 : 1[41000] -> 0[42000] [send] via NET/Socket/0
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Connected all rings
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO Connected all trees
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
  Mon May 24 06:43:04 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:5377:5532 [0] NCCL INFO comm 0x7f440c35e630 rank 1 nranks 2 cudaDev 0 busId 41000 - Init COMPLETE
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Connected all rings
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Connected all trees
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO comm 0x7f9f9c37d480 rank 0 nranks 2 cudaDev 0 busId 42000 - Init COMPLETE
  Mon May 24 06:43:04 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:5328:5459 [0] NCCL INFO Launch mode Parallel
  Mon May 24 06:43:09 2021[1,0]<stderr>:INFO:root:[Epoch 0][Batch 9], LR: 1.71E-07, Speed: 19.202 samples/sec, ObjLoss=6658.178, BoxCenterLoss=20.613, BoxScaleLoss=24.689, ClassLoss=435.474
  Mon May 24 06:43:12 2021[1,0]<stderr>:INFO:root:[Epoch 0][Batch 19], LR: 3.41E-07, Speed: 17.696 samples/sec, ObjLoss=7650.106, BoxCenterLoss=16.439, BoxScaleLoss=18.395, ClassLoss=339.499
  Mon May 24 06:43:14 2021[1,0]<stderr>:INFO:root:[Epoch 0][Batch 29], LR: 5.12E-07, Speed: 19.079 samples/sec, ObjLoss=6205.702, BoxCenterLoss=18.171, BoxScaleLoss=19.766, ClassLoss=378.784
  Mon May 24 06:43:17 2021[1,0]<stderr>:INFO:root:[Epoch 0][Batch 39], LR: 6.82E-07, Speed: 9.979 samples/sec, ObjLoss=5282.987, BoxCenterLoss=17.172, BoxScaleLoss=18.486, ClassLoss=358.852
  Mon May 24 06:47:08 2021[1,0]<stderr>:[Mon May 24 06:47:08 2021[1,0]<stderr>:2021-05-24 06:47:08.577207: W /tmp/pip-install-8i6hkcw7/horovod_78663a94386f4ebabc36dedd9415f5c2/horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock.
  Mon May 24 06:47:08 2021[1,0]<stderr>:Missing ranks:
  Mon May 24 06:47:08 2021[1,0]<stderr>:1: [horovod_allreduce.0, horovod_allreduce.1, horovod_allreduce.100, horovod_allreduce.101, horovod_allreduce.104, horovod_allreduce.105 ...]
  ```
  It seems that problems happen in allreduce time out，but it's ok when I running demo horovod provided  in multi machine.
  
  run.sh as following:
  ```
  export NCCL_DEBUG=info
  export NCCL_SOCKET_IFNAME=eth0
  export NCCL_IB_DISABLE=1
  
  python3 mxnet_mnist.py
  ```
  ```
  root@yq01-sys-hic-v100-box-a223-0155:/home/users/liuyuhui/workspace/horovod/examples/mxnet# /usr/bin/mpirun bash run.sh
  Mon May 24 07:12:45 2021[1,1]<stderr>:INFO:root:Namespace(batch_size=64, dtype='float32', epochs=5, gradient_predivide_factor=1.0, lr=0.01, momentum=0.9, no_cuda=False)
  Mon May 24 07:12:45 2021[1,0]<stderr>:INFO:root:Namespace(batch_size=64, dtype='float32', epochs=5, gradient_predivide_factor=1.0, lr=0.01, momentum=0.9, no_cuda=False)
  Mon May 24 07:12:45 2021[1,0]<stderr>:INFO:root:data-0/mnist.zip exists, skipping download
  Mon May 24 07:12:45 2021[1,1]<stderr>:INFO:root:data-1/mnist.zip exists, skipping download
  Mon May 24 07:12:46 2021[1,1]<stderr>:[07:12:46] src/io/iter_mnist.cc:Mon May 24 07:12:46 2021[1,1]<stderr>:113: MNISTIter: load 30000 images, shuffle=1, shape=[64,1,28,28]
  Mon May 24 07:12:46 2021[1,0]<stderr>:[07:12:46Mon May 24 07:12:46 2021[1,0]<stderr>:] src/io/iter_mnist.cc:113: MNISTIter: load 30000 images, shuffle=1, shape=[64,1,28,28]
  Mon May 24 07:12:46 2021[1,1]<stderr>:[07:12:46] src/io/iter_mnist.cc:113: MNISTIter: load 10000 images, shuffle=1, shape=[Mon May 24 07:12:46 2021[1,1]<stderr>:64,1,28,28]
  Mon May 24 07:12:47 2021[1,0]<stderr>:[07:12:47] src/io/iter_mnist.cc:113: MNISTIter: load 10000 images, shuffle=Mon May 24 07:12:47 2021[1,0]<stderr>:1, shape=[64,1,28,28]
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Bootstrap : Using eth0:10.255.100.13<0>
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO NET/Socket : Using [0]eth0:10.255.100.13<0>
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Using network Socket
  Mon May 24 07:12:50 2021[1,0]<stdout>:NCCL version 2.9.6+cuda10.2
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Bootstrap : Using xgbe0:10.127.28.15<0>
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO NET/Socket : Using [0]xgbe0:10.127.28.15<0>
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Using network Socket
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 00/02 :    0   1
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 01/02 :    0   1
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Setting affinity for GPU 0 to ffffff
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 00 : 1[3f000] -> 0[40000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 00 : 0[40000] -> 1[3f000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 01 : 1[3f000] -> 0[40000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 01 : 0[40000] -> 1[3f000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 00 : 0[40000] -> 1[3f000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 00 : 1[3f000] -> 0[40000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 01 : 0[40000] -> 1[3f000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 01 : 1[3f000] -> 0[40000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Connected all rings
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Connected all trees
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Connected all rings
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO comm 0x7ff8d8346190 rank 0 nranks 2 cudaDev 0 busId 40000 - Init COMPLETE
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Connected all trees
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO comm 0x7f22c834adb0 rank 1 nranks 2 cudaDev 0 busId 3f000 - Init COMPLETE
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Launch mode Parallel
  Mon May 24 07:12:50 2021[1,0]<stderr>:[07:12:50] Mon May 24 07:12:50 2021[1,0]<stderr>:src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
  Mon May 24 07:12:50 2021[1,1]<stderr>:[07:12:50] Mon May 24 07:12:50 2021[1,1]<stderr>:src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
  Mon May 24 07:12:51 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 100] Training: accuracy=0.867500
  Mon May 24 07:12:51 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 100] Training: accuracy=0.865469
  Mon May 24 07:12:51 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 200] Training: accuracy=0.914297
  Mon May 24 07:12:51 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 200] Training: accuracy=0.915547
  Mon May 24 07:12:51 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 300] Training: accuracy=0.934219
  Mon May 24 07:12:51 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 300] Training: accuracy=0.934010
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 400] Training: accuracy=0.944688
  Mon May 24 07:12:52 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 400] Training: accuracy=0.944688
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:Epoch[0]	Speed=28557.88 samples/s	Time cost=2.097635
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:Epoch[0]	Train: accuracy=0.949753	Validation: accuracy=0.983373
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 100] Training: accuracy=0.984375
  Mon May 24 07:12:52 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 100] Training: accuracy=0.981094
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 200] Training: accuracy=0.985078
  Mon May 24 07:12:53 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 200] Training: accuracy=0.981016
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 300] Training: accuracy=0.985000
  Mon May 24 07:12:53 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 300] Training: accuracy=0.981563
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 400] Training: accuracy=0.985117
  Mon May 24 07:12:53 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 400] Training: accuracy=0.982500
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:Epoch[1]	Speed=40224.99 samples/s	Time cost=1.489223
  Mon May 24 07:12:54 2021[1,0]<stderr>:INFO:root:Epoch[1]	Train: accuracy=0.985544	Validation: accuracy=0.986378
  Mon May 24 07:12:54 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 100] Training: accuracy=0.990469
  Mon May 24 07:12:54 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 100] Training: accuracy=0.989219
  Mon May 24 07:12:54 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 200] Training: accuracy=0.990703
  Mon May 24 07:12:54 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 200] Training: accuracy=0.988047
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 300] Training: accuracy=0.990417
  Mon May 24 07:12:55 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 300] Training: accuracy=0.988229
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 400] Training: accuracy=0.990156
  Mon May 24 07:12:55 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 400] Training: accuracy=0.988711
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:Epoch[2]	Speed=40484.51 samples/s	Time cost=1.479677
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:Epoch[2]	Train: accuracy=0.990151	Validation: accuracy=0.988482
  Mon May 24 07:12:55 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 100] Training: accuracy=0.993594
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 100] Training: accuracy=0.993437
  Mon May 24 07:12:56 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 200] Training: accuracy=0.992266
  Mon May 24 07:12:56 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 200] Training: accuracy=0.993359
  Mon May 24 07:12:56 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 300] Training: accuracy=0.993229
  Mon May 24 07:12:56 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 300] Training: accuracy=0.992344
  Mon May 24 07:12:56 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 400] Training: accuracy=0.993203
  Mon May 24 07:12:56 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 400] Training: accuracy=0.992500
  Mon May 24 07:12:57 2021[1,0]<stderr>:INFO:root:Epoch[3]	Speed=34536.84 samples/s	Time cost=1.734496
  Mon May 24 07:12:57 2021[1,0]<stderr>:INFO:root:Epoch[3]	Train: accuracy=0.993356	Validation: accuracy=0.989083
  Mon May 24 07:12:57 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 100] Training: accuracy=0.996094
  Mon May 24 07:12:57 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 100] Training: accuracy=0.995469
  Mon May 24 07:12:58 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 200] Training: accuracy=0.995547
  Mon May 24 07:12:58 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 200] Training: accuracy=0.994531
  Mon May 24 07:12:58 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 300] Training: accuracy=0.995469
  Mon May 24 07:12:58 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 300] Training: accuracy=0.994375
  Mon May 24 07:12:58 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 400] Training: accuracy=0.995273
  Mon May 24 07:12:58 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 400] Training: accuracy=0.994609
  Mon May 24 07:12:59 2021[1,0]<stderr>:INFO:root:Epoch[4]	Speed=34617.01 samples/s	Time cost=1.730479
  Mon May 24 07:12:59 2021[1,0]<stderr>:INFO:root:Epoch[4]	Train: accuracy=0.995259	Validation: accuracy=0.990084
  root@yq01-sys-hic-v100-box-a223-0155:/home/users/liuyuhui/workspace/horovod/examples/mxnet#
  ```
  It seems that allreduce time out  using multi machine, but I got OK when I train demo  provided by horovod.
  there are run.sh as following:
  
  ```
  export NCCL_DEBUG=info
  export NCCL_SOCKET_IFNAME=eth0
  export NCCL_IB_DISABLE=1
  
  python3 mxnet_mnist.py
  ```
  
  ```
  root@yq01-sys-hic-v100-box-a223-0155:/home/users/liuyuhui/workspace/horovod/examples/mxnet# /usr/bin/mpirun bash run.sh
  Mon May 24 07:12:45 2021[1,1]<stderr>:INFO:root:Namespace(batch_size=64, dtype='float32', epochs=5, gradient_predivide_factor=1.0, lr=0.01, momentum=0.9, no_cuda=False)
  Mon May 24 07:12:45 2021[1,0]<stderr>:INFO:root:Namespace(batch_size=64, dtype='float32', epochs=5, gradient_predivide_factor=1.0, lr=0.01, momentum=0.9, no_cuda=False)
  Mon May 24 07:12:45 2021[1,0]<stderr>:INFO:root:data-0/mnist.zip exists, skipping download
  Mon May 24 07:12:45 2021[1,1]<stderr>:INFO:root:data-1/mnist.zip exists, skipping download
  Mon May 24 07:12:46 2021[1,1]<stderr>:[07:12:46] src/io/iter_mnist.cc:Mon May 24 07:12:46 2021[1,1]<stderr>:113: MNISTIter: load 30000 images, shuffle=1, shape=[64,1,28,28]
  Mon May 24 07:12:46 2021[1,0]<stderr>:[07:12:46Mon May 24 07:12:46 2021[1,0]<stderr>:] src/io/iter_mnist.cc:113: MNISTIter: load 30000 images, shuffle=1, shape=[64,1,28,28]
  Mon May 24 07:12:46 2021[1,1]<stderr>:[07:12:46] src/io/iter_mnist.cc:113: MNISTIter: load 10000 images, shuffle=1, shape=[Mon May 24 07:12:46 2021[1,1]<stderr>:64,1,28,28]
  Mon May 24 07:12:47 2021[1,0]<stderr>:[07:12:47] src/io/iter_mnist.cc:113: MNISTIter: load 10000 images, shuffle=Mon May 24 07:12:47 2021[1,0]<stderr>:1, shape=[64,1,28,28]
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Bootstrap : Using eth0:10.255.100.13<0>
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO NET/Socket : Using [0]eth0:10.255.100.13<0>
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Using network Socket
  Mon May 24 07:12:50 2021[1,0]<stdout>:NCCL version 2.9.6+cuda10.2
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Bootstrap : Using xgbe0:10.127.28.15<0>
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO NET/Socket : Using [0]xgbe0:10.127.28.15<0>
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Using network Socket
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 00/02 :    0   1
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 01/02 :    0   1
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Setting affinity for GPU 0 to ffffff
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 00 : 1[3f000] -> 0[40000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 00 : 0[40000] -> 1[3f000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 01 : 1[3f000] -> 0[40000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 01 : 0[40000] -> 1[3f000] [receive] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 00 : 0[40000] -> 1[3f000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 00 : 1[3f000] -> 0[40000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Channel 01 : 0[40000] -> 1[3f000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Channel 01 : 1[3f000] -> 0[40000] [send] via NET/Socket/0
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Connected all rings
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Connected all trees
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Connected all rings
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO comm 0x7ff8d8346190 rank 0 nranks 2 cudaDev 0 busId 40000 - Init COMPLETE
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO Connected all trees
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
  Mon May 24 07:12:50 2021[1,1]<stdout>:yq01-sys-hic-k8s-v100-box-a225-0562:6159:6315 [0] NCCL INFO comm 0x7f22c834adb0 rank 1 nranks 2 cudaDev 0 busId 3f000 - Init COMPLETE
  Mon May 24 07:12:50 2021[1,0]<stdout>:yq01-sys-hic-v100-box-a223-0155:6243:6375 [0] NCCL INFO Launch mode Parallel
  Mon May 24 07:12:50 2021[1,0]<stderr>:[07:12:50] Mon May 24 07:12:50 2021[1,0]<stderr>:src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
  Mon May 24 07:12:50 2021[1,1]<stderr>:[07:12:50] Mon May 24 07:12:50 2021[1,1]<stderr>:src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
  Mon May 24 07:12:51 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 100] Training: accuracy=0.867500
  Mon May 24 07:12:51 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 100] Training: accuracy=0.865469
  Mon May 24 07:12:51 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 200] Training: accuracy=0.914297
  Mon May 24 07:12:51 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 200] Training: accuracy=0.915547
  Mon May 24 07:12:51 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 300] Training: accuracy=0.934219
  Mon May 24 07:12:51 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 300] Training: accuracy=0.934010
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:[Epoch 0 Batch 400] Training: accuracy=0.944688
  Mon May 24 07:12:52 2021[1,1]<stderr>:INFO:root:[Epoch 0 Batch 400] Training: accuracy=0.944688
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:Epoch[0]	Speed=28557.88 samples/s	Time cost=2.097635
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:Epoch[0]	Train: accuracy=0.949753	Validation: accuracy=0.983373
  Mon May 24 07:12:52 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 100] Training: accuracy=0.984375
  Mon May 24 07:12:52 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 100] Training: accuracy=0.981094
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 200] Training: accuracy=0.985078
  Mon May 24 07:12:53 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 200] Training: accuracy=0.981016
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 300] Training: accuracy=0.985000
  Mon May 24 07:12:53 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 300] Training: accuracy=0.981563
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:[Epoch 1 Batch 400] Training: accuracy=0.985117
  Mon May 24 07:12:53 2021[1,1]<stderr>:INFO:root:[Epoch 1 Batch 400] Training: accuracy=0.982500
  Mon May 24 07:12:53 2021[1,0]<stderr>:INFO:root:Epoch[1]	Speed=40224.99 samples/s	Time cost=1.489223
  Mon May 24 07:12:54 2021[1,0]<stderr>:INFO:root:Epoch[1]	Train: accuracy=0.985544	Validation: accuracy=0.986378
  Mon May 24 07:12:54 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 100] Training: accuracy=0.990469
  Mon May 24 07:12:54 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 100] Training: accuracy=0.989219
  Mon May 24 07:12:54 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 200] Training: accuracy=0.990703
  Mon May 24 07:12:54 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 200] Training: accuracy=0.988047
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 300] Training: accuracy=0.990417
  Mon May 24 07:12:55 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 300] Training: accuracy=0.988229
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:[Epoch 2 Batch 400] Training: accuracy=0.990156
  Mon May 24 07:12:55 2021[1,1]<stderr>:INFO:root:[Epoch 2 Batch 400] Training: accuracy=0.988711
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:Epoch[2]	Speed=40484.51 samples/s	Time cost=1.479677
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:Epoch[2]	Train: accuracy=0.990151	Validation: accuracy=0.988482
  Mon May 24 07:12:55 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 100] Training: accuracy=0.993594
  Mon May 24 07:12:55 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 100] Training: accuracy=0.993437
  Mon May 24 07:12:56 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 200] Training: accuracy=0.992266
  Mon May 24 07:12:56 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 200] Training: accuracy=0.993359
  Mon May 24 07:12:56 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 300] Training: accuracy=0.993229
  Mon May 24 07:12:56 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 300] Training: accuracy=0.992344
  Mon May 24 07:12:56 2021[1,0]<stderr>:INFO:root:[Epoch 3 Batch 400] Training: accuracy=0.993203
  Mon May 24 07:12:56 2021[1,1]<stderr>:INFO:root:[Epoch 3 Batch 400] Training: accuracy=0.992500
  Mon May 24 07:12:57 2021[1,0]<stderr>:INFO:root:Epoch[3]	Speed=34536.84 samples/s	Time cost=1.734496
  Mon May 24 07:12:57 2021[1,0]<stderr>:INFO:root:Epoch[3]	Train: accuracy=0.993356	Validation: accuracy=0.989083
  Mon May 24 07:12:57 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 100] Training: accuracy=0.996094
  Mon May 24 07:12:57 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 100] Training: accuracy=0.995469
  Mon May 24 07:12:58 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 200] Training: accuracy=0.995547
  Mon May 24 07:12:58 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 200] Training: accuracy=0.994531
  Mon May 24 07:12:58 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 300] Training: accuracy=0.995469
  Mon May 24 07:12:58 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 300] Training: accuracy=0.994375
  Mon May 24 07:12:58 2021[1,0]<stderr>:INFO:root:[Epoch 4 Batch 400] Training: accuracy=0.995273
  Mon May 24 07:12:58 2021[1,1]<stderr>:INFO:root:[Epoch 4 Batch 400] Training: accuracy=0.994609
  Mon May 24 07:12:59 2021[1,0]<stderr>:INFO:root:Epoch[4]	Speed=34617.01 samples/s	Time cost=1.730479
  Mon May 24 07:12:59 2021[1,0]<stderr>:INFO:root:Epoch[4]	Train: accuracy=0.995259	Validation: accuracy=0.990084
  root@yq01-sys-hic-v100-box-a223-0155:/home/users/liuyuhui/workspace/horovod/examples/mxnet#
  ```
  
  Any suggestions will be highly welcome!
  
  