all reduce also has such issue:
10.4.22.101: ip-10-4-22-101:744:3711 [6] NCCL INFO Channel 01 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
10.4.22.101:     self.overflow = self.overflow_checker.check_using_norm(norm_groups)
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/utils.py", line 102, in check_using_norm
10.4.22.101:     dist.all_reduce(cuda_overflow, op=torch.distributed.ReduceOp.MAX)
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 1206, in all_reduce
10.4.22.101:     work = default_pg.allreduce([tensor], opts)
10.4.22.101: RuntimeError: NCCL error in: /tmp/pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 21.0.3