The training hangs without printing any logs. Observations/configurations:

4 nodes. 4 GPU/node.
distributed training with each process taking 1 GPU.
pytorch version = 1.1; cuda version = 9.0; gpu driver version: 410.78
use the code base of facebook/maskrcnn-benchmark, but i thought it is just normal pytorch code.
GPU utility is close to 100%, but there is no more log.
it has finished 278K iterations, and then hangs there without any progress (no more snapshot, no more logs)
gdb attached to one of the process (sudo gdb -p process_id) and it seems like it hangs at cuMemcpyHtoDAsync_v2.

(gdb) where
#0  0x00007ffe309e1b6d in clock_gettime ()
#1  0x00007f8cc536f876 in __GI___clock_gettime (clock_id=4, tp=0x7ffe30898660) at ../sysdeps/unix/clock_gettime.c:115
#2  0x00007f8c6c7ecc4e in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#3  0x00007f8c6c87b8d3 in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#4  0x00007f8c6c89b81f in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#5  0x00007f8c6c7c8737 in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#6  0x00007f8c6c6d9e4e in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#7  0x00007f8c6c6dbfc3 in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#8  0x00007f8c6c829c82 in cuMemcpyHtoDAsync_v2 () from /usr/local/nvidia/lib64/libcuda.so.1
#9  0x00007f8cbe7ad49c in ?? () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcudart-f7fdd8d7.so.9.0
#10 0x00007f8cbe78a573 in ?? () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcudart-f7fdd8d7.so.9.0
#11 0x00007f8cbe7c3d86 in cudaMemcpyAsync () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcudart-f7fdd8d7.so.9.0
#12 0x00007f8c836a9f4b in (anonymous namespace)::copy_from_cpu(at::Tensor&, at::Tensor const&) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#13 0x00007f8c8374a875 in void (anonymous namespace)::_copy__cuda<float>(at::Tensor&, at::Tensor const&, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#14 0x00007f8c836aafb8 in at::native::_s_copy__cuda(at::Tensor&, at::Tensor const&, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#15 0x00007f8c826d47ef in at::CUDAType::s_copy_(at::Tensor&, at::Tensor const&, bool) const () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#16 0x00007f8c7764033d in at::native::copy_(at::Tensor&, at::Tensor const&, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#17 0x00007f8cbf546dc9 in torch::autograd::VariableType::copy_(at::Tensor&, at::Tensor const&, bool) const () from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1
#18 0x00007f8c777829cc in at::native::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#19 0x00007f8c77a01857 in at::TypeDefault::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) const () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#20 0x00007f8cbf31cb52 in torch::autograd::VariableType::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) const ()
   from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1
#21 0x00007f8cc0bb8eb3 in torch::autograd::dispatch_to(at::Tensor const&, c10::Device, bool, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#22 0x00007f8cc0bb9598 in torch::autograd::THPVariable_to(_object*, _object*, _object*) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#23 0x0000556d518096a6 in PyCFunction_Call () at /tmp/build/80754af9/python_1546130271559/work/Objects/methodobject.c:98
#24 0x0000556d518b74ad in do_call_core (kwdict=0x7f8b9b42b168, callargs=0x7f8bb1deec18, func=0x7f8b9b42b510) at /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:5116
#25 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:3404