I trained my model in two nodes, and then it hangs up in initiation.
截屏2021-02-01 下午10.40.241214×410 50.8 KB
截屏2021-02-01 下午10.41.371960×592 183 KB
then I add NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=all to see what’s going on, but there is no output file.