ptrblck:

NCCL_DEBUG=DEBUG


Thanks for the response.
That flag does not generate much info other than the usual output. I reran the code with NCCL_DEBUG=INFO, below is the log (machine is named gpu123):

$ NCCL_DEBUG=INFO python main.py -a resnet18 --dist-url ‘tcp://127.0.0.1:6840’ --dist-backend ‘nccl’ --multiprocessing-distributed --world-size 1 --local_rank 0 ~/tiny-imagenet-200
Use GPU: 3 for training                                                                                                                                   Use GPU: 5 for training                                                                                                                                   Use GPU: 7 for training                                                                                                                                   Use GPU: 9 for training                                                                                                                                   Use GPU: 6 for training                                                                                                                                   Use GPU: 1 for training                                                                                                                                   Use GPU: 0 for training                                                                                                                                   Use GPU: 4 for training                                                                                                                                   => creating model ‘resnet18’                                                                                                                              Use GPU: 8 for training                                                                                                                                   => creating model ‘resnet18’                                                                                                                              Use GPU: 2 for training                                                                                                                                   => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              => creating model ‘resnet18’                                                                                                                              gpu123:7932:7932 [0] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7932:7932 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7932:7932 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   NCCL version 2.4.8+cuda10.0                                                                                                                               gpu123:7941:7941 [9] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7936:7936 [4] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7934:7934 [2] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7933:7933 [1] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7937:7937 [5] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7935:7935 [3] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7941:7941 [9] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7936:7936 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7934:7934 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7933:7933 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7937:7937 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7935:7935 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7940:7940 [8] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7940:7940 [8] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7941:7941 [9] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7935:7935 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7933:7933 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7936:7936 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7934:7934 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7937:7937 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7940:7940 [8] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7932:8011 [0] NCCL INFO Setting affinity for GPU 0 to 0fff                                                                                         gpu123:7941:8013 [9] NCCL INFO Setting affinity for GPU 9 to 0fff                                                                                         gpu123:7940:8025 [8] NCCL INFO Setting affinity for GPU 8 to 0fff                                                                                         gpu123:7934:8022 [2] NCCL INFO Setting affinity for GPU 2 to 0fff                                                                                         gpu123:7937:8023 [5] NCCL INFO Setting affinity for GPU 5 to 0fff                                                                                         gpu123:7935:8017 [3] NCCL INFO Setting affinity for GPU 3 to 0fff                                                                                         gpu123:7936:8020 [4] NCCL INFO Setting affinity for GPU 4 to 0fff                                                                                         gpu123:7933:8018 [1] NCCL INFO Setting affinity for GPU 1 to 0fff                                                                                         gpu123:7939:7939 [7] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7939:7939 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7939:7939 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7938:7938 [6] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223<0>                                                                                  gpu123:7938:7938 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7938:7938 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223<0>                                                                   gpu123:7939:8027 [7] NCCL INFO Setting affinity for GPU 7 to 0fff                                                                                         gpu123:7938:8029 [6] NCCL INFO Setting affinity for GPU 6 to 0fff                                                                                         gpu123:7932:8011 [0] NCCL INFO Channel 00 :    0   1   2   3   4   5   6   7   8   9                                                                      gpu123:7935:8017 [3] NCCL INFO Ring 00 : 3[3] -> 4[4] via P2P/IPC                                                                                         gpu123:7938:8029 [6] NCCL INFO Ring 00 : 6[6] -> 7[7] via P2P/IPC                                                                                         gpu123:7936:8020 [4] NCCL INFO Ring 00 : 4[4] -> 5[5] via P2P/IPC                                                                                         gpu123:7937:8023 [5] NCCL INFO Ring 00 : 5[5] -> 6[6] via P2P/IPC                                                                                         gpu123:7939:8027 [7] NCCL INFO Ring 00 : 7[7] -> 8[8] via P2P/IPC                                                                                         gpu123:7941:8013 [9] NCCL INFO Ring 00 : 9[9] -> 0[0] via P2P/IPC                                                                                         gpu123:7940:8025 [8] NCCL INFO Ring 00 : 8[8] -> 9[9] via P2P/IPC                                                                                         gpu123:7933:8018 [1] NCCL INFO Ring 00 : 1[1] -> 2[2] via P2P/IPC                                                                                         gpu123:7932:8011 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/IPC                                                                                         gpu123:7934:8022 [2] NCCL INFO Ring 00 : 2[2] -> 3[3] via P2P/IPC                                                                                                                                                                                                                                                   gpu123:7941:8013 [9] transport/p2p.cc:574 NCCL WARN failed to open CUDA IPC handle : 60 peer mapping resources exhausted                                  gpu123:7941:8013 [9] NCCL INFO init.cc:669 -> 1                                                                                                           gpu123:7941:8013 [9] NCCL INFO init.cc:815 -> 1                                                                                                           gpu123:7941:8013 [9] NCCL INFO init.cc:951 -> 1                                                                                                           gpu123:7941:8013 [9] NCCL INFO misc/group.cc:69 -> 1 [Async thread]                                                                                                                                                                                                                                                 gpu123:7932:8011 [0] transport/p2p.cc:604 NCCL WARN failed to open CUDA IPC handle : 60 peer mapping resources exhausted                                  gpu123:7932:8011 [0] NCCL INFO init.cc:679 -> 1                                                                                                           gpu123:7932:8011 [0] NCCL INFO init.cc:815 -> 1                                                                                                           gpu123:7932:8011 [0] NCCL INFO init.cc:951 -> 1                                                                                                           gpu123:7932:8011 [0] NCCL INFO misc/group.cc:69 -> 1 [Async thread]                                                                                       gpu123:7935:8017 [3] NCCL INFO comm 0x2abf58001e10 rank 3 nranks 10 cudaDev 3 nvmlDev 3 - Init COMPLETE                                                   gpu123:7934:8022 [2] NCCL INFO comm 0x2b8408001e10 rank 2 nranks 10 cudaDev 2 nvmlDev 2 - Init COMPLETE                                                   gpu123:7937:8023 [5] NCCL INFO comm 0x2b011c001e10 rank 5 nranks 10 cudaDev 5 nvmlDev 5 - Init COMPLETE                                                   gpu123:7936:8020 [4] NCCL INFO comm 0x2b4b28001e10 rank 4 nranks 10 cudaDev 4 nvmlDev 4 - Init COMPLETE                                                   gpu123:7938:8029 [6] NCCL INFO comm 0x2b58d8001e10 rank 6 nranks 10 cudaDev 6 nvmlDev 6 - Init COMPLETE                                                   gpu123:7933:8018 [1] NCCL INFO comm 0x2af070001e10 rank 1 nranks 10 cudaDev 1 nvmlDev 1 - Init COMPLETE                                                   gpu123:7939:8027 [7] NCCL INFO comm 0x2b2180001e10 rank 7 nranks 10 cudaDev 7 nvmlDev 7 - Init COMPLETE                                                   gpu123:7940:8025 [8] NCCL INFO comm 0x2b015c001e10 rank 8 nranks 10 cudaDev 8 nvmlDev 8 - Init COMPLETE                                                   /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             Traceback (most recent call last):                                                                                                                          File “main.py”, line 425, in                                                                                                                        main()                                                                                                                                                  File “main.py”, line 109, in main                                                                                                                           mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))                                                                               File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn                           while not spawn_context.join():                                                                                                                         File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 118, in join                            raise Exception(msg)                                                                                                                                  Exception:                                                                                                                                                                                                                                                                                                          – Process 9 terminated with the following error:                                                                                                         Traceback (most recent call last):                                                                                                                          File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap                            fn(i, *args)                                                                                                                                            File “/nfs/scistore08/alistgrp/bchatter/workspace/async-opt/dist_data_parallel/imagenet_training/main.py”, line 151, in main_worker                         model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])                                                                         File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 298, in init                      self.broadcast_bucket_size)                                                                                                                             File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 480, in _distributed_broadcast_coalesced                                                                                                                                                        dist._broadcast_coalesced(self.process_group, tensors, buffer_size)                                                                                   RuntimeError: NCCL error in: /tmp/pip-req-build-4baxydiv/torch/lib/c10d/ProcessGroupNCCL.cpp:400, unhandled cuda error