Hi, all,
I met a NcclError while running someone’s code. So I try a simpler way to test DataParallel function. So I changed the official mnist tutorial code 4 into DataParallel version like this
if args.cuda: model =torch.nn.DataParallel(model).cuda()
Then it shows the same error as I met in someone’s code.
Traceback (most recent call last): File "tu_mnist.py", line 126, in <module> train(epoch) File "tu_mnist.py", line 90, in train output = model(data) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 206, in __call__ result = self.forward(*input, **kwargs) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 60, in forward replicas = self.replicate(self.module, self.device_ids[:len(inputs)]) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 65, in replicate return replicate(module, device_ids) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 12, in replicate param_copies = Broadcast(devices)(*params) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 18, in forward outputs = comm.broadcast_coalesced(inputs, self.target_gpus) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/cuda/comm.py", line 57, in broadcast_coalesced results = broadcast(_flatten_tensors(chunk), devices) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/cuda/comm.py", line 26, in broadcast nccl.broadcast(tensors) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/cuda/nccl.py", line 180, in broadcast comm = communicator(inputs) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/cuda/nccl.py", line 133, in communicator _communicators[key] = NcclCommList(devices) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/cuda/nccl.py", line 106, in __init__ check_error(lib.ncclCommInitAll(self, len(devices), int_array(devices))) File "/root/Util/miniconda/envs/pt/lib/python3.6/site-packages/torch/cuda/nccl.py", line 118, in check_error raise NcclError(status) torch.cuda.nccl.NcclError: System Error (2)
I run my experimens in  Nvidia docker enviroment, CUDA 7.5, K80. Does anybody know why?