smth:

Issue: Multi-GPU K80s


Sadly, I met the error again.
I am going to see the issue, to find solutions.
The error is the loss.backward().
0it [00:00, ?it/s]Traceback (most recent call last):
  File "train.py", line 310, in <module>
    train_batch_conf()
  File "train.py", line 148, in train_batch_conf
    train(params_dict)
  File "train.py", line 193, in train
    loss.backward()
  File "/data1/public/research_venv/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 156, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/data1/public/research_venv/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 98, in backward
    variables, grad_variables, retain_graph)
  File "/data1/public/research_venv/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 25, in backward
    return comm.reduce_add_coalesced(grad_outputs, self.input_device)
  File "/data1/public/research_venv/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py", line 122, in reduce_add_coalesced
    result = reduce_add(flattened, destination)
  File "/data1/public/research_venv/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py", line 92, in reduce_add
    nccl.reduce(inputs, outputs, root=destination)
  File "/data1/public/research_venv/anaconda3/lib/python3.6/site-packages/torch/cuda/nccl.py", line 161, in reduce
    assert(root >= 0 and root < len(inputs))
AssertionError