nn.DataParallel(model).cuda() hangs
Hi, <SCODE>Traceback (most recent call last):
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/polphit/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 28, in _worker_loop
    r = index_queue.get()
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/queues.py", line 343, in get
    res = self._reader.recv_bytes()
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/polphit/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
<ECODE> I tried on two different machines and got the same issue. Ubuntu 16.04, conda 4.3.14, pytorch installed from source, python 3.6.0.final.0 requests 2.12.4 CUDA 8.0 cuDNN 5.1 When I run the same code on a machine without conda, and python3, it works well.
thnkim
That’s a stack trace of a data loader process, can you paste a full error into a gist and link it here?
apaszke
(I have two GPU devices) Flow is <SCODE>i (input) -> netA ---> netB -> x (output #1)
                   +-> netC -> y (output #2)
                   +-> netD -> z (output #3)
<ECODE> If this is not helpful to guess the cause, I would like to simplify my codes to reproduce the issue with minimal data upload.
thnkim
Oh, when I add torch.cuda.synchronize() at the end of a batch, one machine works properly, although the other machine still has the same issue.
thnkim
apaszke
Unfortunately even if we add these locs, doing that in two processes that use the same GPUs in DataParallel will deadlock too…
apaszke
so…is it a bug of pytorch? I met the same issue.
melody-rain
No, it’s a bug in NCCL (NVIDIA’s library). But you probably shouldn’t be using the same GPU in multiple data parallel jobs anyway.
apaszke
octopusyo
Please have a look and see if it applies to you.
smth
Mohit_Chhabra
It was some hardware issue, although the p2platency test passed, when i changed the CUDA_VISIBLE_DEVICES , and ran the code without DataParallel it gave illegal memory access error for the faulty GPU. Changed the GPU and now the code works fine.
Mohit_Chhabra