I’m currently trying to run an NLP model using DistributedDataParallel and I’ve been receiving the following error if I use more than one worker for DataLoader (this error appears for each worker process):
Traceback (most recent call last):
  File "<string>", line 1 in <module>
  File "/opt/conda/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(df)
  File "/opt/conda/lib/python3.6/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
  File "/opt/conda/lib/site-packages/torch/nn/parallel/distributed.py", line 396, in __setstate__
    self.process_group = _get_default_group()
  File "/opt/conda/lib/site-packages/torch/distributed/distributed_c10d.py", line 286, in _get_default_group
    raise RuntimeError("Default process group has not been initialized, "
Default process group has not been initialized, please make sure to call init_process_group.

In main() that I call after torch.multiprocessing.spawn(), I use the following call:
dist.init_process_group("nccl", rank=rank, world_size=args.gpu, init_method="file:///app/tmp/sharedfile")

I don’t receive this error if I set the number of workers to 0. I still receive this error if I change init_method to env:// (and I have the port and address variables set). I would like this to work in file mode though, since I can’t change the size of /dev/shm.
The error itself seems to trigger when I start iterating through dataloader for my epoch (which means I don’t begin a single training loop before the error).
I’m using 4 GPUs on a single node centos docker image with pytorch 1.4.0 and python 3.6.9. Let me know if you need further info, appreciate any tips!