Good luck! One other thing I do remember now, I ran into some issues with the MP queuing. It was fairly sensitive to how the processes were spawned and would blow up in some configs with unexpected lockups or crashes. I think the settings in the repo worked with the version of PyTorch + CUDA I was using at the time but it may very well have changed…
I had simple test method in the mp_feeder that would crash or lock up with some spawn/sharing strategy combos…

  
      github.com
  
  
    rwightman/pytorch-nips2017-adversarial/blob/master/python/mp_feeder.py
import sys
import torch
import torch.utils.data
import torch.multiprocessing as mp
try:
    mp.set_sharing_strategy('file_system')
    mp.set_start_method('spawn')
except RuntimeError:
    pass


class MpFeeder:
    def __init__(self, dataset, maxsize=2):
        self.queue = mp.Queue(maxsize=maxsize)
        self.dataset = dataset
        self.is_done = mp.Event()
        self.is_shutdown = mp.Event()
        self.process = mp.Process(target=self._run)
        self.process.start()



  This file has been truncated. show original