Did anyone succeed to reproduce their code when using multiGPUs?
If yes, could you share how you did it? (general idea)
My code is totally reproducible when using one single GPU (independently of the number of workers >= 0); however, it loses its reproducibility when using multiple GPUs.
Randomness on samples (such as transformations) is controlled (fixed using seeds for each sample).
I use torch.nn.DataParallel to wrap the model for multiGPUs.
Who to blame for this non-reproducibility in the case of multiGPUs? atomic operations (I hope not)?
Pytorch reproducibility note 17.
I use Pytorch 1.0.0, Python 3.7.0.
I use standard guide to fix the seeds of the modules:
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)  # for multiGPUs.
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

Thanks!