I would like to train, say, 10 independent neural networks on 5 GPUs in parallel (by training two on each GPU assuming there are no memory constraints). Also, I would like the code to be reproducible. Therefore, I have been training each network by re-running the same Python script, changing only the GPU device. I specify torch.manual_seed for reproducibility.
Is there anyway to do this in a single Python script where PyTorch does the work of distributing the networks to the GPUs while maintaining reproducibility? I am aware of threading as one way to do this (something similar to ModelParallel 3), but I am worried about reproducibility of my code when using threading.
Thank you for the help!