Hi,
I am trying to train multiple neural networks on a machine with multiple GPUs. I have already used DataParallel module to parallelize this process. My understanding of DataParallel is that it can only help train each model one by one parallelly. However, all the GPUs are not fully utilized if I train these networks one by one.
I am wondering if there is any method to perform multiple networks training at the same time and fully use all my GPUs? I tried to think about using torch.multiprocessing. Yet it seems like pretty painful to use that in my case.
Any suggestions will be really appreciated. Thanks!