nn.DataParallel allows to replicate and parallelize the execution of a model by sharding over the batch.
This assumes the model can fit inside of GPU memory. Is there a natural way in pytorch to run across multi-GPU a single model.
On a similar topic, give a GAN setting with a generator and a discriminator and two GPUs, what is the recommendation to speed-up the computation, given the dependency between discriminator and generator?