Hello all,
I have a question regarding the usage of DataParallel:
model = nn.DataParallel(model)
When working with GANs, I use the above line where model is replaced by the Generator and Critic respectively. However, except for the opportunity to use a higher batch size as there is more memory, there is no actual speedup for the training process.
In fact, it is slower then training the whole network on a single GPU.
Is this expected when working with generative adversarial networks or networks in general?
The GPUs that I am running on are the same, so I do not see the possibility of a bottleneck by a slower GPU.
Thanks in advance.