If I use DataParallel() in tranformer model (https://pytorch.org/tutorials/beginner/transformer_tutorial.html)
The time for each epoch increases as we increase the number of GPUs,
Example: time for one epoch
1 GPU = 42 sec - 44 sec
4 GPUs = 175 sec - 180 sec
Why performance is decreasing if we increase the number of GPUs, is it a drawback in DataParallel()?