I am pretty sure that if you e.g., set the minibatch size to 32 in the dataloader and have say 8 GPUs, you get 4 data points per GPU during DataParallel (you can kind of see that based on the number of training instances per minibatch and also memory use) . I am wondering if you are referring to DataParallel in your question?
In practice, I found that yes, using e.g., 4 GPUs can maybe speed up computation by ~2.5 times, but then I also need to train 1.5 more epochs to get comparable results (compared to 1 GPU).
Also, I usually use minibatchsize * number of GPUs as my minibatch param to make best use of your GPUs. I.e., if you have a model with a current batch size that fills up 80% of the GPU memory, and you want to maintain that level of utilization, you’d need to increase the batch size then of course as the batch gets scattered across the GPUs.