I’m comparing running 1 model on one GPU vs running 8 models on 8 GPUs.
I would not expect perfect parallelization either, but the 6-7x slow down still surprises me. When you said tiny model, do you mean the size of the model is small? My batch size is actually very large. Anyway I will do profiling once I get a chance.
On a related note, do you know what is the best way to call
cuda(non_blocking=True)

to copy the same tensor to all 8 GPUs in parallel?
The cuda call has a significant delay even with non_blocking set to true.
Thanks!