yousiyu:

The run time is significantly slower, probably 6x~7x than just running it on one single GPU


It seems you are comparing the execution of a single model on a single GPU vs. the execution of all 8 models on 8 different GPUs. Is my understanding correct or were you comparing the 8-GPU run vs. running 8 models sequentially on a single GPU?
I wouldn’t expect the latter case to be slower in the multi-GPU setup, but would expect to not see a perfect scaling in the former use case.
While the models will be executed in parallel, the actual kernel launch overhead might be visible for tiny models. To verify it, you could create a profile with e.g. Nsight Systems and check the achieved overlap.