Yeah the original model might occupies multiple GPUs, in this case if you try to parallel to models to do the inference on the set of GPUs, there might be more synchronizations happens to ensure two model computation not interfering each other, which might reduce the GPU utility in general. You can try what @eqy suggested. Also, if you like to know how your two models use GPU separately, you can do a profling for the two models separately with pytorch profiler, and see if there’re already operations occuping multiple GPUs.