If you are worried about host-device/device-host data copy from one model blocks computation from another model, you can try using multiple CUDA streams 74, one stream per model. Operations in different streams can run in parallel.