Your CPU workload might be the bottleneck, e.g. if both processes need to load and process data before feeding it into the model. To verify it you could profile these processes in isolation and while both are running and compare the time lines e.g. via Nsight Systems.