I am currently trying to infer 2 torch models on the same GPU, but my observation is that if 2 of them run at the same time in 2 different threads, the inference time is much larger than running them individually.
Is there any way to make use of single GPU for running multiple models in parallel?

Reference:

  
    
    
    Inference Multiple Models on a Single GPU distributed
  
  
    I am trying to build a system and i need to do inference on 60 segmentation models at same time ( Same models but different inputs). I wonder if this is possible to do on pytorch? 
I am not sure about what kinda system i need to use for this, i am planning to use 4X RTX8000 and if it is not enough i can use two system with 4X RTX8000 each or with a better gpu. 
Would i lose too much performance because of using multiple models? How many models i can put on a gpu, what the performance of it depen…