This will be a good use case for TorchServe which supports multiple models. Please give that a try and provide your feedback.

  
      
      GitHub
  
  
    

pytorch/serve 3

Model Serving on PyTorch. Contribute to pytorch/serve development by creating an account on GitHub.