As title suggests, I am wondering what’s the right way to expose a PyTorch model that thousands of requests a second?
Would one write a REST API in C++ and serve model using TorchScript?
Or can one be lazy and serve through Python based Flask app (or something more resilient) and use PyTorch as is?
Or can one take the extra step and try to serve using GoLang based service and somehow use a model saved as TorchScript?  (If so, how would one leverage GPU based inference?
What are the pros cons of the above methods?