You’re right about latency - so to improve that I’d first try using PyTorch’s native quantization  to see if it meets your needs, followed by ONNX + ORT (as you seem to be exploring in a separate thread :))

PyTorch’s native quantization isn’t the same as the dynamic quantization proposed by the hugging face pipeline ? Secondly, ONNX and ORT are separate things? The accelaration isn’t provided by the ONNX model itself and need to be run by ONNX runtime ?

This is definitely a common workflow when you want to deploy you model as a web application that other services can interact with (an alternative to FastAPI is Flask, but I personally feel it is more clunky to use)

Could you precise me more about the status of web application ? My goal isn’t to provide a web site where client could run inference on their data via web navigator. I just want to put my model on the cloud and provide an API from which I could send request in real-time for the need of my global application. Is that filled the needs of FastAPI + Docker ?

Depending on your use case / environment restrictions, an alternative to the whole web app / docker / k8s complexity could be cortex labs: GitHub - cortexlabs/cortex: Deploy, manage, and scale machine learning models in production 3

Thanks for the link, I indeed already had a look at cortex which seems interesting !

Ah that’s a good point re hardware - as far as I know AWS has more options than GCP (sometimes I couldn’t get access to a specific VM on GCP). I’ve usually used EC2 instances in the past so can’t comment on Sagemaker, although some friends of mine warned me that Sagemaker can be pricey 

Is it possible to deploy on EC2 instances without k8s ? If yes, what are the methods ?