You’re right about latency - so to improve that I’d first try using PyTorch’s native quantization 2 to see if it meets your needs, followed by ONNX + ORT (as you seem to be exploring in a separate thread :))
If that’s still not fast enough, you could try fine-pruning your models with the brand-new nn_pruning library (link) followed by ORT optimization

Is using FastAPI + Docker a common easy way to deploy on kubernetes ? Would you have any reference to guide me step by step?

This is definitely a common workflow when you want to deploy you model as a web application that other services can interact with (an alternative to FastAPI is Flask, but I personally feel it is more clunky to use). Here’s a pretty good tutorial on getting started with them (just ignore the dependency-injection stuff because your model is way to big!): https://towardsdatascience.com/how-to-properly-ship-and-deploy-your-machine-learning-model-8a8664b763c4 8
Depending on your use case / environment restrictions, an alternative to the whole web app / docker / k8s complexity could be cortex labs: GitHub - cortexlabs/cortex: Deploy, manage, and scale machine learning models in production 4
I haven’t tried it myself but it looks like a nice approach to deploying models by just defining a few python classes and performance testing them via a CLI

Thanks for guide through the choice of better UX cloud, but does GCP contrains the multitude of choices concerning the different target architecture ?
Would you use directly EC2 instances or Sagemaker ?

Ah that’s a good point re hardware - as far as I know AWS has more options than GCP (sometimes I couldn’t get access to a specific VM on GCP). I’ve usually used EC2 instances in the past so can’t comment on Sagemaker, although some friends of mine warned me that Sagemaker can be pricey