Hi @lewtun,
It is effectively cheaper to use CPUs and scale horizontally so you can achieve a better throughput. However, this method doesn’t improve latency in the case of real-time inference with batch size=1.
https://toriml.medium.com/why-you-should-use-different-backends-when-deploying-dl-models-e3d23ee58a9d 10
Is using FastAPI + Docker a common easy way to deploy on kubernetes ? Would you have any reference to guide me step by step?
Thanks for guide through the choice of better UX cloud, but does GCP contrains the multitude of choices concerning the different target architecture ?
Would you use directly EC2 instances or Sagemaker ?
Thanks !