Hi, Yes NLP models tend to use quite a bit of RAM. And maintaining them throughout in RAM is the only way to get fast inference.
You could use our hosted API inference (https://huggingface.co/pricing 11). This is the exact problem we aim to solve, and we can host your own finetuned model.
If you want to stick to your own hosted solution then:
1- Attempt to stick everything in RAM (get 1 pod per model to prevent loading/unloading issues)
2- You can try to distil your model, use fp16, onnx_runtime. That requires quite a bit of work and have various caveats.
3- Use a GPU even for inference to batch multiple inferences at once
If you can/want to use our hosted API, you can upload your model on the hub (https://huggingface.co/transformers/model_sharing.html 4) and then directly use the API
curl -X POST -d 'My sentence'  https://api-inference.huggingface.co/models/mymodel
We solve 1. and 3. for you automatically. And we are working hard to get 2. working (but it’s tricky).