Hardware recomendations help

#100
by MLSDev - opened

Hi I want to deploy the model for use test it in a Rag.

what is the gpu ram recommended for inference with 1024 chunk token size?

is cpu usage possible? with my Hetzner dedicated VPS doesn't appear enought

I try to use the api of jina ai, but tokens of my api key seems disapear without my use.

I find xinference like a method to integrate the use, any other recomendation?

Sign up or log in to comment