How Is LLaMa.cpp Possible?
I'm shocked - before I seen it working myself, I was not believe, so large (smart) LLM could run on desktop cpu.- core i7, 4 cores, 3.3GHz, 64G ddr4-2400, 70b model, ~0.5 tokens/s; 30b model ~1 tokens/s.