Update the instructions on requirements
#10
by
segmond
- opened
It is recommended to have at least 128GB unified RAM memory to run the small quants. With 16GB VRAM and 256 RAM, expect 5+ tokens/sec. For best results, use any 2-bit XL quant or above.
You definitely need more than this to run even the smallest model.
It is recommended to have at least 128GB unified RAM memory to run the small quants. With 16GB VRAM and 256 RAM, expect 5+ tokens/sec. For best results, use any 2-bit XL quant or above.
You definitely need more than this to run even the smallest model.
What do you mean? It can run on that requirement. Also we wrote 'at least'
The different question is whether it's even worth running the lowest potato quant of K2 compared to R1 at 2bit XL (not for me it isn't).