tq1_0 and tq2_0 are *not* for this model! for ternary style models only!
#1
by
ubergarm
- opened
Heya @volker-mauel thanks for the excitement to quantize a new model, but just a heads up that @compilade (the creator of tq1_0 and tq2_0) are for bitnet only https://github.com/ggml-org/llama.cpp/pull/8151 and https://www.reddit.com/r/LocalLLaMA/comments/1la1v4d/comment/mxioh75/
So I'd suggest trying some other quants designed for this type of LLM instead and deleting these to prevent confusion. Some others have made this mistake before, I tried it myself in the beginning but did not release my mistake into the wild haha...
Cheers!