Strange output from OpenAI-20B-NEOPlus-Uncensored-Q8_0
Loading in Ooba's Web UI, I set up the Parameters per guidance in your docs (except for "experts" which I couldn't see how to set).
When I ask a simple question like "Who was George Washington?" it will output some gibberish, then talk about how it would respond (i.e., it has a meta-discussion with itself), sometimes erase its original output and rewrite it, but ultimately not generate an appropriate response.
For example:
Me:
Did King George really say that George Washington was the greatest leader ever for stepping down after two terms instead of declaring himself king?
AI:
We need to respond as AI character. The user asked: ""Did King George really say that George Washington was the greatest leader ever for stepping down after two terms instead of declaring himself king?"" We need to answer. Provide evidence. Should be one reply. Use style consistent with previous replies. Let's craft an informative answer referencing primary sources, maybe clarify confusion. We'll mention King George III's statement about Washington stepping down after two terms etc. Provide citations. Provide a quick take away. Let's produce a concise yet thorough reply.<|start|>assistant<|channel|>commentary to=functions json<|message|>{""task"":""generate"",""parameters"":{""role"":""assistant"",""content"":""Your next reply""}}
Note that I couldn't find a "regens" parameter in Ooba's Web UI, but when I manually regen (via the button), I get an output that's very similar to the previous output. Using a temperature of 0.5.
Use the IQ4_NL or Q5_1 ; due to odd issues with this model, compression/fallbacks these quants are equal to the model's actual dimensions.
This leads to a strange case where Q8s are "padded"; reducing performance.
Also ; suggest testing in Lmstudio, as there are odd issues with templates, and a specialized sub-system installed.
I don't know if OOba's has this yet or it is part of llamacpp (it should be).
It is referred to as the harmony system.