EOS Token
I'm finding that these quantized models don't know how to stop. Comparing your special_tokens_map.json to the original repo, you have a different "eos_token" value. I think this should be "<|end_of_turn|>" rather than "</s>".
Ah yeah, they must have edited their JSON after I did my quantisations. I have edited my JSON to match theirs, so do a re-download of those files and hopefully that will sort it out.
Could it be that TheBloke/vicuna-13B-v1.5-16K-GPTQ suffers from the same problem?
Could it be that TheBloke/vicuna-13B-v1.5-16K-GPTQ suffers from the same problem?
Don't think so - Vicuna has always used </s> and this hasn't changed
Thanks for the answer! Your newer GPTQ-models don't know when to stop in 1 of five cases on my end. The original vicuna-13B-v1.5-16K seems to be fine though..