Sadly Q8_0 just gives gibberish with llama.cpp
#1
by
HenkPoley
- opened
./main -e -p "<|im_start|>user\nHow to get in a good university?<|im_end|>\n<|im_start|>assistant\n" -m models/s3nh/jeff31415-TinyLlama-1.1B-1.5T-OpenOrca-Alpha-GGUF/jeff31415-TinyLlama-1.1B-1.5T-OpenOrca-Alpha.Q8_0.gguf
<|im_start|>user
How to get in a good university?<|im_end|>
<|im_start|>assistant
ive hereowhere "wallishivehereightiveiveoereiveo#hereo...#o#o.o"hist.a andhusto-oohoâxomhun....in...c.haxahi.
r...n.ax) 1:nh")ndelastr"r.st
Just in case, llama.cpp 2b4ea35e56792064598e922e46d081e02bc96b94 (current)
Have to check it out, thanks for the feedback
Maybe it requires an <|im_start|>system\nSome system message\n<|in_end|>
Or special single tokens for <|im_start|> need to be used by the LLM runtime, and llama.cpp doesn’t ?