bartowski

/

ai21labs_AI21-Jamba-Mini-1.7-GGUF

like 0

It's definitely the format that's in the original model template, so that's weird!

Not sure why it would work better with a format other than the one they provide 🤔

My bad, llama-server did it for me:

srv    load_model: load_model: Chat template parsing error: this custom template is not supported, try using --jinja
srv    load_model: load_model: The chat template that comes with this model is not yet supported, falling back to chatml. This may cause the model to output suboptimal responses
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32768
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv  update_slots: all slots are idle

Just out of curiosity, how could I run this with the correct template?

Edit: Right, all I had to do is add --jinja to the llama-server command. Closing!