Tool calls not supported in template?
#1
by
jkrauss82
- opened
It seems the template shipped with the files does not support tool calls. When I use the regular template from the file in textgen webui, I get a prompt like this:
<s>[SYSTEM_PROMPT]You are a helpful assistant.[/SYSTEM_PROMPT][INST]What time is it currently in New York City?[/INST]
Model answer:
I'm unable to provide real-time information because my knowledge was last updated in October 2023, and I don't have access to the internet. To find the current time in New York City, you can check a reliable world clock website or use a search engine. New York City is in the Eastern Time Zone (ET), so you might also want to consider whether Daylight Saving Time is currently in effect.
Passing a tool prompt like this:
<s>[SYSTEM_PROMPT]You are a helpful assistant.
## Tools
You have access to the following tools:
### get_current_time
{ "type": "function", "function": { "name": "get_current_time", "description": "Get current time in a specific timezones", "parameters": { "type": "object", "required": ["timezone"], "properties": { "timezone": { "type": "string", "description": "IANA timezone name (e.g., America/New_York, Europe/London). Use Europe/Berlin as local timezone if no timezone provided by the user." } } } } }
## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs:
<tool_call>
{ "name": <function_name>, "parameters": { <parameters> } }
</tool_call>[/SYSTEM_PROMPT][INST]What time is it currently in New York City?[/INST]
let's the model successfully generate a tool call:
<tool_call>
{ "name": "get_current_time", "parameters": { "timezone": "America/New_York" } }
</tool_call>
There is this discussion / proposed fix of the template for Ollama (might work for llama.cpp, too): https://www.reddit.com/r/LocalLLaMA/comments/1j82o15/fixed_ollama_template_for_mistral_small_3/
Yes I can see from the chat template no mention of tools. So you will need to replace the template in the gguf meta data. Start from the bf16 version then edit the meta data to replace the template. Then quantize the updated bf16 gguf to your required quant size.
Thanks for your response.
It would be nicer if it could be fixed in upstream though, otherwise everybody has to do this manually after downloading the models. Is there a chance this might get fixed?
On a side note: quants by mradermacher are affected as well, guess Mistral team has supplied an incomplete template.
Ok I will fix it and requantize do you have a direct link to a working template . I am mostly offline until next week. But I can do when I am back
Thanks, that would be great!
I will try to find / create a working template and post that here ASAP.
Below is a template which I have briefly tested and found it working in textgen webui. But further tests are required, especially with the tool calling.
I have taken it from this open PR by CISCai to the Mistral Small HF repo: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/24/files
{%- set today = strftime_now(\"%Y-%m-%d\") %}
{%- set default_system_message = \"You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\
Your knowledge base was last updated on 2023-10-01. The current date is \" + today + \".\
\
When you're not sure about some information, you say that you don't have the information and don't make up anything.\
If the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \\\"What are some good restaurants around me?\\\" => \\\"Where are you?\\\" or \\\"When is the next flight to Tokyo\\\" => \\\"Where do you travel from?\\\")\" %}
{{- bos_token }}
{%- if messages[0]['role'] == 'system' %}
{%- set system_message = messages[0]['content'] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set system_message = default_system_message %}
{%- set loop_messages = messages %}
{%- endif %}
{%- set user_messages = loop_messages | selectattr('role', 'equalto', 'user') | list %}
{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}
{%- for message in loop_messages %}
{%- if message['role'] == 'user' %}
{%- if tools and (message == user_messages[-1]) %}
{{- '[AVAILABLE_TOOLS][' }}
{%- for tool in tools %}
{%- set tool = tool.function %}
{{- '{\"type\": \"function\", \"function\": {' }}
{%- for key, val in tool.items() if key != 'return' %}
{{- key|tojson + ': ' + val|tojson }}
{%- if not loop.last %}
{{- ', ' }}
{%- endif %}
{%- endfor %}
{{- '}}' }}
{%- if not loop.last %}
{{- ', ' }}
{%- else %}
{{- ']' }}
{%- endif %}
{%- endfor %}
{{- '[/AVAILABLE_TOOLS]' }}
{%- endif %}
{%- if message['content'] is string %}
{{- '[INST]' + message['content'] + '[/INST]' }}
{%- else %}
{{- '[INST]' }}
{%- for block in message['content'] %}
{%- if block['type'] == 'text' %}
{{- block['text'] }}
{%- elif block['type'] == 'image' or block['type'] == 'image_url' %}
{{- '[IMG]' }}
{%- else %}
{{- raise_exception('Only text and image blocks are supported in message content!') }}
{%- endif %}
{%- endfor %}
{{- '[/INST]' }}
{%- endif %}
{%- elif message['role'] == 'system' %}
{{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}
{%- elif message['role'] == 'assistant' %}
{{- message.content if message.content }}
{%- if message.tool_calls %}
{{- '[TOOL_CALLS][' }}
{%- for tool_call in message.tool_calls %}
{%- set out = tool_call.function|tojson %}
{{- out[:-1] }}
{{- ', \"id\": ' + tool_call.id|tojson if tool_call.id }}
{{- '}' }}
{%- if not loop.last %}
{{- ', ' }}
{%- else %}
{{- ']' }}
{%- endif %}
{%- endfor %}
{%- endif %}
{{- eos_token }}
{%- elif message['role'] == 'tool' %}
{%- if message.tool_call_id is undefined or message.tool_call_id is none %}
{{- raise_exception(\"Tool call must have a valid tool_call_id!\") }}
{%- endif %}
{%- if message.content and message.content.content is defined %}
{%- set content = message.content.content %}
{%- else %}
{%- set content = message.content %}
{%- endif %}
{{- '[TOOL_RESULTS]' }}
{{- message.tool_call_id }}
{{- '[TOOL_CONTENT]' + (content if content is string else content|tojson) + '[/TOOL_RESULTS]' }}
{%- else %}
{{- raise_exception('Only user, tool, system and assistant roles are supported!') }}
{%- endif %}
{%- endfor %}
I have just tested the new quants uploaded and it works!
curl http: //127.0.0.1:5000/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What time is it currently in New York City?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get current time in a specific timezones",
"parameters": {
"type": "object",
"required": [
"timezone"
],
"properties": {
"timezone": {
"type": "string",
"description": "IANA timezone name (e.g., America/New_York, Europe/London). Use Europe/Berlin as local timezone if no timezone provided by the user."
}
}
}
}
}
]
}' | jq
Response:
{
"id": "chatcmpl-1747923254299705344",
"object": "chat.completion",
"created": 1747923254,
"model": "Mistral-Small-3.1-24B-Instruct-2503-q5_k_l.gguf",
"choices": [
{
"index": 0,
"finish_reason": "tool_calls",
"message": {
"role": "assistant",
"content": "[{\"name\": \"get_current_time\", \"arguments\": {\"timezone\": \"America/New_York\"}}]"
},
"tool_calls": [
{
"type": "function",
"function": {
"name": "get_current_time",
"arguments": "{\"timezone\": \"America/New_York\"}"
},
"id": "call_lg0i3u37",
"index": "0"
}
]
}
],
"usage": {
"prompt_tokens": 123,
"completion_tokens": 24,
"total_tokens": 147
}
}
Thank you very much for the fast fix!
I have just tested the new quants uploaded and it works!
curl http: //127.0.0.1:5000/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What time is it currently in New York City?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_time", "description": "Get current time in a specific timezones", "parameters": { "type": "object", "required": [ "timezone" ], "properties": { "timezone": { "type": "string", "description": "IANA timezone name (e.g., America/New_York, Europe/London). Use Europe/Berlin as local timezone if no timezone provided by the user." } } } } } ] }' | jqResponse:
{ "id": "chatcmpl-1747923254299705344", "object": "chat.completion", "created": 1747923254, "model": "Mistral-Small-3.1-24B-Instruct-2503-q5_k_l.gguf", "choices": [ { "index": 0, "finish_reason": "tool_calls", "message": { "role": "assistant", "content": "[{\"name\": \"get_current_time\", \"arguments\": {\"timezone\": \"America/New_York\"}}]" }, "tool_calls": [ { "type": "function", "function": { "name": "get_current_time", "arguments": "{\"timezone\": \"America/New_York\"}" }, "id": "call_lg0i3u37", "index": "0" } ] } ], "usage": { "prompt_tokens": 123, "completion_tokens": 24, "total_tokens": 147 } }Thank you very much for the fast fix!
Thank you for the link to the template and testing
Mungert
changed discussion status to
closed