benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF_cline

benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF

This model was converted to GGUF format from rombodawg/Rombos-Coder-V2.5-Qwen-7b using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Use with Cline and Ollama

use this template file from https://github.com/maryasov/ollama-models-instruct-for-cline

FROM rombos-coder-v2.5-qwen-7b-q8_0.gguf
TEMPLATE """{{- /* Initial system message with core instructions */ -}}
{{- if .Messages }}
{{- if or .System .Tools }}
<|im_start|>system
{{- if .System }}
{{ .System }}
{{- end }}                                                                                                                                                                                                  {{- if .Tools }}
# Tools and XML Schema
You have access to the following tools. Each tool must be used according to this XML schema:

<tools>
{{- range .Tools }}
{{ .Function }}
{{- end }}
</tools>

## Tool Use Format
1. Think about the approach in <thinking> tags
2. Call tool using XML format:
   <tool_name>
     <param_name>value</param_name>
   </tool_name>
3. Process tool response from:
   <tool_response>result</tool_response>
{{- end }}
<|im_end|>
{{- end }}

{{- /* Message handling loop */ -}}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}

{{- /* User messages */ -}}
{{- if eq .Role "user" }}
<|im_start|>user
{{ .Content }}
<|im_end|>

{{- /* Assistant messages */ -}}
{{- else if eq .Role "assistant" }}
<|im_start|>assistant
{{- if .Content }}
{{ .Content }}
{{- else if .ToolCalls }}
{{- range .ToolCalls }}
<thinking>
[Analysis of current state and next steps]
</thinking>

<{{ .Function.Name }}>
{{- range $key, $value := .Function.Arguments }}
<{{ $key }}>{{ $value }}</{{ $key }}>
{{- end }}
</{{ .Function.Name }}>
{{- end }}
{{- end }}
{{- if not $last }}<|im_end|>{{- end }}

{{- /* Tool response handling */ -}}
{{- else if eq .Role "tool" }}
<|im_start|>user
<tool_response>
{{ .Content }}
</tool_response>
<|im_end|>
{{- end }}

{{- /* Prepare for next assistant response if needed */ -}}
{{- if and (ne .Role "assistant") $last }}
<|im_start|>assistant
{{- end }}
{{- end }}

{{- /* Handle single message case */ -}}
{{- else }}
{{- if .System }}
<|im_start|>system
{{ .System }}
<|im_end|>
{{- end }}

{{- if .Prompt }}
<|im_start|>user
{{ .Prompt }}
<|im_end|>
{{- end }}

<|im_start|>assistant
{{- end }}
{{ .Response }}
{{- if .Response }}<|im_end|>{{- end }}
"""
PARAMETER repeat_last_n 64
PARAMETER repeat_penalty 1.1
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.1
PARAMETER top_k 40
PARAMETER top_p 0.9

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -c 2048