Update README.md
Browse files
README.md
CHANGED
@@ -1,69 +1,82 @@
|
|
1 |
-
# Dante-7B (GGUF) β for Ollama
|
2 |
|
3 |
-
This repository
|
|
|
4 |
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
-
|
|
|
|
|
9 |
```bash
|
10 |
python3 convert_hf_to_gguf.py /path/to/original/model --outfile Dante-7B.gguf
|
11 |
```
|
12 |
|
|
|
|
|
13 |
## π§ Quantization
|
14 |
-
The model has been quantized to reduce
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
|
20 |
-
|
21 |
-
|
22 |
-
|
23 |
-
|
24 |
-
|
25 |
-
|
26 |
-
|
27 |
-
|
28 |
-
|
29 |
-
|
30 |
-
|
31 |
-
|
32 |
-
|
33 |
-
|
34 |
-
|
35 |
-
|
36 |
-
|
37 |
-
|
38 |
-
|
39 |
-
|
40 |
-
|
41 |
-
|
42 |
-
|
43 |
-
|
44 |
-
|
45 |
-
|
46 |
-
|
47 |
-
|
48 |
-
|
49 |
-
|
50 |
-
|
51 |
-
|
52 |
-
|
53 |
-
|
54 |
-
|
55 |
-
|
56 |
-
|
57 |
-
|
58 |
-
|
59 |
-
|
60 |
-
|
61 |
-
|
62 |
-
|
63 |
-
|
64 |
-
|
65 |
-
|
66 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
67 |
|
68 |
## π License
|
69 |
-
|
|
|
|
1 |
+
# π¦
Dante-7B (GGUF) β Optimized for Ollama
|
2 |
|
3 |
+
This repository provides a **quantized and GGUF-converted** version of the [Dante-7B](https://huggingface.co/outflanknl/Dante-7B) model, based on the **Qwen2 7B** architecture.
|
4 |
+
It is optimized for use with [Ollama](https://ollama.ai) π» or any backend compatible with [`llama.cpp`](https://github.com/ggerganov/llama.cpp).
|
5 |
|
6 |
+
---
|
7 |
+
|
8 |
+
## π¦ Model Origin
|
9 |
+
- **Base model**: [`outflanknl/Dante-7B`](https://huggingface.co/outflanknl/Dante-7B)
|
10 |
+
- **Architecture**: Qwen2 7B
|
11 |
+
- **Format conversion**: Performed with the official `llama.cpp` conversion script:
|
12 |
```bash
|
13 |
python3 convert_hf_to_gguf.py /path/to/original/model --outfile Dante-7B.gguf
|
14 |
```
|
15 |
|
16 |
+
---
|
17 |
+
|
18 |
## π§ Quantization
|
19 |
+
The model has been **quantized** to reduce memory usage and improve inference speed while keeping high-quality outputs.
|
20 |
+
|
21 |
+
---
|
22 |
+
|
23 |
+
## π Files in This Repository
|
24 |
+
- `Dante-7B.gguf` β Ready-to-use model file (GGUF format)
|
25 |
+
- Example **`Modelfile`** for Ollama
|
26 |
+
|
27 |
+
---
|
28 |
+
|
29 |
+
## π Quick Start with Ollama
|
30 |
+
|
31 |
+
### 1οΈβ£ Download the repository
|
32 |
+
```bash
|
33 |
+
git lfs install
|
34 |
+
git clone https://huggingface.co/ganchito/dante-7b.gguf
|
35 |
+
cd dante-7b.gguf
|
36 |
+
```
|
37 |
+
|
38 |
+
### 2οΈβ£ Create a `Modelfile`
|
39 |
+
Example configuration:
|
40 |
+
```dockerfile
|
41 |
+
FROM Dante-7B.gguf
|
42 |
+
|
43 |
+
# Model configuration
|
44 |
+
PARAMETER stop "<|im_end|>"
|
45 |
+
PARAMETER stop "<|endoftext|>"
|
46 |
+
PARAMETER stop "<|im_start|>"
|
47 |
+
PARAMETER stop "<|endoftext|>"
|
48 |
+
|
49 |
+
SYSTEM \"\"\"You are Dante, a 7B parameter language model based on Qwen2 architecture.
|
50 |
+
You are a helpful, creative, and intelligent AI assistant.
|
51 |
+
You can engage in conversations, answer questions, help with tasks, and provide thoughtful responses.
|
52 |
+
Always be respectful, honest, and helpful while maintaining a conversational and engaging tone.\"\"\"
|
53 |
+
|
54 |
+
TEMPLATE \"\"\"{{ if .System }}<|im_start|>system
|
55 |
+
{{ .System }}<|im_end|>
|
56 |
+
{{ end }}{{ if .Prompt }}<|im_start|>user
|
57 |
+
{{ .Prompt }}<|im_end|>
|
58 |
+
{{ end }}<|im_start|>assistant
|
59 |
+
{{ .Response }}<|im_end|>\"\"\"
|
60 |
+
|
61 |
+
PARAMETER temperature 0.7
|
62 |
+
PARAMETER top_p 0.9
|
63 |
+
PARAMETER top_k 40
|
64 |
+
PARAMETER repeat_penalty 1.1
|
65 |
+
PARAMETER num_ctx 32768
|
66 |
+
```
|
67 |
+
|
68 |
+
### 3οΈβ£ Build the model in Ollama
|
69 |
+
```bash
|
70 |
+
ollama create dante-7b -f Modelfile
|
71 |
+
```
|
72 |
+
|
73 |
+
### 4οΈβ£ Run the model
|
74 |
+
```bash
|
75 |
+
ollama run dante-7b
|
76 |
+
```
|
77 |
+
|
78 |
+
---
|
79 |
|
80 |
## π License
|
81 |
+
This GGUF version is subject to the **same license** as the original [Dante-7B](https://huggingface.co/outflanknl/Dante-7B) model.
|
82 |
+
|