ganchito commited on
Commit
29ee19e
Β·
verified Β·
1 Parent(s): 510a5da

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +73 -60
README.md CHANGED
@@ -1,69 +1,82 @@
1
- # Dante-7B (GGUF) – for Ollama
2
 
3
- This repository contains a **quantized and GGUF-converted** version of the [Dante-7B](https://huggingface.co/outflanknl/Dante-7B) model (based on **Qwen2** architecture), optimized for use with [Ollama](https://ollama.ai) or any `llama.cpp`-compatible backend.
 
4
 
5
- ## πŸ“¦ Model origin
6
- - Original model: [`outflanknl/Dante-7B`](https://huggingface.co/outflanknl/Dante-7B)
7
- - Architecture: **Qwen2 7B**
8
- - Converted to **GGUF** format using the official [`llama.cpp`](https://github.com/ggerganov/llama.cpp) conversion script:
 
 
9
  ```bash
10
  python3 convert_hf_to_gguf.py /path/to/original/model --outfile Dante-7B.gguf
11
  ```
12
 
 
 
13
  ## πŸ”§ Quantization
14
- The model has been quantized to reduce size and improve inference speed while preserving quality.
15
-
16
- ## πŸ“‚ Repository contents
17
- - `Dante-7B.gguf` β†’ Model ready to use with Ollama/llama.cpp
18
- - Example `Modelfile` for Ollama
19
-
20
- ## πŸš€ Using in Ollama
21
-
22
- 1. Clone or download this repository:
23
- ```bash
24
- git lfs install
25
- git clone https://huggingface.co/ganchito/dante-7b.gguf
26
- cd dante-7b.gguf
27
- ```
28
-
29
- 2. Create a `Modelfile` (example included in this repo):
30
- ```dockerfile
31
- FROM Dante-7B.gguf
32
-
33
- # Model configuration
34
- PARAMETER stop "<|im_end|>"
35
- PARAMETER stop "<|endoftext|>"
36
- PARAMETER stop "<|im_start|>"
37
- PARAMETER stop "<|endoftext|>"
38
-
39
- SYSTEM \"\"\"You are Dante, a 7B parameter language model based on Qwen2 architecture.
40
- You are a helpful, creative, and intelligent AI assistant.
41
- You can engage in conversations, answer questions, help with tasks, and provide thoughtful responses.
42
- Always be respectful, honest, and helpful while maintaining a conversational and engaging tone.\"\"\"
43
-
44
- TEMPLATE \"\"\"{{ if .System }}<|im_start|>system
45
- {{ .System }}<|im_end|>
46
- {{ end }}{{ if .Prompt }}<|im_start|>user
47
- {{ .Prompt }}<|im_end|>
48
- {{ end }}<|im_start|>assistant
49
- {{ .Response }}<|im_end|>\"\"\"
50
-
51
- PARAMETER temperature 0.7
52
- PARAMETER top_p 0.9
53
- PARAMETER top_k 40
54
- PARAMETER repeat_penalty 1.1
55
- PARAMETER num_ctx 32768
56
- ```
57
-
58
- 3. Build the model in Ollama:
59
- ```bash
60
- ollama create dante-7b -f Modelfile
61
- ```
62
-
63
- 4. Run the model:
64
- ```bash
65
- ollama run dante-7b
66
- ```
 
 
 
 
 
 
 
67
 
68
  ## πŸ“œ License
69
- Refer to the original model's license in [outflanknl/Dante-7B](https://huggingface.co/outflanknl/Dante-7B).
 
 
1
+ # πŸ¦… Dante-7B (GGUF) – Optimized for Ollama
2
 
3
+ This repository provides a **quantized and GGUF-converted** version of the [Dante-7B](https://huggingface.co/outflanknl/Dante-7B) model, based on the **Qwen2 7B** architecture.
4
+ It is optimized for use with [Ollama](https://ollama.ai) πŸ’» or any backend compatible with [`llama.cpp`](https://github.com/ggerganov/llama.cpp).
5
 
6
+ ---
7
+
8
+ ## πŸ“¦ Model Origin
9
+ - **Base model**: [`outflanknl/Dante-7B`](https://huggingface.co/outflanknl/Dante-7B)
10
+ - **Architecture**: Qwen2 7B
11
+ - **Format conversion**: Performed with the official `llama.cpp` conversion script:
12
  ```bash
13
  python3 convert_hf_to_gguf.py /path/to/original/model --outfile Dante-7B.gguf
14
  ```
15
 
16
+ ---
17
+
18
  ## πŸ”§ Quantization
19
+ The model has been **quantized** to reduce memory usage and improve inference speed while keeping high-quality outputs.
20
+
21
+ ---
22
+
23
+ ## πŸ“‚ Files in This Repository
24
+ - `Dante-7B.gguf` β†’ Ready-to-use model file (GGUF format)
25
+ - Example **`Modelfile`** for Ollama
26
+
27
+ ---
28
+
29
+ ## πŸš€ Quick Start with Ollama
30
+
31
+ ### 1️⃣ Download the repository
32
+ ```bash
33
+ git lfs install
34
+ git clone https://huggingface.co/ganchito/dante-7b.gguf
35
+ cd dante-7b.gguf
36
+ ```
37
+
38
+ ### 2️⃣ Create a `Modelfile`
39
+ Example configuration:
40
+ ```dockerfile
41
+ FROM Dante-7B.gguf
42
+
43
+ # Model configuration
44
+ PARAMETER stop "<|im_end|>"
45
+ PARAMETER stop "<|endoftext|>"
46
+ PARAMETER stop "<|im_start|>"
47
+ PARAMETER stop "<|endoftext|>"
48
+
49
+ SYSTEM \"\"\"You are Dante, a 7B parameter language model based on Qwen2 architecture.
50
+ You are a helpful, creative, and intelligent AI assistant.
51
+ You can engage in conversations, answer questions, help with tasks, and provide thoughtful responses.
52
+ Always be respectful, honest, and helpful while maintaining a conversational and engaging tone.\"\"\"
53
+
54
+ TEMPLATE \"\"\"{{ if .System }}<|im_start|>system
55
+ {{ .System }}<|im_end|>
56
+ {{ end }}{{ if .Prompt }}<|im_start|>user
57
+ {{ .Prompt }}<|im_end|>
58
+ {{ end }}<|im_start|>assistant
59
+ {{ .Response }}<|im_end|>\"\"\"
60
+
61
+ PARAMETER temperature 0.7
62
+ PARAMETER top_p 0.9
63
+ PARAMETER top_k 40
64
+ PARAMETER repeat_penalty 1.1
65
+ PARAMETER num_ctx 32768
66
+ ```
67
+
68
+ ### 3️⃣ Build the model in Ollama
69
+ ```bash
70
+ ollama create dante-7b -f Modelfile
71
+ ```
72
+
73
+ ### 4️⃣ Run the model
74
+ ```bash
75
+ ollama run dante-7b
76
+ ```
77
+
78
+ ---
79
 
80
  ## πŸ“œ License
81
+ This GGUF version is subject to the **same license** as the original [Dante-7B](https://huggingface.co/outflanknl/Dante-7B) model.
82
+