File size: 1,330 Bytes
47bc728 d45a7b9 6967e9f 47bc728 cc7fe76 24cdd9f cc7fe76 3fe1443 cc7fe76 c6b5514 404fc42 36c6cdc 404fc42 36c6cdc 404fc42 17c26ad cc7fe76 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
---
license: apache-2.0
language:
- zh
- en
tags:
- moe
---
# Chinese-Mixtral-Instruct-GGUF
<p align="center">
<img src="https://ymcui.com/images/chinese-mixtral-banner.png" width="600"/>
</p>
This repository contains the GGUF-v3 models (llama.cpp compatible) for **Chinese-Mixtral-Instruct** (chat/instruction model).
**Note: When using instruction/chat model, you MUST follow the official prompt template! Example: [chat.sh](https://github.com/ymcui/Chinese-Mixtral/blob/main/scripts/llamacpp/chat.sh)**
## Performance
Metric: PPL, lower is better
| Quant | PPL |
| ----- | ---- |
| IQ2_XXS | - |
| IQ2_XS | 7.4175 +/- 0.08420 |
| Q2_K | 4.5758 +/- 0.03959 |
| Q3_K | 4.5563 +/- 0.04126 |
| Q4_0 | 3.9757 +/- 0.03455 |
| Q4_K | 3.9265 +/- 0.03407 |
| Q5_0 | 3.9167 +/- 0.03399 |
| Q5_K | 3.9232 +/- 0.03403 |
| Q6_K | 3.9242 +/- 0.03415 |
| Q8_0 | 3.9159 +/- 0.03402 |
| F16 | x |
Due to the file size limitation, for F16 model, please use `cat` command to concatenate all parts into a single file. **You must concatenate these parts in order.**
## Others
For Hugging Face version, please see: https://huggingface.co/hfl/chinese-mixtral-instruct
Please refer to [https://github.com/ymcui/Chinese-Mixtral/](https://github.com/ymcui/Chinese-Mixtral/) for more details. |