|
--- |
|
license: apache-2.0 |
|
language: |
|
- zh |
|
- en |
|
tags: |
|
- moe |
|
--- |
|
|
|
# Chinese-Mixtral-Instruct-GGUF |
|
<p align="center"> |
|
<a href="https://github.com/ymcui/Chinese-Mixtral"><img src="https://ymcui.com/images/chinese-mixtral-banner.png" width="600"/></a> |
|
</p> |
|
|
|
**Chinese Mixtral GitHub repository: https://github.com/ymcui/Chinese-Mixtral** |
|
|
|
This repository contains the GGUF-v3 models (llama.cpp compatible) for **Chinese-Mixtral-Instruct** (chat/instruction model). |
|
|
|
**Note: When using instruction/chat model, you MUST follow the official prompt template! Example: [chat.sh](https://github.com/ymcui/Chinese-Mixtral/blob/main/scripts/llamacpp/chat.sh)** |
|
|
|
## Performance |
|
|
|
Metric: PPL, lower is better |
|
|
|
| Quant | Size ↓ | PPL | |
|
| ------- | ------- | ------------------ | |
|
| IQ1_S | 9.8 GB | 9.5782 +/- 0.08909 | |
|
| IQ1_M | 10.8 GB | 7.4666 +/- 0.06741 | |
|
| IQ2_XXS | 12.3 GB | 6.3923 +/- 0.05674 | |
|
| IQ2_XS | 13.7 GB | 6.0606 +/- 0.05834 | |
|
| IQ2_S | 14.1 GB | 4.7617 +/- 0.04177 | |
|
| IQ2_M | 15.5 GB | 4.5911 +/- 0.04054 | |
|
| Q2_K | 17.3 GB | 4.8592 +/- 0.04303 | |
|
| IQ3_XXS | 18.3 GB | 4.3557 +/- 0.03846 | |
|
| IQ3_XS | 19.3 GB | 4.3328 +/- 0.03779 | |
|
| IQ3_S | 20.4 GB | 4.3138 +/- 0.03785 | |
|
| IQ3_M | 21.4 GB | 4.3024 +/- 0.03775 | |
|
| Q3_K | 22.5 GB | 4.4334 +/- 0.03937 | |
|
| IQ4_XS | 25.1 GB | 4.2324 +/- 0.03757 | |
|
| Q4_0 | 26.4 GB | 4.2688 +/- 0.03787 | |
|
| IQ4_NL | 26.5 GB | 4.2384 +/- 0.03763 | |
|
| Q4_K | 28.4 GB | 4.2433 +/- 0.03768 | |
|
| Q5_0 | 32.2 GB | 4.2142 +/- 0.03733 | |
|
| Q5_K | 33.2 GB | 4.2177 +/- 0.03743 | |
|
| Q6_K | 38.4 GB | 4.2184 +/- 0.03754 | |
|
| Q8_0 | 49.6 GB | 4.2053 +/- 0.03732 | |
|
| F16 | 93.5 GB | x | |
|
|
|
Due to the file size limitation, for F16 model, please use `cat` command to concatenate all parts into a single file. **You must concatenate these parts in order.** |
|
|
|
|
|
## Others |
|
|
|
For Hugging Face version, please see: https://huggingface.co/hfl/chinese-mixtral-instruct |
|
|
|
Please refer to [https://github.com/ymcui/Chinese-Mixtral/](https://github.com/ymcui/Chinese-Mixtral/) for more details. |
|
|
|
|
|
## Citation |
|
|
|
Please consider cite our paper if you use the resource of this repository. |
|
Paper link: https://arxiv.org/abs/2403.01851 |
|
``` |
|
@article{chinese-mixtral, |
|
title={Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral}, |
|
author={Cui, Yiming and Yao, Xin}, |
|
journal={arXiv preprint arXiv:2403.01851}, |
|
url={https://arxiv.org/abs/2403.01851}, |
|
year={2024} |
|
} |
|
``` |
