Chinese-LLaMA-2-13B-16K-GGUF

This repository contains the GGUF-v3 models (llama.cpp compatible) for Chinese-LLaMA-2-13B-16K.

Performance

Metric: PPL, lower is better

Quant	original	imatrix (-im)
Q2_K	11.8958 +/- 0.20739	13.0017 +/- 0.23003
Q3_K	9.7130 +/- 0.17037	9.3443 +/- 0.16582
Q4_0	9.2002 +/- 0.16219	-
Q4_K	9.0055 +/- 0.15918	8.9848 +/- 0.15908
Q5_0	8.8441 +/- 0.15690	-
Q5_K	8.8999 +/- 0.15751	8.8983 +/- 0.15753
Q6_K	8.8944 +/- 0.15776	8.8833 +/- 0.15760
Q8_0	8.8745 +/- 0.15745	-
F16	8.8687 +/- 0.15729	-

The model with -im suffix is generated with important matrix, which has generally better performance (not always though).

Others

For Hugging Face version, please see: https://huggingface.co/hfl/chinese-llama-2-13b-16k

Please refer to https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/ for more details.