InferenceIllusionist
commited on
update readme table
Browse files
README.md
CHANGED
@@ -22,17 +22,19 @@ Starting out with Q4_K_M, and iterating from there. Future plans for imatrix/IQ
|
|
22 |
|
23 |
First time doing quantizations so any feedback is greatly appreciated.
|
24 |
|
25 |
-
| Name | Quant method | Bits |
|
26 |
-
| ---- | ---- | ---- |
|
27 |
-
| [Mixtral-Instruct-ITR-8x7B.Q2_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q2_K.gguf) | Q2_K | 2 |
|
28 |
-
| [Mixtral-Instruct-ITR-8x7B.Q3_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_S.gguf) | Q3_K_S | 3 |
|
29 |
-
| [Mixtral-Instruct-ITR-8x7B.Q3_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_M.gguf) | Q3_K_M | 3 |
|
30 |
-
| [Mixtral-Instruct-ITR-8x7B.Q3_K_L.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_L.gguf) | Q3_K_L | 4 |
|
31 |
-
| [Mixtral-Instruct-ITR-8x7B.Q4_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q4_K_M.gguf) | Q4_K_M | 5 |
|
32 |
-
| [Mixtral-Instruct-ITR-8x7B.Q5_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_S.gguf) | Q5_K_S | 5 |
|
33 |
-
| [Mixtral-Instruct-ITR-8x7B.Q5_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_M.gguf) |
|
34 |
-
| [Mixtral-Instruct-ITR-8x7B.Q6_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q6_K.gguf) | Q6_K | 6 |
|
35 |
-
| [Mixtral-Instruct-ITR-8x7B.Q8_0.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q8_0.gguf) | Q8_0 | 8 |
|
|
|
|
|
36 |
|
37 |
Original model card below for reference.
|
38 |
|
|
|
22 |
|
23 |
First time doing quantizations so any feedback is greatly appreciated.
|
24 |
|
25 |
+
| Name | Quant method | Bits |ppl*
|
26 |
+
| ---- | ---- | ---- | ---- |
|
27 |
+
| [Mixtral-Instruct-ITR-8x7B.Q2_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q2_K.gguf) | Q2_K | 2 | +0.6717 ppl|
|
28 |
+
| [Mixtral-Instruct-ITR-8x7B.Q3_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_S.gguf) | Q3_K_S | 3 | +0.5551 ppl|
|
29 |
+
| [Mixtral-Instruct-ITR-8x7B.Q3_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_M.gguf) | Q3_K_M | 3 | +0.2496 ppl|
|
30 |
+
| [Mixtral-Instruct-ITR-8x7B.Q3_K_L.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_L.gguf) | Q3_K_L | 4 | +0.1764 ppl|
|
31 |
+
| [Mixtral-Instruct-ITR-8x7B.Q4_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q4_K_M.gguf) | Q4_K_M | 5 | +0.0532 ppl|
|
32 |
+
| [Mixtral-Instruct-ITR-8x7B.Q5_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_S.gguf) | Q5_K_S | 5 | +0.0400 ppl|
|
33 |
+
| [Mixtral-Instruct-ITR-8x7B.Q5_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_M.gguf) | Q5_K_M | 6 | +0.0122 ppl|
|
34 |
+
| [Mixtral-Instruct-ITR-8x7B.Q6_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q6_K.gguf) | Q6_K | 6 | +0.008 ppl|
|
35 |
+
| [Mixtral-Instruct-ITR-8x7B.Q8_0.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q8_0.gguf) | Q8_0 | 8 | +0.004 ppl|
|
36 |
+
|
37 |
+
*Perplexity @ LLaMA-v1-7B for reference
|
38 |
|
39 |
Original model card below for reference.
|
40 |
|