InferenceIllusionist commited on
Commit
a89d6a1
·
verified ·
1 Parent(s): c5b96f8

update readme table

Browse files
Files changed (1) hide show
  1. README.md +13 -11
README.md CHANGED
@@ -22,17 +22,19 @@ Starting out with Q4_K_M, and iterating from there. Future plans for imatrix/IQ
22
 
23
  First time doing quantizations so any feedback is greatly appreciated.
24
 
25
- | Name | Quant method | Bits |
26
- | ---- | ---- | ---- |
27
- | [Mixtral-Instruct-ITR-8x7B.Q2_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q2_K.gguf) | Q2_K | 2 |
28
- | [Mixtral-Instruct-ITR-8x7B.Q3_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_S.gguf) | Q3_K_S | 3 |
29
- | [Mixtral-Instruct-ITR-8x7B.Q3_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_M.gguf) | Q3_K_M | 3 |
30
- | [Mixtral-Instruct-ITR-8x7B.Q3_K_L.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_L.gguf) | Q3_K_L | 4 |
31
- | [Mixtral-Instruct-ITR-8x7B.Q4_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q4_K_M.gguf) | Q4_K_M | 5 |
32
- | [Mixtral-Instruct-ITR-8x7B.Q5_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_S.gguf) | Q5_K_S | 5 |
33
- | [Mixtral-Instruct-ITR-8x7B.Q5_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_M.gguf) | Q6_K_M | 5 |
34
- | [Mixtral-Instruct-ITR-8x7B.Q6_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q6_K.gguf) | Q6_K | 6 | 49.62 GB|
35
- | [Mixtral-Instruct-ITR-8x7B.Q8_0.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q8_0.gguf) | Q8_0 | 8 | 49.62 GB|
 
 
36
 
37
  Original model card below for reference.
38
 
 
22
 
23
  First time doing quantizations so any feedback is greatly appreciated.
24
 
25
+ | Name | Quant method | Bits |ppl*
26
+ | ---- | ---- | ---- | ---- |
27
+ | [Mixtral-Instruct-ITR-8x7B.Q2_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q2_K.gguf) | Q2_K | 2 | +0.6717 ppl|
28
+ | [Mixtral-Instruct-ITR-8x7B.Q3_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_S.gguf) | Q3_K_S | 3 | +0.5551 ppl|
29
+ | [Mixtral-Instruct-ITR-8x7B.Q3_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_M.gguf) | Q3_K_M | 3 | +0.2496 ppl|
30
+ | [Mixtral-Instruct-ITR-8x7B.Q3_K_L.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_L.gguf) | Q3_K_L | 4 | +0.1764 ppl|
31
+ | [Mixtral-Instruct-ITR-8x7B.Q4_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q4_K_M.gguf) | Q4_K_M | 5 | +0.0532 ppl|
32
+ | [Mixtral-Instruct-ITR-8x7B.Q5_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_S.gguf) | Q5_K_S | 5 | +0.0400 ppl|
33
+ | [Mixtral-Instruct-ITR-8x7B.Q5_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_M.gguf) | Q5_K_M | 6 | +0.0122 ppl|
34
+ | [Mixtral-Instruct-ITR-8x7B.Q6_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q6_K.gguf) | Q6_K | 6 | +0.008 ppl|
35
+ | [Mixtral-Instruct-ITR-8x7B.Q8_0.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q8_0.gguf) | Q8_0 | 8 | +0.004 ppl|
36
+
37
+ *Perplexity @ LLaMA-v1-7B for reference
38
 
39
  Original model card below for reference.
40