Hugging Face's logo

InferenceIllusionist
/
Mixtral-Instruct-ITR-8x7B-GGUF

GGUF
Not-For-All-Audiences
conversational
Model card Files Community
InferenceIllusionist commited on
Commit
c5b96f8
·
verified ·
1 Parent(s): dcde20a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -0
README.md CHANGED
@@ -22,6 +22,18 @@ Starting out with Q4_K_M, and iterating from there. Future plans for imatrix/IQ
22
 
23
  First time doing quantizations so any feedback is greatly appreciated.
24
 
 
 
 
 
 
 
 
 
 
 
 
 
25
  Original model card below for reference.
26
 
27
  ---
 
22
 
23
  First time doing quantizations so any feedback is greatly appreciated.
24
 
25
+ | Name | Quant method | Bits |
26
+ | ---- | ---- | ---- |
27
+ | [Mixtral-Instruct-ITR-8x7B.Q2_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q2_K.gguf) | Q2_K | 2 |
28
+ | [Mixtral-Instruct-ITR-8x7B.Q3_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_S.gguf) | Q3_K_S | 3 |
29
+ | [Mixtral-Instruct-ITR-8x7B.Q3_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_M.gguf) | Q3_K_M | 3 |
30
+ | [Mixtral-Instruct-ITR-8x7B.Q3_K_L.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q3_K_L.gguf) | Q3_K_L | 4 |
31
+ | [Mixtral-Instruct-ITR-8x7B.Q4_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q4_K_M.gguf) | Q4_K_M | 5 |
32
+ | [Mixtral-Instruct-ITR-8x7B.Q5_K_S.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_S.gguf) | Q5_K_S | 5 |
33
+ | [Mixtral-Instruct-ITR-8x7B.Q5_K_M.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q5_K_M.gguf) | Q6_K_M | 5 |
34
+ | [Mixtral-Instruct-ITR-8x7B.Q6_K.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q6_K.gguf) | Q6_K | 6 | 49.62 GB|
35
+ | [Mixtral-Instruct-ITR-8x7B.Q8_0.gguf](https://huggingface.co/InferenceIllusionist/Mixtral-Instruct-ITR-8x7B-GGUF/blob/main/Mixtral-Instruct-ITR-8x7B-Q8_0.gguf) | Q8_0 | 8 | 49.62 GB|
36
+
37
  Original model card below for reference.
38
 
39
  ---