Hugging Face's logo

ThomasBaruzier
/
EXAONE-3.5-7.8B-Instruct-GGUF

Text Generation
Transformers
GGUF
English
Korean
lg-ai
exaone
exaone-3.5
conversational
Model card Files Community
ThomasBaruzier commited on
Commit
65b6ef6
·
verified ·
1 Parent(s): 5ad0d86

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +31 -33
README.md CHANGED
@@ -13,9 +13,7 @@ pipeline_tag: text-generation
13
  library_name: transformers
14
  ---
15
 
16
- <hr>
17
-
18
- <p align="left"><img src="https://cdn-uploads.huggingface.co/production/uploads/646410e04bf9122922289dc7/YSQx1nvIpnwpW1icO3Nby.jpeg", width="50%"><br>
19
 
20
  # Llama.cpp imatrix quantizations of [LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct](https://huggingface.co/LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct)
21
 
@@ -27,36 +25,36 @@ All quants were made using the imatrix option and Bartowski's [calibration file]
27
 
28
  # Perplexity table (the lower the better)
29
 
30
- | Quant | Size (MB) | PPL | Size (%) | Accuracy (%) | PPL error rate |
31
- | ------ | --------- | ------- | -------- | ------------ | -------------- |
32
- | IQ1_S | 1820 | 26.3205 | 12.20 | 33.81 | 0.40 |
33
- | IQ1_M | 1955 | 19.0360 | 13.10 | 46.75 | 0.28 |
34
- | IQ2_XXS | 2182 | 13.3276 | 14.63 | 66.77 | 0.20 |
35
- | IQ2_XS | 2379 | 11.7742 | 15.95 | 75.58 | 0.18 |
36
- | IQ2_S | 2514 | 11.3084 | 16.85 | 78.69 | 0.17 |
37
- | IQ2_M | 2695 | 10.3850 | 18.07 | 85.69 | 0.16 |
38
- | Q2_K_S | 2730 | 11.2910 | 18.30 | 78.82 | 0.17 |
39
- | Q2_K | 2912 | 11.1386 | 19.52 | 79.89 | 0.17 |
40
- | IQ3_XXS | 3006 | 9.5453 | 20.15 | 93.23 | 0.14 |
41
- | IQ3_XS | 3226 | 9.2103 | 21.63 | 96.62 | 0.14 |
42
- | Q3_K_S | 3365 | 10.0571 | 22.56 | 88.49 | 0.16 |
43
- | IQ3_S | 3382 | 9.2420 | 22.67 | 96.29 | 0.14 |
44
- | IQ3_M | 3479 | 9.0709 | 23.32 | 98.11 | 0.13 |
45
- | Q3_K_M | 3703 | 9.2078 | 24.82 | 96.65 | 0.14 |
46
- | Q3_K_L | 3992 | 9.1908 | 26.76 | 96.83 | 0.14 |
47
- | IQ4_XS | 4101 | 9.0166 | 27.49 | 98.70 | 0.14 |
48
- | Q4_0 | 4316 | 9.4186 | 28.93 | 94.49 | 0.14 |
49
- | IQ4_NL | 4318 | 9.0297 | 28.95 | 98.55 | 0.14 |
50
- | Q4_K_S | 4332 | 8.9634 | 29.04 | 99.28 | 0.13 |
51
- | Q4_K_M | 4549 | 8.9107 | 30.50 | 99.87 | 0.13 |
52
- | Q4_1 | 4743 | 8.9614 | 31.80 | 99.31 | 0.13 |
53
- | Q5_K_S | 5184 | 8.9042 | 34.75 | 99.94 | 0.13 |
54
- | Q5_0 | 5198 | 9.0533 | 34.85 | 98.30 | 0.14 |
55
- | Q5_K_M | 5311 | 8.9100 | 35.60 | 99.88 | 0.13 |
56
- | Q5_1 | 5625 | 8.9230 | 37.71 | 99.73 | 0.13 |
57
- | Q6_K | 6121 | 8.8800 | 41.03 | 100.22 | 0.13 |
58
- | Q8_0 | 7927 | 8.8534 | 53.14 | 100.52 | 0.13 |
59
- | F16 | 14917 | 8.8992 | 100 | 100 | 0.13 |
60
 
61
  <hr>
62
 
 
13
  library_name: transformers
14
  ---
15
 
16
+ <br><img src="https://cdn-uploads.huggingface.co/production/uploads/646410e04bf9122922289dc7/b1ahV_SP1O43rTXACjmyz.webp" width="720"><br>
 
 
17
 
18
  # Llama.cpp imatrix quantizations of [LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct](https://huggingface.co/LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct)
19
 
 
25
 
26
  # Perplexity table (the lower the better)
27
 
28
+ | Quant | Size (MB) | PPL | Size (%) | Accuracy (%) | PPL error rate |
29
+ | ------------------------------------------------------------------------------------------------------------------------------ | --------- | ------- | -------- | ------------ | -------------- |
30
+ | [IQ1_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ1_S.gguf) | 1820 | 26.3205 | 12.20 | 33.81 | 0.40 |
31
+ | [IQ1_M](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ1_M.gguf) | 1955 | 19.0360 | 13.10 | 46.75 | 0.28 |
32
+ | [IQ2_XXS](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ2_XXS.gguf) | 2182 | 13.3276 | 14.63 | 66.77 | 0.20 |
33
+ | [IQ2_XS](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ2_XS.gguf) | 2379 | 11.7742 | 15.95 | 75.58 | 0.18 |
34
+ | [IQ2_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ2_S.gguf) | 2514 | 11.3084 | 16.85 | 78.69 | 0.17 |
35
+ | [IQ2_M](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ2_M.gguf) | 2695 | 10.3850 | 18.07 | 85.69 | 0.16 |
36
+ | [Q2_K_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q2_K_S.gguf) | 2730 | 11.2910 | 18.30 | 78.82 | 0.17 |
37
+ | [Q2_K](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q2_K.gguf) | 2912 | 11.1386 | 19.52 | 79.89 | 0.17 |
38
+ | [IQ3_XXS](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ3_XXS.gguf) | 3006 | 9.5453 | 20.15 | 93.23 | 0.14 |
39
+ | [IQ3_XS](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ3_XS.gguf) | 3226 | 9.2103 | 21.63 | 96.62 | 0.14 |
40
+ | [Q3_K_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q3_K_S.gguf) | 3365 | 10.0571 | 22.56 | 88.49 | 0.16 |
41
+ | [IQ3_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ3_S.gguf) | 3382 | 9.2420 | 22.67 | 96.29 | 0.14 |
42
+ | [IQ3_M](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ3_M.gguf) | 3479 | 9.0709 | 23.32 | 98.11 | 0.13 |
43
+ | [Q3_K_M](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q3_K_M.gguf) | 3703 | 9.2078 | 24.82 | 96.65 | 0.14 |
44
+ | [Q3_K_L](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q3_K_L.gguf) | 3992 | 9.1908 | 26.76 | 96.83 | 0.14 |
45
+ | [IQ4_XS](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ4_XS.gguf) | 4101 | 9.0166 | 27.49 | 98.70 | 0.14 |
46
+ | [Q4_0](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q4_0.gguf) | 4316 | 9.4186 | 28.93 | 94.49 | 0.14 |
47
+ | [IQ4_NL](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-IQ4_NL.gguf) | 4318 | 9.0297 | 28.95 | 98.55 | 0.14 |
48
+ | [Q4_K_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q4_K_S.gguf) | 4332 | 8.9634 | 29.04 | 99.28 | 0.13 |
49
+ | [Q4_K_M](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q4_K_M.gguf) | 4549 | 8.9107 | 30.50 | 99.87 | 0.13 |
50
+ | [Q4_1](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q4_1.gguf) | 4743 | 8.9614 | 31.80 | 99.31 | 0.13 |
51
+ | [Q5_K_S](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q5_K_S.gguf) | 5184 | 8.9042 | 34.75 | 99.94 | 0.13 |
52
+ | [Q5_0](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q5_0.gguf) | 5198 | 9.0533 | 34.85 | 98.30 | 0.14 |
53
+ | [Q5_K_M](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q5_K_M.gguf) | 5311 | 8.9100 | 35.60 | 99.88 | 0.13 |
54
+ | [Q5_1](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q5_1.gguf) | 5625 | 8.9230 | 37.71 | 99.73 | 0.13 |
55
+ | [Q6_K](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q6_K.gguf) | 6121 | 8.8800 | 41.03 | 100.22 | 0.13 |
56
+ | [Q8_0](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-Q8_0.gguf) | 7927 | 8.8534 | 53.14 | 100.52 | 0.13 |
57
+ | [F16](https://huggingface.co/ThomasBaruzier/EXAONE-3.5-7.8B-Instruct-GGUF/blob/main/EXAONE-3.5-7.8B-Instruct-F16.gguf) | 14917 | 8.8992 | 100 | 100 | 0.13 |
58
 
59
  <hr>
60