File size: 7,528 Bytes
bc32e07 30a0b69 bc32e07 30a0b69 6aa5bab 30a0b69 f2eb604 30a0b69 bc32e07 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 |
---
quantized_by: ubergarm
pipeline_tag: text-generation
base_model: zai-org/GLM-4.5-Air
license: mit
base_model_relation: quantized
tags:
- imatrix
- conversational
- ik_llama.cpp
---
This is an experimental place-holder with an imatrix not for general purpose use just yet. I'm not releasing any quants for this just yet until the various PRs are in place and tested better.
Check the References below for the the github discussion as folks are working on adding support for this model.
Keep an eye out for new PR and follow along, once this thing is tested and considered working correctly I hope to release some quants for both this smaller Air model and the larger one too..
## `ik_llama.cpp` imatrix Quantizations of zai-org/GLM-4.5-Air
This quant collection **REQUIRES** [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp/) fork to support the ik's latest SOTA quants and optimizations! Do **not** download these big files and expect them to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc!
*NOTE* `ik_llama.cpp` can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.
Some of ik's new quants are supported with [Nexesenex/croco.cpp](https://github.com/Nexesenex/croco.cpp) fork of KoboldCPP.
These quants provide best in class perplexity for the given memory footprint.
## Big Thanks
Shout out to Wendell and the **Level1Techs** crew, the community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs)! **BIG thanks** for providing **BIG hardware** expertise and access to run these experiments and make these great quants available to the community!!!
Also thanks to all the folks in the quanting and inferencing community on [BeaverAI Club Discord](https://huggingface.co/BeaverAI) and on [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) for tips and tricks helping each other run, test, and benchmark all the fun new models!
## Quant Collection
Perplexity computed against *wiki.test.raw*.
These first two are just test quants for baseline perplexity comparison:
* `BF16` 203.436 GiB (16.004 BPW)
- Final estimate: PPL = TODO
* `Q8_0` 108.119 GiB (8.505 BPW)
- Final estimate: PPL = TODO
## IQ4_KSS 54.124 GiB (4.258 BPW)
<details>
<summary>👈 Secret Recipe</summary>
```bash
#!/usr/bin/env bash
# 620756992 | 4096, 151552, 1, 1 | Q8_0 | token_embd.weight
#
# 44826624 | 10944, 4096, 1, 1 | Q8_0 | blk.0.ffn_down.weight
# 44826624 | 4096, 10944, 1, 1 | Q8_0 | blk.0.ffn_gate.weight
# 44826624 | 4096, 10944, 1, 1 | Q8_0 | blk.0.ffn_up.weight
# 4096 | 4096, 1, 1, 1 | F32 | blk.0.attn_norm.weight
# 4096 | 4096, 1, 1, 1 | F32 | blk.0.ffn_norm.weight
# 1024 | 1024, 1, 1, 1 | F32 | blk.0.attn_k.bias
# 4194304 | 4096, 1024, 1, 1 | Q8_0 | blk.0.attn_k.weight
# 50331648 | 12288, 4096, 1, 1 | Q8_0 | blk.0.attn_output.weight
# 4194304 | 4096, 1024, 1, 1 | Q8_0 | blk.0.attn_v.weight
# 50331648 | 4096, 12288, 1, 1 | Q8_0 | blk.0.attn_q.weight
# 12288 | 12288, 1, 1, 1 | F32 | blk.0.attn_q.bias
# 1024 | 1024, 1, 1, 1 | F32 | blk.0.attn_v.bias
#
# 738197504 | 1408, 4096, 128, 1 | Q8_0 | blk.1.ffn_down_exps.weight
# 738197504 | 4096, 1408, 128, 1 | Q8_0 | blk.1.ffn_gate_exps.weight
# 738197504 | 4096, 1408, 128, 1 | Q8_0 | blk.1.ffn_up_exps.weight
# 4096 | 4096, 1, 1, 1 | F32 | blk.1.attn_norm.weight
# 128 | 128, 1, 1, 1 | F32 | blk.1.ffn_gate_inp.bias
# 524288 | 4096, 128, 1, 1 | F32 | blk.1.ffn_gate_inp.weight
# 5767168 | 1408, 4096, 1, 1 | Q8_0 | blk.1.ffn_down_shexp.weight
# 5767168 | 4096, 1408, 1, 1 | Q8_0 | blk.1.ffn_gate_shexp.weight
# 5767168 | 4096, 1408, 1, 1 | Q8_0 | blk.1.ffn_up_shexp.weight
# 4194304 | 4096, 1024, 1, 1 | Q8_0 | blk.1.attn_k.weight
# 50331648 | 12288, 4096, 1, 1 | Q8_0 | blk.1.attn_output.weight
# 50331648 | 4096, 12288, 1, 1 | Q8_0 | blk.1.attn_q.weight
# 4194304 | 4096, 1024, 1, 1 | Q8_0 | blk.1.attn_v.weight
# 4096 | 4096, 1, 1, 1 | F32 | blk.1.ffn_norm.weight
# 1024 | 1024, 1, 1, 1 | F32 | blk.1.attn_k.bias
# 12288 | 12288, 1, 1, 1 | F32 | blk.1.attn_q.bias
# 1024 | 1024, 1, 1, 1 | F32 | blk.1.attn_v.bias
# 620756992 | 4096, 151552, 1, 1 | Q8_0 | output.weight
custom="
# 47 Repeating Layers [0-46]
# Attention
#blk\.(0)\.attn_q.*=q8_0
#blk\.(0)\.attn_k.*=q8_0
#blk\.(0)\.attn_v.*=q8_0
#blk\.(0)\.attn_output.*=q8_0
blk\..*\.attn_q.*=iq5_ks
blk\..*\.attn_k.*=iq5_ks
blk\..*\.attn_v.*=iq5_ks
blk\..*\.attn_output.*=iq5_ks
# First 1 Dense Layers [0]
blk\..*\.ffn_down\.weight=q6_0
blk\..*\.ffn_(gate|up)\.weight=iq5_ks
# Shared Expert Layers [2-46]
blk\..*\.ffn_down_shexp\.weight=q6_0
blk\..*\.ffn_(gate|up)_shexp\.weight=iq5_ks
# Routed Experts Layers [2-46]
#blk\.(3|92)\.ffn_down_exps\.weight=q8_0
#blk\.(3|92)\.ffn_(gate|up)_exps\.weight=q8_0
blk\..*\.ffn_down_exps\.weight=iq4_nl
blk\..*\.ffn_(gate|up)_exps\.weight=iq4_kss
# Non-Repeating Layers
token_embd\.weight=iq4_k
output\.weight=iq6_k
"
custom=$(
echo "$custom" | grep -v '^#' | \
sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)
numactl -N 1 -m 1 \
./build/bin/llama-quantize \
--custom-q "$custom" \
--imatrix /mnt/raid/models/ubergarm/GLM-4.5-Air-GGUF/imatrix-GLM-4.5-Air-BF16.dat \
/mnt/raid/models/ubergarm/GLM-4.5-Air-GGUF/GLM-4.5-Air-128x8.1B-BF16-00001-of-00005.gguf \
/mnt/raid/models/ubergarm/GLM-4.5-Air-GGUF/GLM-4.5-Air-IQ4_KSS.gguf \
IQ4_KSS \
192
```
</details>
## Quick Start
```bash
# Clone and checkout experimental PR
$ git clone https://github.com/ikawrakow/ik_llama.cpp
$ cd ik_llama.cpp
$ git remote add Thireus https://github.com/Thireus/ik_llama.cpp.git
$ git fetch Thireus
$ git checkout glm-4.5-clean
# Build for hybrid CPU+CUDA
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DGGML_BLAS=OFF -DGGML_SCHED_MAX_COPIES=1
$ cmake --build build --config Release -j $(nproc)
# Test Experimental GGUF
$ ./build/bin/llama-server \
--model WARNING-EXPERIMENTAL-IKLLAMACPP-ONLY-GLM-4.5-Air-IQ4_KSS-00001-of-00002.gguf \
--alias ubergarm/GLM-4.5-Air-IQ4_KSS \
--ctx-size 32768 \
-fa -fmoe \
-ctk q8_0 -ctv q8_0 \
--chat-template chatglm4 \
--override-kv tokenizer.ggml.eot_token_id=int:151336 \
-ub 4096 -b 4096 \
-ngl 99 \
-ot exps=CPU \
--parallel 1 \
--threads 8 \
--host 127.0.0.1 \
--port 8080 \
--no-mmap
```
## References
* [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)
* [Getting Started Guide (already out of date lol)](https://github.com/ikawrakow/ik_llama.cpp/discussions/258)
* [ubergarm-imatrix-calibration-corpus-v02.txt](https://gist.github.com/ubergarm/edfeb3ff9c6ec8b49e88cdf627b0711a?permalink_comment_id=5682584#gistcomment-5682584)
* [Mainline llama.cpp Draft PR14939](https://github.com/ggml-org/llama.cpp/pull/14939)
* [ik_llama.cpp GLM-4.5 MoE PR668](https://github.com/ikawrakow/ik_llama.cpp/pull/668)
|