content.raw: Tensor-wise (TWQ) and Layer-wise quantization (LWQ) now available in llama.cpp!
content.type: text
content.value: Tensor-wise (TWQ) and Layer-wise quantization (LWQ) now available in llama.cpp!
content.type: new_line
content.type: new_line
content.raw: As of version b5125 users can now do TWQ, whereby you quantize a whole tensor at a specific level, or perform LWQ by choosing specific layers per tensor/s
content.type: text
content.value: As of version b5125 users can now do TWQ, whereby you quantize a whole tensor at a specific level, or perform LWQ by choosing specific layers per tensor/s
content.type: new_line
content.type: new_line
content.raw: The new --tensor-type option enables llama-quantize to apply user-defined quant levels to any combination of allowed tensors (i.e. tensors with 2 or more dimensions)  and layer number, with support for regex patterns.
content.type: text
content.value: The new --tensor-type option enables llama-quantize to apply user-defined quant levels to any combination of allowed tensors (i.e. tensors with 2 or more dimensions)  and layer number, with support for regex patterns.
content.type: new_line
content.type: new_line
content.raw: For example, to TWQ the Attention Value tensor you would use --tensor-type attn_v=q6_k and to perform LWQ you'll use something like --tensor-type "\.([0-9]|1[01257]|31)\.attn_v=q4_k"
content.type: text
content.value: For example, to TWQ the Attention Value tensor you would use --tensor-type attn_v=q6_k and to perform LWQ you'll use something like --tensor-type "\.([0-9]|1[01257]|31)\.attn_v=q4_k"
content.type: new_line
content.type: new_line
content.raw: In the next few days/weeks I'll update the models in my HF repo (and will add some others) but 
content.type: text
content.value: In the next few days/weeks I'll update the models in my HF repo (and will add some others) but 
content.raw: https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF
content.id: eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF
content.type: model
content.type: resource
content.url: https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF
content.raw:  and 
content.type: text
content.value:  and 
content.raw: https://huggingface.co/eaddario/DeepSeek-R1-Distill-Qwen-7B-GGUF
content.id: eaddario/DeepSeek-R1-Distill-Qwen-7B-GGUF
content.type: model
content.type: resource
content.url: https://huggingface.co/eaddario/DeepSeek-R1-Distill-Qwen-7B-GGUF
content.raw:  have been already LWQed.
content.type: text
content.value:  have been already LWQed.
content.type: new_line
content.type: new_line
content.raw: For reference, compared to the naive Q4_K_M model, the LWQ Qwen-7B is almost 11% smaller (4.68GB vs 4.18GB) with only a 0.35% penalty on PPL!
content.type: text
content.value: For reference, compared to the naive Q4_K_M model, the LWQ Qwen-7B is almost 11% smaller (4.68GB vs 4.18GB) with only a 0.35% penalty on PPL!
content.type: new_line
content.type: new_line
content.raw: I'll update the 
content.type: text
content.value: I'll update the 
content.href: https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca
content.raw: https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca
content.type: link
content.raw:  post to explain the process in detail, but in the meantime the following links will provide some background:
content.type: text
content.value:  post to explain the process in detail, but in the meantime the following links will provide some background:
content.type: new_line
content.type: new_line
content.raw: - Changes to llama-quantize: 
content.type: text
content.value: - Changes to llama-quantize: 
content.href: https://github.com/ggml-org/llama.cpp/pull/12511
content.raw: https://github.com/ggml-org/llama.cpp/pull/12511
content.type: link
content.type: new_line
content.raw: - TWQ & LWQ tests: 
content.type: text
content.value: - TWQ & LWQ tests: 
content.href: https://github.com/ggml-org/llama.cpp/discussions/12741
content.raw: https://github.com/ggml-org/llama.cpp/discussions/12741
content.type: link
content.type: new_line
content.raw: - Modified llama-imatrix (not yet merged) used to generate imatrix statistics to guide the TWQ and LWQ process: 
content.type: text
content.value: - Modified llama-imatrix (not yet merged) used to generate imatrix statistics to guide the TWQ and LWQ process: 
content.href: https://github.com/ggml-org/llama.cpp/pull/12718
content.raw: https://github.com/ggml-org/llama.cpp/pull/12718
content.type: link
rawContent: Tensor-wise (TWQ) and Layer-wise quantization (LWQ) now available in llama.cpp!
  
  As of version b5125 users can now do TWQ, whereby you quantize a whole tensor at a specific level, or perform LWQ by choosing specific layers per tensor/s
  
  The new --tensor-type option enables llama-quantize to apply user-defined quant levels to any combination of allowed tensors (i.e. tensors with 2 or more dimensions)  and layer number, with support for regex patterns.
  
  For example, to TWQ the Attention Value tensor you would use --tensor-type attn_v=q6_k and to perform LWQ you'll use something like --tensor-type "\.([0-9]|1[01257]|31)\.attn_v=q4_k"
  
  In the next few days/weeks I'll update the models in my HF repo (and will add some others) but https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF and https://huggingface.co/eaddario/DeepSeek-R1-Distill-Qwen-7B-GGUF have been already LWQed.
  
  For reference, compared to the naive Q4_K_M model, the LWQ Qwen-7B is almost 11% smaller (4.68GB vs 4.18GB) with only a 0.35% penalty on PPL!
  
  I'll update the https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca post to explain the process in detail, but in the meantime the following links will provide some background:
  
  - Changes to llama-quantize: https://github.com/ggml-org/llama.cpp/pull/12511
  - TWQ & LWQ tests: https://github.com/ggml-org/llama.cpp/discussions/12741
  - Modified llama-imatrix (not yet merged) used to generate imatrix statistics to guide the TWQ and LWQ process: https://github.com/ggml-org/llama.cpp/pull/12718
reactions.reaction: 🔥
reactions.users: Felladrin John6666 JohnRoger dark-pen