Squeezing Tensor Bits: the quest for smaller LLMs
text
Squeezing Tensor Bits: the quest for smaller LLMs
new_line
new_line
An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. 
text
An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. 
new_line
new_line
The method that I'm using to produce these experimental versions, for example 
text
The method that I'm using to produce these experimental versions, for example 
https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF
eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF
model
resource
https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF
 is explained in 
text
 is explained in 
https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca
https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca
link
new_line
new_line
At a high level it involves using a custom version of the llama-quantize tool to selectively quantize different tensors at different levels. On average a 10% or more reduction with little loss of quality is possible. 
text
At a high level it involves using a custom version of the llama-quantize tool to selectively quantize different tensors at different levels. On average a 10% or more reduction with little loss of quality is possible. 
new_line
new_line
There’re two PRs to merge these changes back into the core project but until then, the modified version will be available on GitHub 
text
There’re two PRs to merge these changes back into the core project but until then, the modified version will be available on GitHub 
https://github.com/EAddario/llama.cpp/tree/quantize
https://github.com/EAddario/llama.cpp/tree/quantize
link
new_line
new_line
Would love to hear if you can achieve smaller sizes at higher quality!
text
Would love to hear if you can achieve smaller sizes at higher quality!
Squeezing Tensor Bits: the quest for smaller LLMs

An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. 

The method that I'm using to produce these experimental versions, for example https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF is explained in https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca

At a high level it involves using a custom version of the llama-quantize tool to selectively quantize different tensors at different levels. On average a 10% or more reduction with little loss of quality is possible. 

There’re two PRs to merge these changes back into the core project but until then, the modified version will be available on GitHub https://github.com/EAddario/llama.cpp/tree/quantize

Would love to hear if you can achieve smaller sizes at higher quality!
🔥
John6666 Mdubbya CATAMERCA xinnn63 Takugen dantezxcd
🚀
Mahdiyyah4 dantezxcd
👍
agentlans dantezxcd
6639a9df7c0ab4fd9df0a5b5
/avatars/38e901a06c04256f97d5d6947657c58c.svg
Mark Washington
Mdubbya
user
663a18c876e6d5b98f3cc94d
/avatars/faa2a47dbdd3bce6d47e0fbed6170627.svg
Alan Tseng
agentlans
user
658f24cb35c41262d64af1a2
https://cdn-avatars.huggingface.co/v1/production/uploads/658f24cb35c41262d64af1a2/vUK_c6K821tq61AxSeV4i.png
Ed Addario
eaddario
user