Layer-wise and Pruned versions of google/gemma-3-12b-it
text
Layer-wise and Pruned versions of google/gemma-3-12b-it
new_line
new_line
After enhancing llama.cpp to handle user-defined quantization levels for arbitrary tensors (
text
After enhancing llama.cpp to handle user-defined quantization levels for arbitrary tensors (
https://github.com/ggml-org/llama.cpp/pull/12511
https://github.com/ggml-org/llama.cpp/pull/12511
link
),  I have added an option to prune whole layers (
text
),  I have added an option to prune whole layers (
https://github.com/ggml-org/llama.cpp/pull/13037
https://github.com/ggml-org/llama.cpp/pull/13037
link
), and have published two versions of 
text
), and have published two versions of 
https://huggingface.co/google/gemma-3-12b-it
google/gemma-3-12b-it
model
resource
https://huggingface.co/google/gemma-3-12b-it
 for demo and testing purposes:
text
 for demo and testing purposes:
new_line
new_line
* Tesor-wise: 
text
* Tesor-wise: 
https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
eaddario/gemma-3-12b-it-GGUF
model
resource
https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
new_line
* Pruned: 
text
* Pruned: 
https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
eaddario/gemma-3-12b-it-pruned-GGUF
model
resource
https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
new_line
new_line
Even though the Perplexity scores of the pruned version are 3 times higher, the ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores are holding remarkably well, considering two layers were removed (26 and 29). This seems to support Xin Men et al conclusions in 
text
Even though the Perplexity scores of the pruned version are 3 times higher, the ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores are holding remarkably well, considering two layers were removed (26 and 29). This seems to support Xin Men et al conclusions in 
ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect (2403.03853)
https://huggingface.co/papers/2403.03853
2403.03853
paper
resource
https://huggingface.co/papers/2403.03853
new_line
new_line
Results summary in the model's card and test results in the ./scores directory. Questions/feedback is always welcomed.
text
Results summary in the model's card and test results in the ./scores directory. Questions/feedback is always welcomed.
Layer-wise and Pruned versions of google/gemma-3-12b-it

After enhancing llama.cpp to handle user-defined quantization levels for arbitrary tensors (https://github.com/ggml-org/llama.cpp/pull/12511),  I have added an option to prune whole layers (https://github.com/ggml-org/llama.cpp/pull/13037), and have published two versions of https://huggingface.co/google/gemma-3-12b-it for demo and testing purposes:

* Tesor-wise: https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
* Pruned: https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF

Even though the Perplexity scores of the pruned version are 3 times higher, the ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores are holding remarkably well, considering two layers were removed (26 and 29). This seems to support Xin Men et al conclusions in https://huggingface.co/papers/2403.03853

Results summary in the model's card and test results in the ./scores directory. Questions/feedback is always welcomed.
🚀
John6666 ubergarm djuna