content.raw: Layer-wise and Pruned versions of google/gemma-3-12b-it
content.type: text
content.value: Layer-wise and Pruned versions of google/gemma-3-12b-it
content.type: new_line
content.type: new_line
content.raw: After enhancing llama.cpp to handle user-defined quantization levels for arbitrary tensors (
content.type: text
content.value: After enhancing llama.cpp to handle user-defined quantization levels for arbitrary tensors (
content.href: https://github.com/ggml-org/llama.cpp/pull/12511
content.raw: https://github.com/ggml-org/llama.cpp/pull/12511
content.type: link
content.raw: ),  I have added an option to prune whole layers (
content.type: text
content.value: ),  I have added an option to prune whole layers (
content.href: https://github.com/ggml-org/llama.cpp/pull/13037
content.raw: https://github.com/ggml-org/llama.cpp/pull/13037
content.type: link
content.raw: ), and have published two versions of 
content.type: text
content.value: ), and have published two versions of 
content.raw: https://huggingface.co/google/gemma-3-12b-it
content.id: google/gemma-3-12b-it
content.type: model
content.type: resource
content.url: https://huggingface.co/google/gemma-3-12b-it
content.raw:  for demo and testing purposes:
content.type: text
content.value:  for demo and testing purposes:
content.type: new_line
content.type: new_line
content.raw: * Tesor-wise: 
content.type: text
content.value: * Tesor-wise: 
content.raw: https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
content.id: eaddario/gemma-3-12b-it-GGUF
content.type: model
content.type: resource
content.url: https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
content.type: new_line
content.raw: * Pruned: 
content.type: text
content.value: * Pruned: 
content.raw: https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
content.id: eaddario/gemma-3-12b-it-pruned-GGUF
content.type: model
content.type: resource
content.url: https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
content.type: new_line
content.type: new_line
content.raw: Even though the Perplexity scores of the pruned version are 3 times higher, the ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores are holding remarkably well, considering two layers were removed (26 and 29). This seems to support Xin Men et al conclusions in 
content.type: text
content.value: Even though the Perplexity scores of the pruned version are 3 times higher, the ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores are holding remarkably well, considering two layers were removed (26 and 29). This seems to support Xin Men et al conclusions in 
content.label: ShortGPT: Layers in Large Language Models are More Redundant Than You
    Expect (2403.03853)
content.raw: https://huggingface.co/papers/2403.03853
content.id: 2403.03853
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2403.03853
content.type: new_line
content.type: new_line
content.raw: Results summary in the model's card and test results in the ./scores directory. Questions/feedback is always welcomed.
content.type: text
content.value: Results summary in the model's card and test results in the ./scores directory. Questions/feedback is always welcomed.
rawContent: Layer-wise and Pruned versions of google/gemma-3-12b-it
  
  After enhancing llama.cpp to handle user-defined quantization levels for arbitrary tensors (https://github.com/ggml-org/llama.cpp/pull/12511),  I have added an option to prune whole layers (https://github.com/ggml-org/llama.cpp/pull/13037), and have published two versions of https://huggingface.co/google/gemma-3-12b-it for demo and testing purposes:
  
  * Tesor-wise: https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
  * Pruned: https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
  
  Even though the Perplexity scores of the pruned version are 3 times higher, the ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores are holding remarkably well, considering two layers were removed (26 and 29). This seems to support Xin Men et al conclusions in https://huggingface.co/papers/2403.03853
  
  Results summary in the model's card and test results in the ./scores directory. Questions/feedback is always welcomed.
reactions.reaction: 🚀
reactions.users: John6666 ubergarm djuna