title: Efficient Hybrid Language Model Compression through Group-Aware SSM
    Pruning
summary: Hybrid LLM architectures that combine Attention and State Space Models (SSMs)
  achieve state-of-the-art accuracy and runtime performance. Recent work has
  demonstrated that applying compression and distillation to Attention-only
  models yields smaller, more accurate models at a fraction of the training cost.
  In this work, we explore the effectiveness of compressing Hybrid architectures.
  We introduce a novel group-aware pruning strategy that preserves the structural
  integrity of SSM blocks and their sequence modeling capabilities. Furthermore,
  we demonstrate the necessity of such SSM pruning to achieve improved accuracy
  and inference speed compared to traditional approaches. Our compression recipe
  combines SSM, FFN, embedding dimension, and layer pruning, followed by
  knowledge distillation-based retraining, similar to the MINITRON technique.
  Using this approach, we compress the Nemotron-H 8B Hybrid model down to 4B
  parameters with up to 40x fewer training tokens. The resulting model surpasses
  the accuracy of similarly-sized models while achieving 2x faster inference,
  significantly advancing the Pareto frontier.
paper_authors._id: 67fff37f995d6bb5fe72e2bb
paper_authors.name: Ali Taghibakhshi
paper_authors._id: 67fff37f995d6bb5fe72e2bc
paper_authors.name: Sharath Turuvekere Sreenivas
paper_authors._id: 67fff37f995d6bb5fe72e2bd
paper_authors.name: Saurav Muralidharan
paper_authors._id: 67fff37f995d6bb5fe72e2be
paper_authors.name: Marcin Chochowski
paper_authors._id: 67fff37f995d6bb5fe72e2bf
paper_authors.name: Yashaswi Karnati
paper_authors._id: 67fff37f995d6bb5fe72e2c0
paper_authors.name: Raviraj Joshi
paper_authors._id: 67fff37f995d6bb5fe72e2c1
paper_authors.name: Ameya Sunil Mahabaleshwarkar
paper_authors._id: 67fff37f995d6bb5fe72e2c2
paper_authors.name: Zijia Chen
paper_authors._id: 67fff37f995d6bb5fe72e2c3
paper_authors.name: Yoshi Suhara
paper_authors._id: 67fff37f995d6bb5fe72e2c4
paper_authors.name: Oluwatobi Olabiyi
paper_authors._id: 67fff37f995d6bb5fe72e2c5
paper_authors.name: Daniel Korzekwa
paper_authors._id: 67fff37f995d6bb5fe72e2c6
paper_authors.name: Mostofa Patwary
paper_authors._id: 67fff37f995d6bb5fe72e2c7
paper_authors.name: Mohammad Shoeybi
paper_authors._id: 67fff37f995d6bb5fe72e2c8
paper_authors.name: Jan Kautz
paper_authors._id: 67fff37f995d6bb5fe72e2c9
paper_authors.name: Bryan Catanzaro
paper_authors._id: 67fff37f995d6bb5fe72e2ca
paper_authors.name: Ashwath Aithal
paper_authors._id: 67fff37f995d6bb5fe72e2cb
paper_authors.name: Nima Tajbakhsh
paper_authors._id: 67fff37f995d6bb5fe72e2cc
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:03:39.466Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_title: Efficient Hybrid Language Model Compression through Group-Aware SSM
    Pruning
paper_summary: Hybrid LLM architectures that combine Attention and State Space Models (SSMs)
  achieve state-of-the-art accuracy and runtime performance. Recent work has
  demonstrated that applying compression and distillation to Attention-only
  models yields smaller, more accurate models at a fraction of the training cost.
  In this work, we explore the effectiveness of compressing Hybrid architectures.
  We introduce a novel group-aware pruning strategy that preserves the structural
  integrity of SSM blocks and their sequence modeling capabilities. Furthermore,
  we demonstrate the necessity of such SSM pruning to achieve improved accuracy
  and inference speed compared to traditional approaches. Our compression recipe
  combines SSM, FFN, embedding dimension, and layer pruning, followed by
  knowledge distillation-based retraining, similar to the MINITRON technique.
  Using this approach, we compress the Nemotron-H 8B Hybrid model down to 4B
  parameters with up to 40x fewer training tokens. The resulting model surpasses
  the accuracy of similarly-sized models while achieving 2x faster inference,
  significantly advancing the Pareto frontier.
paper_ai_summary: A novel group-aware pruning strategy is introduced to compress Hybrid LLM architectures, enhancing accuracy and inference speed while reducing training costs and parameters.
paper_ai_keywords: Hybrid LLM architectures Attention State Space Models SSM FFN embedding dimension layer pruning knowledge distillation MINITRON Nemotron-H parameter compression training tokens