title: Hymba: A Hybrid-head Architecture for Small Language Models
summary: We propose Hymba, a family of small language models featuring a hybrid-head
  parallel architecture that integrates transformer attention mechanisms with
  state space models (SSMs) for enhanced efficiency. Attention heads provide
  high-resolution recall, while SSM heads enable efficient context summarization.
  Additionally, we introduce learnable meta tokens that are prepended to prompts,
  storing critical information and alleviating the "forced-to-attend" burden
  associated with attention mechanisms. This model is further optimized by
  incorporating cross-layer key-value (KV) sharing and partial sliding window
  attention, resulting in a compact cache size. During development, we conducted
  a controlled study comparing various architectures under identical settings and
  observed significant advantages of our proposed architecture. Notably, Hymba
  achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model
  surpasses all sub-2B public models in performance and even outperforms
  Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size
  reduction, and 3.49x throughput.
paper_authors._id: 674003e7188f876068a7f1be
paper_authors.name: Xin Dong
paper_authors._id: 674003e7188f876068a7f1bf
paper_authors.name: Yonggan Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:13:01.574Z
paper_authors._id: 66011091d2a378a1633b3a83
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66011091d2a378a1633b3a83/0a46rrZbY_gR8q_I7ce2B.jpeg
paper_authors.fullname: Yonggan Fu
paper_authors.type: user
paper_authors.user: tilmto
paper_authors._id: 674003e7188f876068a7f1c0
paper_authors.name: Shizhe Diao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:13:22.128Z
paper_authors._id: 633bd54b00732349209a18fe
paper_authors.avatarUrl: /avatars/150aa5b8d9bcca24366402932bf2d049.svg
paper_authors.fullname: Shizhe Diao
paper_authors.type: user
paper_authors.user: shizhediao
paper_authors._id: 674003e7188f876068a7f1c1
paper_authors.name: Wonmin Byeon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:13:28.425Z
paper_authors._id: 66bf958296583c59b049085b
paper_authors.avatarUrl: /avatars/04df8dd45835b7ea0991e242784e7810.svg
paper_authors.fullname: Wonmin Byeon
paper_authors.type: user
paper_authors.user: wbyeon
paper_authors._id: 674003e7188f876068a7f1c2
paper_authors.name: Zijia Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:14:30.968Z
paper_authors._id: 65c2f8350aa2d53135b47b34
paper_authors.avatarUrl: /avatars/b08e12905077cf441593586a98b63b2e.svg
paper_authors.fullname: Zijia Chen
paper_authors.type: user
paper_authors.user: zijiac-nvidia
paper_authors._id: 674003e7188f876068a7f1c3
paper_authors.name: Ameya Sunil Mahabaleshwarkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:14:36.408Z
paper_authors._id: 659ef64b419de2ea602cf271
paper_authors.avatarUrl: /avatars/b5313218c009cf6d3fb8e3a7ac11091a.svg
paper_authors.fullname: Ameya Sunil Mahabaleshwarkar
paper_authors.type: user
paper_authors.user: ameyasunilm
paper_authors._id: 674003e7188f876068a7f1c4
paper_authors.name: Shih-Yang Liu
paper_authors._id: 674003e7188f876068a7f1c5
paper_authors.name: Matthijs Van Keirsbilck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:14:47.245Z
paper_authors._id: 66476d517ab33d141de67f86
paper_authors.avatarUrl: /avatars/e0681c0666c9db2060f91db064112fe9.svg
paper_authors.fullname: matthijs van keirsbilck
paper_authors.type: user
paper_authors.user: matthijsvk
paper_authors._id: 674003e7188f876068a7f1c6
paper_authors.name: Min-Hung Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:14:52.517Z
paper_authors._id: 64ae22dd1aee69ece065cdcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
paper_authors.fullname: Min-Hung Chen
paper_authors.type: user
paper_authors.user: cmhungsteve
paper_authors._id: 674003e7188f876068a7f1c7
paper_authors.name: Yoshi Suhara
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:14:57.438Z
paper_authors._id: 5fc181c4ea82dd667bb0ffae
paper_authors.avatarUrl: /avatars/311461de0933d7e4a8a222d0e76f3754.svg
paper_authors.fullname: Yoshi Suhara
paper_authors.type: user
paper_authors.user: suhara
paper_authors._id: 674003e7188f876068a7f1c8
paper_authors.name: Yingyan Lin
paper_authors._id: 674003e7188f876068a7f1c9
paper_authors.name: Jan Kautz
paper_authors._id: 674003e7188f876068a7f1ca
paper_authors.name: Pavlo Molchanov
paper_title: Hymba: A Hybrid-head Architecture for Small Language Models
paper_summary: We propose Hymba, a family of small language models featuring a hybrid-head
  parallel architecture that integrates transformer attention mechanisms with
  state space models (SSMs) for enhanced efficiency. Attention heads provide
  high-resolution recall, while SSM heads enable efficient context summarization.
  Additionally, we introduce learnable meta tokens that are prepended to prompts,
  storing critical information and alleviating the "forced-to-attend" burden
  associated with attention mechanisms. This model is further optimized by
  incorporating cross-layer key-value (KV) sharing and partial sliding window
  attention, resulting in a compact cache size. During development, we conducted
  a controlled study comparing various architectures under identical settings and
  observed significant advantages of our proposed architecture. Notably, Hymba
  achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model
  surpasses all sub-2B public models in performance and even outperforms
  Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size
  reduction, and 3.49x throughput.
paper_ai_keywords: hybrid-head architecture transformer attention mechanisms state space models (SSMs) attention heads high-resolution recall SSM heads context summarization learnable meta tokens prompts cross-layer key (KV) sharing partial sliding window attention compact cache size sub-2B models performance average accuracy throughput