title: Cost-Optimal Grouped-Query Attention for Long-Context LLMs
summary: Building effective and efficient Transformer-based large language models
  (LLMs) has recently become a research focus, requiring maximizing model
  language capabilities and minimizing training and deployment costs. Existing
  efforts have primarily described complex relationships among model performance,
  parameter size, and data size, as well as searched for the optimal compute
  allocation to train LLMs. However, they overlook the impacts of context length
  and attention head configuration (the number of query and key-value heads in
  grouped-query attention) on training and inference. In this paper, we
  systematically compare models with different parameter sizes, context lengths,
  and attention head configurations in terms of model performance, computational
  cost, and memory cost. Then, we extend the existing scaling methods, which are
  based solely on parameter size and training compute, to guide the construction
  of cost-optimal LLMs during both training and inference. Our quantitative
  scaling studies show that, when processing sufficiently long sequences, a
  larger model with fewer attention heads can achieve a lower loss while
  incurring lower computational and memory costs. Our findings provide valuable
  insights for developing practical LLMs, especially in long-context processing
  scenarios. We will publicly release our code and data.
paper_authors._id: 67d26ae40a955727b9687d8f
paper_authors.name: Yingfa Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:24:03.831Z
paper_authors._id: 6144e4667f2544bb450787b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
paper_authors.fullname: Yingfa Chen
paper_authors.type: user
paper_authors.user: chen-yingfa
paper_authors._id: 67d26ae40a955727b9687d90
paper_authors.name: Yutong Wu
paper_authors._id: 67d26ae40a955727b9687d91
paper_authors.name: Xu Han
paper_authors._id: 67d26ae40a955727b9687d92
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:52:30.729Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 67d26ae40a955727b9687d93
paper_authors.name: Maosong Sun
paper_title: Cost-Optimal Grouped-Query Attention for Long-Context LLMs
paper_summary: Building effective and efficient Transformer-based large language models
  (LLMs) has recently become a research focus, requiring maximizing model
  language capabilities and minimizing training and deployment costs. Existing
  efforts have primarily described complex relationships among model performance,
  parameter size, and data size, as well as searched for the optimal compute
  allocation to train LLMs. However, they overlook the impacts of context length
  and attention head configuration (the number of query and key-value heads in
  grouped-query attention) on training and inference. In this paper, we
  systematically compare models with different parameter sizes, context lengths,
  and attention head configurations in terms of model performance, computational
  cost, and memory cost. Then, we extend the existing scaling methods, which are
  based solely on parameter size and training compute, to guide the construction
  of cost-optimal LLMs during both training and inference. Our quantitative
  scaling studies show that, when processing sufficiently long sequences, a
  larger model with fewer attention heads can achieve a lower loss while
  incurring lower computational and memory costs. Our findings provide valuable
  insights for developing practical LLMs, especially in long-context processing
  scenarios. We will publicly release our code and data.
paper_ai_summary: Research compares different configurations of Transformer-based large language models to optimize performance, cost, and memory usage, especially for long sequences.
paper_ai_keywords: Transformer-based large language models context length attention head configuration model performance computational cost memory cost parameter size grouped-query attention