title: Block Transformer: Global-to-Local Language Modeling for Fast Inference
summary: This paper presents the Block Transformer architecture which adopts
  hierarchical global-to-local modeling to autoregressive transformers to
  mitigate the inference bottlenecks of self-attention. To apply self-attention,
  the key-value (KV) cache of all previous sequences must be retrieved from
  memory at every decoding step. Thereby, this KV cache IO becomes a significant
  bottleneck in batch inference. We notice that these costs stem from applying
  self-attention on the global context, therefore we isolate the expensive
  bottlenecks of global modeling to lower layers and apply fast local modeling in
  upper layers. To mitigate the remaining costs in the lower layers, we aggregate
  input tokens into fixed size blocks and then apply self-attention at this
  coarse level. Context information is aggregated into a single embedding to
  enable upper layers to decode the next block of tokens, without global
  attention. Free of global attention bottlenecks, the upper layers can fully
  utilize the compute hardware to maximize inference throughput. By leveraging
  global and local modules, the Block Transformer architecture demonstrates
  10-20x gains in inference throughput compared to vanilla transformers with
  equivalent perplexity. Our work introduces a new approach to optimize language
  model inference through novel application of global-to-local modeling. Code is
  available at https://github.com/itsnamgyu/block-transformer.
paper_authors._id: 666120834e5af73c6a4ea91d
paper_authors.name: Namgyu Ho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:13:42.601Z
paper_authors._id: 62d0f7faad741b94f5d13744
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677826870505-62d0f7faad741b94f5d13744.jpeg
paper_authors.fullname: Namgyu Ho
paper_authors.type: user
paper_authors.user: itsnamgyu
paper_authors._id: 666120834e5af73c6a4ea91e
paper_authors.name: Sangmin Bae
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:13:59.532Z
paper_authors._id: 6602ca1e10a1441af41637be
paper_authors.avatarUrl: /avatars/5880e699def320beb352cbed77495b2f.svg
paper_authors.fullname: Sangmin Bae
paper_authors.type: user
paper_authors.user: raymin0223
paper_authors._id: 666120834e5af73c6a4ea91f
paper_authors.name: Taehyeon Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:14:45.383Z
paper_authors._id: 639f32952e13e54dcbb9bf8d
paper_authors.avatarUrl: /avatars/a797b72363e06cd2222165493bf7e39c.svg
paper_authors.fullname: Taehyeon Kim
paper_authors.type: user
paper_authors.user: Kthyeon
paper_authors._id: 666120834e5af73c6a4ea920
paper_authors.name: Hyunjik Jo
paper_authors._id: 666120834e5af73c6a4ea921
paper_authors.name: Yireun Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:20:39.322Z
paper_authors._id: 660260cf1737e5cd4a826550
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/660260cf1737e5cd4a826550/AlSfoM2WtqPjLtYR6x7Wf.jpeg
paper_authors.fullname: Yireun Kim
paper_authors.type: user
paper_authors.user: yireun
paper_authors._id: 666120834e5af73c6a4ea922
paper_authors.name: Tal Schuster
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:20:45.770Z
paper_authors._id: 60440983efd035140b99c814
paper_authors.avatarUrl: /avatars/4afb3cc5b940b24bd2794d07f694b9ef.svg
paper_authors.fullname: Tal Schuster
paper_authors.type: user
paper_authors.user: tals
paper_authors._id: 666120834e5af73c6a4ea923
paper_authors.name: Adam Fisch
paper_authors._id: 666120834e5af73c6a4ea924
paper_authors.name: James Thorne
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-07T09:20:34.411Z
paper_authors._id: 63b54db5889aa6707f08f2d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b54db5889aa6707f08f2d1/zUnyx-8Zabxr54poZtUo4.jpeg
paper_authors.fullname: James Thorne
paper_authors.type: user
paper_authors.user: j6mes
paper_authors._id: 666120834e5af73c6a4ea925
paper_authors.name: Se-Young Yun
paper_title: Block Transformer: Global-to-Local Language Modeling for Fast Inference
paper_summary: This paper presents the Block Transformer architecture which adopts
  hierarchical global-to-local modeling to autoregressive transformers to
  mitigate the inference bottlenecks of self-attention. To apply self-attention,
  the key-value (KV) cache of all previous sequences must be retrieved from
  memory at every decoding step. Thereby, this KV cache IO becomes a significant
  bottleneck in batch inference. We notice that these costs stem from applying
  self-attention on the global context, therefore we isolate the expensive
  bottlenecks of global modeling to lower layers and apply fast local modeling in
  upper layers. To mitigate the remaining costs in the lower layers, we aggregate
  input tokens into fixed size blocks and then apply self-attention at this
  coarse level. Context information is aggregated into a single embedding to
  enable upper layers to decode the next block of tokens, without global
  attention. Free of global attention bottlenecks, the upper layers can fully
  utilize the compute hardware to maximize inference throughput. By leveraging
  global and local modules, the Block Transformer architecture demonstrates
  10-20x gains in inference throughput compared to vanilla transformers with
  equivalent perplexity. Our work introduces a new approach to optimize language
  model inference through novel application of global-to-local modeling. Code is
  available at https://github.com/itsnamgyu/block-transformer.
paper_ai_keywords: Block Transformer architecture hierarchical global-to-local modeling autoregressive transformers self-attention key-value (KV) cache inference bottlenecks batch inference global modeling fast local modeling input tokens fixed size blocks context information single embedding global attention compute hardware inference throughput vanilla transformers perplexity