title: Zebra: Extending Context Window with Layerwise Grouped Local-Global
    Attention
summary: This paper introduces a novel approach to enhance the capabilities of Large
  Language Models (LLMs) in processing and understanding extensive text
  sequences, a critical aspect in applications requiring deep comprehension and
  synthesis of large volumes of information. Recognizing the inherent challenges
  in extending the context window for LLMs, primarily built on Transformer
  architecture, we propose a new model architecture, referred to as Zebra. This
  architecture efficiently manages the quadratic time and memory complexity
  issues associated with full attention in the Transformer by employing grouped
  local-global attention layers. Our model, akin to a zebra's alternating
  stripes, balances local and global attention layers, significantly reducing
  computational requirements and memory consumption. Comprehensive experiments,
  including pretraining from scratch, continuation of long context adaptation
  training, and long instruction tuning, are conducted to evaluate the Zebra's
  performance. The results show that Zebra achieves comparable or superior
  performance on both short and long sequence benchmarks, while also enhancing
  training and inference efficiency.
paper_authors._id: 657bc91afcba5f698c23000d
paper_authors.name: Kaiqiang Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:59:29.731Z
paper_authors._id: 63a261f13c003e409322c191
paper_authors.avatarUrl: /avatars/dc4be6ce1983d0d32620275df124fe45.svg
paper_authors.fullname: Kaiqiang Song
paper_authors.type: user
paper_authors.user: kqsong
paper_authors._id: 657bc91afcba5f698c23000e
paper_authors.name: Xiaoyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-18T08:48:16.037Z
paper_authors._id: 657cd228138b7e391444a65d
paper_authors.avatarUrl: /avatars/c7c984ae483144fab627aa2c54d91d0f.svg
paper_authors.fullname: Xiaoyang Wang
paper_authors.type: user
paper_authors.user: xywang1
paper_authors._id: 657bc91afcba5f698c23000f
paper_authors.name: Sangwoo Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-01T08:44:35.869Z
paper_authors._id: 64dbc6251d19239f50752356
paper_authors.avatarUrl: /avatars/3d60998db3dd8f37a5158e1758c0f706.svg
paper_authors.fullname: Sangwoo Cho
paper_authors.type: user
paper_authors.user: swcho3
paper_authors._id: 657bc91afcba5f698c230010
paper_authors.name: Xiaoman Pan
paper_authors._id: 657bc91afcba5f698c230011
paper_authors.name: Dong Yu
paper_title: Zebra: Extending Context Window with Layerwise Grouped Local-Global
    Attention
paper_summary: This paper introduces a novel approach to enhance the capabilities of Large
  Language Models (LLMs) in processing and understanding extensive text
  sequences, a critical aspect in applications requiring deep comprehension and
  synthesis of large volumes of information. Recognizing the inherent challenges
  in extending the context window for LLMs, primarily built on Transformer
  architecture, we propose a new model architecture, referred to as Zebra. This
  architecture efficiently manages the quadratic time and memory complexity
  issues associated with full attention in the Transformer by employing grouped
  local-global attention layers. Our model, akin to a zebra's alternating
  stripes, balances local and global attention layers, significantly reducing
  computational requirements and memory consumption. Comprehensive experiments,
  including pretraining from scratch, continuation of long context adaptation
  training, and long instruction tuning, are conducted to evaluate the Zebra's
  performance. The results show that Zebra achieves comparable or superior
  performance on both short and long sequence benchmarks, while also enhancing
  training and inference efficiency.
paper_ai_summary: A new model architecture called Zebra addresses the challenges of handling large text sequences in LLMs by using grouped local-global attention layers, improving efficiency and performance.
paper_ai_keywords: Transformer architecture grouped local-global attention layers computational requirements memory consumption pretraining long context adaptation training long instruction tuning