title: SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
summary: The costly self-attention layers in modern Transformers require memory and
  compute quadratic in sequence length. Existing approximation methods usually
  underperform and fail to obtain significant speedups in practice. Here we
  present SwitchHead - a novel method that reduces both compute and memory
  requirements and achieves wall-clock speedup, while matching the language
  modeling performance of baseline Transformers with the same parameter budget.
  SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output
  projections and requires 4 to 8 times fewer attention matrices than standard
  Transformers. Our novel attention can also be combined with MoE MLP layers,
  resulting in an efficient fully-MoE "SwitchAll" Transformer model. Our code is
  public.
paper_authors._id: 657a68811ccc3c2a5ea6685d
paper_authors.name: Róbert Csordás
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T15:52:06.375Z
paper_authors._id: 64e5d1bef450dca5ca0d05ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
paper_authors.fullname: Robert Csordas
paper_authors.type: user
paper_authors.user: robertcsordas
paper_authors._id: 657a68811ccc3c2a5ea6685e
paper_authors.name: Piotr Piękos
paper_authors._id: 657a68811ccc3c2a5ea6685f
paper_authors.name: Kazuki Irie
paper_authors._id: 657c1561536a3e1d5c5a3482
paper_authors.name: Jürgen Schmidhuber
paper_title: SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
paper_summary: The costly self-attention layers in modern Transformers require memory and
  compute quadratic in sequence length. Existing approximation methods usually
  underperform and fail to obtain significant speedups in practice. Here we
  present SwitchHead - a novel method that reduces both compute and memory
  requirements and achieves wall-clock speedup, while matching the language
  modeling performance of baseline Transformers with the same parameter budget.
  SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output
  projections and requires 4 to 8 times fewer attention matrices than standard
  Transformers. Our novel attention can also be combined with MoE MLP layers,
  resulting in an efficient fully-MoE "SwitchAll" Transformer model. Our code is
  public.
paper_ai_summary: SwitchHead reduces memory and compute for Transformers by using Mixture-of-Experts layers, matching performance with significant speedup.
paper_ai_keywords: self-attention layers memory compute Mixture-of-Experts value projections output projections attention matrices fully-MoE SwitchAll Transformer model