SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
The costly self-attention layers in modern Transformers require memory and
compute quadratic in sequence length. Existing approximation methods usually
underperform and fail to obtain significant speedups in practice. Here we
present SwitchHead - a novel method that reduces both compute and memory
requirements and achieves wall-clock speedup, while matching the language
modeling performance of baseline Transformers with the same parameter budget.
SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output
projections and requires 4 to 8 times fewer attention matrices than standard
Transformers. Our novel attention can also be combined with MoE MLP layers,
resulting in an efficient fully-MoE "SwitchAll" Transformer model. Our code is
public.
657a68811ccc3c2a5ea6685d
Róbert Csordás
admin_assigned
2023-12-14T15:52:06.375Z
64e5d1bef450dca5ca0d05ed
https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
Robert Csordas
user
robertcsordas
657a68811ccc3c2a5ea6685e
Piotr Piękos
657a68811ccc3c2a5ea6685f
Kazuki Irie
657c1561536a3e1d5c5a3482
Jürgen Schmidhuber
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
The costly self-attention layers in modern Transformers require memory and
compute quadratic in sequence length. Existing approximation methods usually
underperform and fail to obtain significant speedups in practice. Here we
present SwitchHead - a novel method that reduces both compute and memory
requirements and achieves wall-clock speedup, while matching the language
modeling performance of baseline Transformers with the same parameter budget.
SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output
projections and requires 4 to 8 times fewer attention matrices than standard
Transformers. Our novel attention can also be combined with MoE MLP layers,
resulting in an efficient fully-MoE "SwitchAll" Transformer model. Our code is
public.
SwitchHead reduces memory and compute for Transformers by using Mixture-of-Experts layers, matching performance with significant speedup.
self-attention layers memory compute Mixture-of-Experts value projections output projections attention matrices fully-MoE SwitchAll Transformer model