title: MoH: Multi-Head Attention as Mixture-of-Head Attention
summary: In this work, we upgrade the multi-head attention mechanism, the core of the
  Transformer model, to improve efficiency while maintaining or surpassing the
  previous accuracy level. We show that multi-head attention can be expressed in
  the summation form. Drawing on the insight that not all attention heads hold
  equal significance, we propose Mixture-of-Head attention (MoH), a new
  architecture that treats attention heads as experts in the Mixture-of-Experts
  (MoE) mechanism. MoH has two significant advantages: First, MoH enables each
  token to select the appropriate attention heads, enhancing inference efficiency
  without compromising accuracy or increasing the number of parameters. Second,
  MoH replaces the standard summation in multi-head attention with a weighted
  summation, introducing flexibility to the attention mechanism and unlocking
  extra performance potential. Extensive experiments on ViT, DiT, and LLMs
  demonstrate that MoH outperforms multi-head attention by using only 50%-90% of
  the attention heads. Moreover, we demonstrate that pre-trained multi-head
  attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH
  models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14
  benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the
  attention heads. We believe the proposed MoH is a promising alternative to
  multi-head attention and provides a strong foundation for developing advanced
  and efficient attention-based models.
paper_authors._id: 6710bf3ff1279fe0dfe4e5b2
paper_authors.name: Peng Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:23:52.506Z
paper_authors._id: 651585ea18cfed0d30bee586
paper_authors.avatarUrl: /avatars/579e468334102472d870875fe40302e6.svg
paper_authors.fullname: Peng Jin
paper_authors.type: user
paper_authors.user: Chat-UniVi
paper_authors._id: 6710bf3ff1279fe0dfe4e5b3
paper_authors.name: Bo Zhu
paper_authors._id: 6710bf3ff1279fe0dfe4e5b4
paper_authors.name: Li Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:53:55.346Z
paper_authors._id: 614030bd8cbdb613b82f36a8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631596713749-noauth.jpeg
paper_authors.fullname: Li Yuan
paper_authors.type: user
paper_authors.user: LiYuan
paper_authors._id: 6710bf3ff1279fe0dfe4e5b5
paper_authors.name: Shuicheng Yan
paper_title: MoH: Multi-Head Attention as Mixture-of-Head Attention
paper_summary: In this work, we upgrade the multi-head attention mechanism, the core of the
  Transformer model, to improve efficiency while maintaining or surpassing the
  previous accuracy level. We show that multi-head attention can be expressed in
  the summation form. Drawing on the insight that not all attention heads hold
  equal significance, we propose Mixture-of-Head attention (MoH), a new
  architecture that treats attention heads as experts in the Mixture-of-Experts
  (MoE) mechanism. MoH has two significant advantages: First, MoH enables each
  token to select the appropriate attention heads, enhancing inference efficiency
  without compromising accuracy or increasing the number of parameters. Second,
  MoH replaces the standard summation in multi-head attention with a weighted
  summation, introducing flexibility to the attention mechanism and unlocking
  extra performance potential. Extensive experiments on ViT, DiT, and LLMs
  demonstrate that MoH outperforms multi-head attention by using only 50%-90% of
  the attention heads. Moreover, we demonstrate that pre-trained multi-head
  attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH
  models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14
  benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the
  attention heads. We believe the proposed MoH is a promising alternative to
  multi-head attention and provides a strong foundation for developing advanced
  and efficient attention-based models.
paper_ai_keywords: multi-head attention Transformer model Mixture-of-Head attention (MoH) Mixture-of-Experts (MoE) mechanism attention heads inference efficiency weighted summation ViT DiT LLMs LLaMA3-8B pre-trained multi-head attention models continue-tuning FID benchmarks MoH-LLaMA3-8B