title: MoBA: Mixture of Block Attention for Long-Context LLMs
summary: Scaling the effective context length is essential for advancing large
  language models (LLMs) toward artificial general intelligence (AGI). However,
  the quadratic increase in computational complexity inherent in traditional
  attention mechanisms presents a prohibitive overhead. Existing approaches
  either impose strongly biased structures, such as sink or window attention
  which are task-specific, or radically modify the attention mechanism into
  linear approximations, whose performance in complex reasoning tasks remains
  inadequately explored.
    In this work, we propose a solution that adheres to the ``less structure''
  principle, allowing the model to determine where to attend autonomously, rather
  than introducing predefined biases. We introduce Mixture of Block Attention
  (MoBA), an innovative approach that applies the principles of Mixture of
  Experts (MoE) to the attention mechanism. This novel architecture demonstrates
  superior performance on long-context tasks while offering a key advantage: the
  ability to seamlessly transition between full and sparse attention, enhancing
  efficiency without the risk of compromising performance. MoBA has already been
  deployed to support Kimi's long-context requests and demonstrates significant
  advancements in efficient attention computation for LLMs. Our code is available
  at https://github.com/MoonshotAI/MoBA.
paper_authors._id: 67b7152f299e4d30f9eb41c2
paper_authors.name: Enzhe Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:48:22.594Z
paper_authors._id: 67aed930cc96f87ce3c3132f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JDrhmbCRcuCtKir7i9z9n.png
paper_authors.fullname: Lu
paper_authors.type: user
paper_authors.user: Enzhe
paper_authors._id: 67b7152f299e4d30f9eb41c3
paper_authors.name: Zhejun Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:48:29.904Z
paper_authors._id: 662c6e8352e194d5d44d873c
paper_authors.avatarUrl: /avatars/385a5cc7299faf2f61ccbabedd827f29.svg
paper_authors.fullname: Zhejun Jiang
paper_authors.type: user
paper_authors.user: Skewed
paper_authors._id: 67b7152f299e4d30f9eb41c4
paper_authors.name: Jingyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:48:36.573Z
paper_authors._id: 64ead9e35349043b2b941a03
paper_authors.avatarUrl: /avatars/e9acef299086f0245ff364d9d7889007.svg
paper_authors.fullname: JingyuanLiu
paper_authors.type: user
paper_authors.user: JingyuanLiu
paper_authors._id: 67b7152f299e4d30f9eb41c5
paper_authors.name: Yulun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:48:47.033Z
paper_authors._id: 6340f31fb78ed99eab04ce33
paper_authors.avatarUrl: /avatars/2e7fcbf0233bdc0bc9a3f4603fd8bf90.svg
paper_authors.fullname: Du
paper_authors.type: user
paper_authors.user: Yulun
paper_authors._id: 67b7152f299e4d30f9eb41c6
paper_authors.name: Tao Jiang
paper_authors._id: 67b7152f299e4d30f9eb41c7
paper_authors.name: Chao Hong
paper_authors._id: 67b7152f299e4d30f9eb41c8
paper_authors.name: Shaowei Liu
paper_authors._id: 67b7152f299e4d30f9eb41c9
paper_authors.name: Weiran He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T08:01:51.861Z
paper_authors._id: 63a41c1f412fd71fb7ec4a20
paper_authors.avatarUrl: /avatars/d6837b0dbbf3507896e98754d3f8a468.svg
paper_authors.fullname: Wayne Ho
paper_authors.type: user
paper_authors.user: hewr2010
paper_authors._id: 67b7152f299e4d30f9eb41ca
paper_authors.name: Enming Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:49:40.245Z
paper_authors._id: 6331606f18711776b4655e67
paper_authors.avatarUrl: /avatars/1479c2ca743b9f92d845b0ed23fcd07b.svg
paper_authors.fullname: Enming Yuan
paper_authors.type: user
paper_authors.user: EnmingYuan
paper_authors._id: 67b7152f299e4d30f9eb41cb
paper_authors.name: Yuzhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:49:48.436Z
paper_authors._id: 67127a470a82509269d738ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/M9qLmI3P6dT2FIwEPFJq0.png
paper_authors.fullname: yuzhi wang
paper_authors.type: user
paper_authors.user: vin-tage
paper_authors._id: 67b7152f299e4d30f9eb41cc
paper_authors.name: Zhiqi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:49:54.439Z
paper_authors._id: 66221f1a90f3fd333c4ec52e
paper_authors.avatarUrl: /avatars/a3173d9603a69020ec24170831c97c2f.svg
paper_authors.fullname: Zhiqi Huang
paper_authors.type: user
paper_authors.user: Angelalilyer
paper_authors._id: 67b7152f299e4d30f9eb41cd
paper_authors.name: Huan Yuan
paper_authors._id: 67b7152f299e4d30f9eb41ce
paper_authors.name: Suting Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:50:08.223Z
paper_authors._id: 649e7c2afbdfd3c16128ce6e
paper_authors.avatarUrl: /avatars/1ff863b0fa39cfe4285255e4417c1db4.svg
paper_authors.fullname: Suting Xu
paper_authors.type: user
paper_authors.user: susu1210
paper_authors._id: 67b7152f299e4d30f9eb41cf
paper_authors.name: Xinran Xu
paper_authors._id: 67b7152f299e4d30f9eb41d0
paper_authors.name: Guokun Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:50:25.601Z
paper_authors._id: 63b4c71758f367a212c4f9ef
paper_authors.avatarUrl: /avatars/d61736e0ae8b333a7c24eb411378698c.svg
paper_authors.fullname: Lai
paper_authors.type: user
paper_authors.user: Guokun
paper_authors._id: 67b7152f299e4d30f9eb41d1
paper_authors.name: Yanru Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:50:32.381Z
paper_authors._id: 6365df6912188d67e65f5c5b
paper_authors.avatarUrl: /avatars/59a1d2f30ba4faea0336bedf4df321a8.svg
paper_authors.fullname: Yanru Chen
paper_authors.type: user
paper_authors.user: AChen-qaq
paper_authors._id: 67b7152f299e4d30f9eb41d2
paper_authors.name: Huabin Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:50:38.272Z
paper_authors._id: 61860e1258cb1f8c362f9441
paper_authors.avatarUrl: /avatars/8dbc8209ad0d918453c1ffacc8f61e7f.svg
paper_authors.fullname: Huabin Zheng
paper_authors.type: user
paper_authors.user: zhenghuabin
paper_authors._id: 67b7152f299e4d30f9eb41d3
paper_authors.name: Junjie Yan
paper_authors._id: 67b7152f299e4d30f9eb41d4
paper_authors.name: Jianlin Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:50:44.579Z
paper_authors._id: 6404982cad54665351d7c1e0
paper_authors.avatarUrl: /avatars/8fb6d01802cbd4a1cbb7f6a0d83faa3a.svg
paper_authors.fullname: jianlin su
paper_authors.type: user
paper_authors.user: bojone
paper_authors._id: 67b7152f299e4d30f9eb41d5
paper_authors.name: Yuxin Wu
paper_authors._id: 67b7152f299e4d30f9eb41d6
paper_authors.name: Neo Y. Zhang
paper_authors._id: 67b7152f299e4d30f9eb41d7
paper_authors.name: Zhilin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:51:03.452Z
paper_authors._id: 64bf74154d2052b1aa5ca6d9
paper_authors.avatarUrl: /avatars/7aa6f2952cdbc20cfa758fdd905f06a6.svg
paper_authors.fullname: ZHILIN YANG
paper_authors.type: user
paper_authors.user: bruceyannnn
paper_authors._id: 67b7152f299e4d30f9eb41d8
paper_authors.name: Xinyu Zhou
paper_authors._id: 67b7152f299e4d30f9eb41d9
paper_authors.name: Mingxing Zhang
paper_authors._id: 67b7152f299e4d30f9eb41da
paper_authors.name: Jiezhong Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:51:14.319Z
paper_authors._id: 64b4fb7146dd1c352b0da25a
paper_authors.avatarUrl: /avatars/b5c15bca8020c4841a87252ce9ed1618.svg
paper_authors.fullname: Jiezhong Qiu
paper_authors.type: user
paper_authors.user: xptree
paper_title: MoBA: Mixture of Block Attention for Long-Context LLMs
paper_summary: Scaling the effective context length is essential for advancing large
  language models (LLMs) toward artificial general intelligence (AGI). However,
  the quadratic increase in computational complexity inherent in traditional
  attention mechanisms presents a prohibitive overhead. Existing approaches
  either impose strongly biased structures, such as sink or window attention
  which are task-specific, or radically modify the attention mechanism into
  linear approximations, whose performance in complex reasoning tasks remains
  inadequately explored.
    In this work, we propose a solution that adheres to the ``less structure''
  principle, allowing the model to determine where to attend autonomously, rather
  than introducing predefined biases. We introduce Mixture of Block Attention
  (MoBA), an innovative approach that applies the principles of Mixture of
  Experts (MoE) to the attention mechanism. This novel architecture demonstrates
  superior performance on long-context tasks while offering a key advantage: the
  ability to seamlessly transition between full and sparse attention, enhancing
  efficiency without the risk of compromising performance. MoBA has already been
  deployed to support Kimi's long-context requests and demonstrates significant
  advancements in efficient attention computation for LLMs. Our code is available
  at https://github.com/MoonshotAI/MoBA.
paper_ai_summary: Mixture of Block Attention (MoBA) improves large language models' efficiency and performance on long-context tasks by dynamically switching between full and sparse attention without predefined biases.
paper_ai_keywords: Mixture of Block Attention MoBA attention mechanisms Mixture of Experts MoE long-context tasks full attention sparse attention efficient attention computation