MoBA: Mixture of Block Attention for Long-Context LLMs
Scaling the effective context length is essential for advancing large
language models (LLMs) toward artificial general intelligence (AGI). However,
the quadratic increase in computational complexity inherent in traditional
attention mechanisms presents a prohibitive overhead. Existing approaches
either impose strongly biased structures, such as sink or window attention
which are task-specific, or radically modify the attention mechanism into
linear approximations, whose performance in complex reasoning tasks remains
inadequately explored.
  In this work, we propose a solution that adheres to the ``less structure''
principle, allowing the model to determine where to attend autonomously, rather
than introducing predefined biases. We introduce Mixture of Block Attention
(MoBA), an innovative approach that applies the principles of Mixture of
Experts (MoE) to the attention mechanism. This novel architecture demonstrates
superior performance on long-context tasks while offering a key advantage: the
ability to seamlessly transition between full and sparse attention, enhancing
efficiency without the risk of compromising performance. MoBA has already been
deployed to support Kimi's long-context requests and demonstrates significant
advancements in efficient attention computation for LLMs. Our code is available
at https://github.com/MoonshotAI/MoBA.
67b7152f299e4d30f9eb41c2
Enzhe Lu
admin_assigned
2025-02-24T15:48:22.594Z
67aed930cc96f87ce3c3132f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JDrhmbCRcuCtKir7i9z9n.png
Lu
user
Enzhe
67b7152f299e4d30f9eb41c3
Zhejun Jiang
admin_assigned
2025-02-24T15:48:29.904Z
662c6e8352e194d5d44d873c
/avatars/385a5cc7299faf2f61ccbabedd827f29.svg
Zhejun Jiang
user
Skewed
67b7152f299e4d30f9eb41c4
Jingyuan Liu
admin_assigned
2025-02-24T15:48:36.573Z
64ead9e35349043b2b941a03
/avatars/e9acef299086f0245ff364d9d7889007.svg
JingyuanLiu
user
JingyuanLiu
67b7152f299e4d30f9eb41c5
Yulun Du
admin_assigned
2025-02-24T15:48:47.033Z
6340f31fb78ed99eab04ce33
/avatars/2e7fcbf0233bdc0bc9a3f4603fd8bf90.svg
Du
user
Yulun
67b7152f299e4d30f9eb41c6
Tao Jiang
67b7152f299e4d30f9eb41c7
Chao Hong
67b7152f299e4d30f9eb41c8
Shaowei Liu
67b7152f299e4d30f9eb41c9
Weiran He
claimed_verified
2025-04-15T08:01:51.861Z
63a41c1f412fd71fb7ec4a20
/avatars/d6837b0dbbf3507896e98754d3f8a468.svg
Wayne Ho
user
hewr2010
67b7152f299e4d30f9eb41ca
Enming Yuan
admin_assigned
2025-02-24T15:49:40.245Z
6331606f18711776b4655e67
/avatars/1479c2ca743b9f92d845b0ed23fcd07b.svg
Enming Yuan
user
EnmingYuan
67b7152f299e4d30f9eb41cb
Yuzhi Wang
admin_assigned
2025-02-24T15:49:48.436Z
67127a470a82509269d738ae
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/M9qLmI3P6dT2FIwEPFJq0.png
yuzhi wang
user
vin-tage
67b7152f299e4d30f9eb41cc
Zhiqi Huang
admin_assigned
2025-02-24T15:49:54.439Z
66221f1a90f3fd333c4ec52e
/avatars/a3173d9603a69020ec24170831c97c2f.svg
Zhiqi Huang
user
Angelalilyer
67b7152f299e4d30f9eb41cd
Huan Yuan
67b7152f299e4d30f9eb41ce
Suting Xu
admin_assigned
2025-02-24T15:50:08.223Z
649e7c2afbdfd3c16128ce6e
/avatars/1ff863b0fa39cfe4285255e4417c1db4.svg
Suting Xu
user
susu1210
67b7152f299e4d30f9eb41cf
Xinran Xu
67b7152f299e4d30f9eb41d0
Guokun Lai
admin_assigned
2025-02-24T15:50:25.601Z
63b4c71758f367a212c4f9ef
/avatars/d61736e0ae8b333a7c24eb411378698c.svg
Lai
user
Guokun
67b7152f299e4d30f9eb41d1
Yanru Chen
admin_assigned
2025-02-24T15:50:32.381Z
6365df6912188d67e65f5c5b
/avatars/59a1d2f30ba4faea0336bedf4df321a8.svg
Yanru Chen
user
AChen-qaq
67b7152f299e4d30f9eb41d2
Huabin Zheng
admin_assigned
2025-02-24T15:50:38.272Z
61860e1258cb1f8c362f9441
/avatars/8dbc8209ad0d918453c1ffacc8f61e7f.svg
Huabin Zheng
user
zhenghuabin
67b7152f299e4d30f9eb41d3
Junjie Yan
67b7152f299e4d30f9eb41d4
Jianlin Su
admin_assigned
2025-02-24T15:50:44.579Z
6404982cad54665351d7c1e0
/avatars/8fb6d01802cbd4a1cbb7f6a0d83faa3a.svg
jianlin su
user
bojone
67b7152f299e4d30f9eb41d5
Yuxin Wu
67b7152f299e4d30f9eb41d6
Neo Y. Zhang
67b7152f299e4d30f9eb41d7
Zhilin Yang
admin_assigned
2025-02-24T15:51:03.452Z
64bf74154d2052b1aa5ca6d9
/avatars/7aa6f2952cdbc20cfa758fdd905f06a6.svg
ZHILIN YANG
user
bruceyannnn
67b7152f299e4d30f9eb41d8
Xinyu Zhou
67b7152f299e4d30f9eb41d9
Mingxing Zhang
67b7152f299e4d30f9eb41da
Jiezhong Qiu
admin_assigned
2025-02-24T15:51:14.319Z
64b4fb7146dd1c352b0da25a
/avatars/b5c15bca8020c4841a87252ce9ed1618.svg
Jiezhong Qiu
user
xptree
MoBA: Mixture of Block Attention for Long-Context LLMs
Scaling the effective context length is essential for advancing large
language models (LLMs) toward artificial general intelligence (AGI). However,
the quadratic increase in computational complexity inherent in traditional
attention mechanisms presents a prohibitive overhead. Existing approaches
either impose strongly biased structures, such as sink or window attention
which are task-specific, or radically modify the attention mechanism into
linear approximations, whose performance in complex reasoning tasks remains
inadequately explored.
  In this work, we propose a solution that adheres to the ``less structure''
principle, allowing the model to determine where to attend autonomously, rather
than introducing predefined biases. We introduce Mixture of Block Attention
(MoBA), an innovative approach that applies the principles of Mixture of
Experts (MoE) to the attention mechanism. This novel architecture demonstrates
superior performance on long-context tasks while offering a key advantage: the
ability to seamlessly transition between full and sparse attention, enhancing
efficiency without the risk of compromising performance. MoBA has already been
deployed to support Kimi's long-context requests and demonstrates significant
advancements in efficient attention computation for LLMs. Our code is available
at https://github.com/MoonshotAI/MoBA.
Mixture of Block Attention (MoBA) improves large language models' efficiency and performance on long-context tasks by dynamically switching between full and sparse attention without predefined biases.
Mixture of Block Attention MoBA attention mechanisms Mixture of Experts MoE long-context tasks full attention sparse attention efficient attention computation