MiniMax-01: Scaling Foundation Models with Lightning Attention
We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01,
which are comparable to top-tier models while offering superior capabilities in
processing longer contexts. The core lies in lightning attention and its
efficient scaling. To maximize computational capacity, we integrate it with
Mixture of Experts (MoE), creating a model with 32 experts and 456 billion
total parameters, of which 45.9 billion are activated for each token. We
develop an optimized parallel strategy and highly efficient
computation-communication overlap techniques for MoE and lightning attention.
This approach enables us to conduct efficient training and inference on models
with hundreds of billions of parameters across contexts spanning millions of
tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens
during training and extrapolate to 4 million tokens during inference at an
affordable cost. Our vision-language model, MiniMax-VL-01 is built through
continued training with 512 billion vision-language tokens. Experiments on both
standard and in-house benchmarks show that our models match the performance of
state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32
times longer context window. We publicly release MiniMax-01 at
https://github.com/MiniMax-AI.
67871e6ef492fb2235af8978
MiniMax
admin_assigned
2025-01-15T16:38:32.547Z
676e38ad04af5bec20bc9faf
https://cdn-avatars.huggingface.co/v1/production/uploads/676e38ad04af5bec20bc9faf/AG8Q9wAUzGtPWyjd5QO2l.jpeg
MiniMax
user
MiniMax-AI
67871e6ef492fb2235af8979
Aonian Li
claimed_verified
2025-07-10T07:10:29.890Z
678724bca6692d020b3d85b2
/avatars/2d72b777f0e5db035e5bce68a55bdcd5.svg
ao
user
leviel
67871e6ef492fb2235af897a
Bangwei Gong
67871e6ef492fb2235af897b
Bo Yang
67871e6ef492fb2235af897c
Boji Shan
admin_assigned
2025-01-15T16:37:05.624Z
656617f5b0e94dc742d5a8ad
https://cdn-avatars.huggingface.co/v1/production/uploads/656617f5b0e94dc742d5a8ad/UKqWKFKhWd6d7w5h1f3wU.jpeg
Boji Shan
user
kamuy-shennai
67871e6ef492fb2235af897d
Chang Liu
admin_assigned
2025-01-15T16:37:12.282Z
6638574ec78619ba63879031
/avatars/c9c95f129abfe7614493e926e7a7e971.svg
Chang Liu
user
changliu01
67871e6ef492fb2235af897e
Cheng Zhu
admin_assigned
2025-01-15T16:37:18.159Z
644b2ae13a619fe72b1806f9
/avatars/ceee1779b26792ffd28c6e9a23d3439a.svg
chengzhu
user
chengzhu
67871e6ef492fb2235af897f
Chunhao Zhang
claimed_verified
2025-01-15T08:48:51.874Z
642662fa22bddcea3d289f0a
/avatars/9b28e1325d866a24d33fdfafcaa85c4b.svg
Enoch Zhang
user
enochzhang
67871e6ef492fb2235af8980
Congchao Guo
67871e6ef492fb2235af8981
Da Chen
67871e6ef492fb2235af8982
Dong Li
claimed_verified
2025-01-16T08:31:45.832Z
61cd713f78c1c6b489a231fc
/avatars/af261b85bbd10a1182372ffc459640b8.svg
Dong Li
user
liddalidd
67871e6ef492fb2235af8983
Enwei Jiao
67871e6ef492fb2235af8984
Gengxin Li
67871e6ef492fb2235af8985
Guojun Zhang
admin_assigned
2025-01-16T09:00:15.292Z
6511d02684bedffb5aa9509e
/avatars/ed4f546f4fef206a71227513995d4d72.svg
Guojun Zhang
user
gordon-z
67871e6ef492fb2235af8986
Haohai Sun
admin_assigned
2025-01-16T09:03:56.423Z
64b76660f92b20f7a37c3df7
/avatars/40158717bb9370f1e5d0ed156a6fed1f.svg
HaohaiSun
user
HaohaiSun
67871e6ef492fb2235af8987
Houze Dong
67871e6ef492fb2235af8988
Jiadai Zhu
67871e6ef492fb2235af8989
Jiaqi Zhuang
67871e6ef492fb2235af898a
Jiayuan Song
67871e6ef492fb2235af898b
Jin Zhu
claimed_verified
2025-06-17T13:07:33.050Z
665d5c955491b1e10dfc4097
/avatars/1d577113e407068d29e61f670e662f81.svg
Jin Zhu
user
GinZhu
67871e6ef492fb2235af898c
Jingtao Han
67871e6ef492fb2235af898d
Jingyang Li
67871e6ef492fb2235af898e
Junbin Xie
67871e6ef492fb2235af898f
Junhao Xu
67871e6ef492fb2235af8990
Junjie Yan
admin_assigned
2025-01-16T09:03:32.671Z
63390ce41718795719635b1e
/avatars/ad03a2b349f01c1ac1fedfb95d02d43e.svg
JunjieYan
user
JunjieYan
67871e6ef492fb2235af8991
Kaishun Zhang
67871e6ef492fb2235af8992
Kecheng Xiao
admin_assigned
2025-01-16T09:02:25.327Z
634788bc6f8773f2a28e0eb5
/avatars/463af54e7a9279ed93e58b403c6197ca.svg
kecheng xiao
user
Drakos
67871e6ef492fb2235af8993
Kexi Kang
67871e6ef492fb2235af8994
Le Han
claimed_verified
2025-06-19T09:11:21.092Z
67872659672a9d8e05bc4bad
https://cdn-avatars.huggingface.co/v1/production/uploads/67872659672a9d8e05bc4bad/4rlrbngAm8DQwPK7UCfjG.png
韩乐
user
lee3306
67871e6ef492fb2235af8995
Leyang Wang
67871e6ef492fb2235af8996
Lianfei Yu
67871e6ef492fb2235af8997
Liheng Feng
67871e6ef492fb2235af8998
Lin Zheng
67871e6ef492fb2235af8999
Linbo Chai
67871e6ef492fb2235af899a
Long Xing
67871e6ef492fb2235af899b
Meizhi Ju
67871e6ef492fb2235af899c
Mingyuan Chi
67871e6ef492fb2235af899d
Mozhi Zhang
67871e6ef492fb2235af899e
Peikai Huang
67871e6ef492fb2235af899f
Pengcheng Niu
67871e6ef492fb2235af89a0
Pengfei Li
67871e6ef492fb2235af89a1
Pengyu Zhao
67871e6ef492fb2235af89a2
Qi Yang
67871e6ef492fb2235af89a3
Qidi Xu
67871e6ef492fb2235af89a4
Qiexiang Wang
admin_assigned
2025-01-16T09:01:58.949Z
665897f1df3aa79d9bb271a4
/avatars/f78e7322e6d7e7f6de5c50f8c87819dc.svg
Qiexiang Wang
user
wzyxwqx
67871e6ef492fb2235af89a5
Qin Wang
67871e6ef492fb2235af89a6
Qiuhui Li
claimed_verified
2025-06-19T09:11:19.163Z
64f0989ddbfec9d148940e90
/avatars/a2ab9ac5513285e1d95be9766dba3454.svg
li
user
DQmyth
67871e6ef492fb2235af89a7
Ruitao Leng
67871e6ef492fb2235af89a8
Shengmin Shi
67871e6ef492fb2235af89a9
Shuqi Yu
67871e6ef492fb2235af89aa
Sichen Li
67871e6ef492fb2235af89ab
Songquan Zhu
67871e6ef492fb2235af89ac
Tao Huang
67871e6ef492fb2235af89ad
Tianrun Liang
admin_assigned
2025-01-15T16:38:47.564Z
667646b9d5ac75b5915003b2
/avatars/44f8c3be66feae5fc828e3e23e52989e.svg
liangtianrun
user
andachi
67871e6ef492fb2235af89ae
Weigao Sun
claimed_verified
2025-02-13T08:26:18.454Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
67871e6ef492fb2235af89af
Weixuan Sun
67871e6ef492fb2235af89b0
Weiyu Cheng
67871e6ef492fb2235af89b1
Wenkai Li
67871e6ef492fb2235af89b2
Xiangjun Song
67871e6ef492fb2235af89b3
Xiao Su
67871e6ef492fb2235af89b4
Xiaodong Han
claimed_verified
2025-01-15T08:48:47.808Z
64638bddc615cbc12447f9f1
/avatars/52910d1717451ff983f745322e5850dd.svg
Xiaodong Han
user
Hannnnnxd
67871e6ef492fb2235af89b5
Xinjie Zhang
67871e6ef492fb2235af89b6
Xinzhu Hou
67871e6ef492fb2235af89b7
Xu Min
67871e6ef492fb2235af89b8
Xun Zou
67871e6ef492fb2235af89b9
Xuyang Shen
claimed_verified
2025-05-26T08:43:42.233Z
642e4d4d6748dd4f8eeb7732
/avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
Xuyang Shen
user
Ryan1122
67871e6ef492fb2235af89ba
Yan Gong
67871e6ef492fb2235af89bb
Yingjie Zhu
67871e6ef492fb2235af89bc
Yipeng Zhou
67871e6ef492fb2235af89bd
Yiran Zhong
claimed_verified
2025-01-17T10:26:51.780Z
64c525e4d68946edad6c7067
/avatars/1b108661634af602717a4ab4b66a151f.svg
Yiran Zhong
user
IanZhong
67871e6ef492fb2235af89be
Yongyi Hu
67871e6ef492fb2235af89bf
Yuanxiang Fan
admin_assigned
2025-01-16T09:00:47.428Z
64813e4b3fb124fc98503a7e
/avatars/f871b7d84f04f827041d4a23cb1cdc9f.svg
Yuanxiang Fan
user
ShiroFFF
67871e6ef492fb2235af89c0
Yue Yu
67871e6ef492fb2235af89c1
Yufeng Yang
67871e6ef492fb2235af89c2
Yuhao Li
67871e6ef492fb2235af89c3
Yunan Huang
67871e6ef492fb2235af89c4
Yunji Li
67871e6ef492fb2235af89c5
Yunpeng Huang
67871e6ef492fb2235af89c6
Yunzhi Xu
67871e6ef492fb2235af89c7
Yuxin Mao
67871e6ef492fb2235af89c8
Zehan Li
67871e6ef492fb2235af89c9
Zekang Li
67871e6ef492fb2235af89ca
Zewei Tao
67871e6ef492fb2235af89cb
Zewen Ying
67871e6ef492fb2235af89cc
Zhaoyang Cong
67871e6ef492fb2235af89cd
Zhen Qin
67871e6ef492fb2235af89ce
Zhenhua Fan
67871e6ef492fb2235af89cf
Zhihang Yu
67871e6ef492fb2235af89d0
Zhuo Jiang
67871e6ef492fb2235af89d1
Zijia Wu
MiniMax-01: Scaling Foundation Models with Lightning Attention
We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01,
which are comparable to top-tier models while offering superior capabilities in
processing longer contexts. The core lies in lightning attention and its
efficient scaling. To maximize computational capacity, we integrate it with
Mixture of Experts (MoE), creating a model with 32 experts and 456 billion
total parameters, of which 45.9 billion are activated for each token. We
develop an optimized parallel strategy and highly efficient
computation-communication overlap techniques for MoE and lightning attention.
This approach enables us to conduct efficient training and inference on models
with hundreds of billions of parameters across contexts spanning millions of
tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens
during training and extrapolate to 4 million tokens during inference at an
affordable cost. Our vision-language model, MiniMax-VL-01 is built through
continued training with 512 billion vision-language tokens. Experiments on both
standard and in-house benchmarks show that our models match the performance of
state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32
times longer context window. We publicly release MiniMax-01 at
https://github.com/MiniMax-AI.
The MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, offer superior long-context processing and match state-of-the-art model performance with longer context windows through lightning attention, Mixture of Experts (MoE), and efficient parallel strategies.
lightning attention Mixture of Experts MoE parallel strategy computation-communication overlap context window vision-language model