title: MiniMax-01: Scaling Foundation Models with Lightning Attention
summary: We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01,
  which are comparable to top-tier models while offering superior capabilities in
  processing longer contexts. The core lies in lightning attention and its
  efficient scaling. To maximize computational capacity, we integrate it with
  Mixture of Experts (MoE), creating a model with 32 experts and 456 billion
  total parameters, of which 45.9 billion are activated for each token. We
  develop an optimized parallel strategy and highly efficient
  computation-communication overlap techniques for MoE and lightning attention.
  This approach enables us to conduct efficient training and inference on models
  with hundreds of billions of parameters across contexts spanning millions of
  tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens
  during training and extrapolate to 4 million tokens during inference at an
  affordable cost. Our vision-language model, MiniMax-VL-01 is built through
  continued training with 512 billion vision-language tokens. Experiments on both
  standard and in-house benchmarks show that our models match the performance of
  state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32
  times longer context window. We publicly release MiniMax-01 at
  https://github.com/MiniMax-AI.
paper_authors._id: 67871e6ef492fb2235af8978
paper_authors.name: MiniMax
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:38:32.547Z
paper_authors._id: 676e38ad04af5bec20bc9faf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/676e38ad04af5bec20bc9faf/AG8Q9wAUzGtPWyjd5QO2l.jpeg
paper_authors.fullname: MiniMax
paper_authors.type: user
paper_authors.user: MiniMax-AI
paper_authors._id: 67871e6ef492fb2235af8979
paper_authors.name: Aonian Li
paper_authors._id: 67871e6ef492fb2235af897a
paper_authors.name: Bangwei Gong
paper_authors._id: 67871e6ef492fb2235af897b
paper_authors.name: Bo Yang
paper_authors._id: 67871e6ef492fb2235af897c
paper_authors.name: Boji Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:37:05.624Z
paper_authors._id: 656617f5b0e94dc742d5a8ad
paper_authors.avatarUrl: /avatars/ea43dbdfdc0e880efa695632367f5608.svg
paper_authors.fullname: Boji Shan
paper_authors.type: user
paper_authors.user: kamuy-shennai
paper_authors._id: 67871e6ef492fb2235af897d
paper_authors.name: Chang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:37:12.282Z
paper_authors._id: 6638574ec78619ba63879031
paper_authors.avatarUrl: /avatars/c9c95f129abfe7614493e926e7a7e971.svg
paper_authors.fullname: Chang Liu
paper_authors.type: user
paper_authors.user: changliu01
paper_authors._id: 67871e6ef492fb2235af897e
paper_authors.name: Cheng Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:37:18.159Z
paper_authors._id: 644b2ae13a619fe72b1806f9
paper_authors.avatarUrl: /avatars/ceee1779b26792ffd28c6e9a23d3439a.svg
paper_authors.fullname: chengzhu
paper_authors.type: user
paper_authors.user: chengzhu
paper_authors._id: 67871e6ef492fb2235af897f
paper_authors.name: Chunhao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:48:51.874Z
paper_authors._id: 642662fa22bddcea3d289f0a
paper_authors.avatarUrl: /avatars/9b28e1325d866a24d33fdfafcaa85c4b.svg
paper_authors.fullname: Enoch Zhang
paper_authors.type: user
paper_authors.user: enochzhang
paper_authors._id: 67871e6ef492fb2235af8980
paper_authors.name: Congchao Guo
paper_authors._id: 67871e6ef492fb2235af8981
paper_authors.name: Da Chen
paper_authors._id: 67871e6ef492fb2235af8982
paper_authors.name: Dong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T08:31:45.832Z
paper_authors._id: 61cd713f78c1c6b489a231fc
paper_authors.avatarUrl: /avatars/af261b85bbd10a1182372ffc459640b8.svg
paper_authors.fullname: Dong Li
paper_authors.type: user
paper_authors.user: liddalidd
paper_authors._id: 67871e6ef492fb2235af8983
paper_authors.name: Enwei Jiao
paper_authors._id: 67871e6ef492fb2235af8984
paper_authors.name: Gengxin Li
paper_authors._id: 67871e6ef492fb2235af8985
paper_authors.name: Guojun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:00:15.292Z
paper_authors._id: 6511d02684bedffb5aa9509e
paper_authors.avatarUrl: /avatars/ed4f546f4fef206a71227513995d4d72.svg
paper_authors.fullname: Guojun Zhang
paper_authors.type: user
paper_authors.user: gordon-z
paper_authors._id: 67871e6ef492fb2235af8986
paper_authors.name: Haohai Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:03:56.423Z
paper_authors._id: 64b76660f92b20f7a37c3df7
paper_authors.avatarUrl: /avatars/40158717bb9370f1e5d0ed156a6fed1f.svg
paper_authors.fullname: HaohaiSun
paper_authors.type: user
paper_authors.user: HaohaiSun
paper_authors._id: 67871e6ef492fb2235af8987
paper_authors.name: Houze Dong
paper_authors._id: 67871e6ef492fb2235af8988
paper_authors.name: Jiadai Zhu
paper_authors._id: 67871e6ef492fb2235af8989
paper_authors.name: Jiaqi Zhuang
paper_authors._id: 67871e6ef492fb2235af898a
paper_authors.name: Jiayuan Song
paper_authors._id: 67871e6ef492fb2235af898b
paper_authors.name: Jin Zhu
paper_authors._id: 67871e6ef492fb2235af898c
paper_authors.name: Jingtao Han
paper_authors._id: 67871e6ef492fb2235af898d
paper_authors.name: Jingyang Li
paper_authors._id: 67871e6ef492fb2235af898e
paper_authors.name: Junbin Xie
paper_authors._id: 67871e6ef492fb2235af898f
paper_authors.name: Junhao Xu
paper_authors._id: 67871e6ef492fb2235af8990
paper_authors.name: Junjie Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:03:32.671Z
paper_authors._id: 63390ce41718795719635b1e
paper_authors.avatarUrl: /avatars/ad03a2b349f01c1ac1fedfb95d02d43e.svg
paper_authors.fullname: JunjieYan
paper_authors.type: user
paper_authors.user: JunjieYan
paper_authors._id: 67871e6ef492fb2235af8991
paper_authors.name: Kaishun Zhang
paper_authors._id: 67871e6ef492fb2235af8992
paper_authors.name: Kecheng Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:02:25.327Z
paper_authors._id: 634788bc6f8773f2a28e0eb5
paper_authors.avatarUrl: /avatars/463af54e7a9279ed93e58b403c6197ca.svg
paper_authors.fullname: kecheng xiao
paper_authors.type: user
paper_authors.user: Drakos
paper_authors._id: 67871e6ef492fb2235af8993
paper_authors.name: Kexi Kang
paper_authors._id: 67871e6ef492fb2235af8994
paper_authors.name: Le Han
paper_authors._id: 67871e6ef492fb2235af8995
paper_authors.name: Leyang Wang
paper_authors._id: 67871e6ef492fb2235af8996
paper_authors.name: Lianfei Yu
paper_authors._id: 67871e6ef492fb2235af8997
paper_authors.name: Liheng Feng
paper_authors._id: 67871e6ef492fb2235af8998
paper_authors.name: Lin Zheng
paper_authors._id: 67871e6ef492fb2235af8999
paper_authors.name: Linbo Chai
paper_authors._id: 67871e6ef492fb2235af899a
paper_authors.name: Long Xing
paper_authors._id: 67871e6ef492fb2235af899b
paper_authors.name: Meizhi Ju
paper_authors._id: 67871e6ef492fb2235af899c
paper_authors.name: Mingyuan Chi
paper_authors._id: 67871e6ef492fb2235af899d
paper_authors.name: Mozhi Zhang
paper_authors._id: 67871e6ef492fb2235af899e
paper_authors.name: Peikai Huang
paper_authors._id: 67871e6ef492fb2235af899f
paper_authors.name: Pengcheng Niu
paper_authors._id: 67871e6ef492fb2235af89a0
paper_authors.name: Pengfei Li
paper_authors._id: 67871e6ef492fb2235af89a1
paper_authors.name: Pengyu Zhao
paper_authors._id: 67871e6ef492fb2235af89a2
paper_authors.name: Qi Yang
paper_authors._id: 67871e6ef492fb2235af89a3
paper_authors.name: Qidi Xu
paper_authors._id: 67871e6ef492fb2235af89a4
paper_authors.name: Qiexiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:01:58.949Z
paper_authors._id: 665897f1df3aa79d9bb271a4
paper_authors.avatarUrl: /avatars/f78e7322e6d7e7f6de5c50f8c87819dc.svg
paper_authors.fullname: Qiexiang Wang
paper_authors.type: user
paper_authors.user: wzyxwqx
paper_authors._id: 67871e6ef492fb2235af89a5
paper_authors.name: Qin Wang
paper_authors._id: 67871e6ef492fb2235af89a6
paper_authors.name: Qiuhui Li
paper_authors._id: 67871e6ef492fb2235af89a7
paper_authors.name: Ruitao Leng
paper_authors._id: 67871e6ef492fb2235af89a8
paper_authors.name: Shengmin Shi
paper_authors._id: 67871e6ef492fb2235af89a9
paper_authors.name: Shuqi Yu
paper_authors._id: 67871e6ef492fb2235af89aa
paper_authors.name: Sichen Li
paper_authors._id: 67871e6ef492fb2235af89ab
paper_authors.name: Songquan Zhu
paper_authors._id: 67871e6ef492fb2235af89ac
paper_authors.name: Tao Huang
paper_authors._id: 67871e6ef492fb2235af89ad
paper_authors.name: Tianrun Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:38:47.564Z
paper_authors._id: 667646b9d5ac75b5915003b2
paper_authors.avatarUrl: /avatars/44f8c3be66feae5fc828e3e23e52989e.svg
paper_authors.fullname: liangtianrun
paper_authors.type: user
paper_authors.user: andachi
paper_authors._id: 67871e6ef492fb2235af89ae
paper_authors.name: Weigao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:26:18.454Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 67871e6ef492fb2235af89af
paper_authors.name: Weixuan Sun
paper_authors._id: 67871e6ef492fb2235af89b0
paper_authors.name: Weiyu Cheng
paper_authors._id: 67871e6ef492fb2235af89b1
paper_authors.name: Wenkai Li
paper_authors._id: 67871e6ef492fb2235af89b2
paper_authors.name: Xiangjun Song
paper_authors._id: 67871e6ef492fb2235af89b3
paper_authors.name: Xiao Su
paper_authors._id: 67871e6ef492fb2235af89b4
paper_authors.name: Xiaodong Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:48:47.808Z
paper_authors._id: 64638bddc615cbc12447f9f1
paper_authors.avatarUrl: /avatars/52910d1717451ff983f745322e5850dd.svg
paper_authors.fullname: Xiaodong Han
paper_authors.type: user
paper_authors.user: Hannnnnxd
paper_authors._id: 67871e6ef492fb2235af89b5
paper_authors.name: Xinjie Zhang
paper_authors._id: 67871e6ef492fb2235af89b6
paper_authors.name: Xinzhu Hou
paper_authors._id: 67871e6ef492fb2235af89b7
paper_authors.name: Xu Min
paper_authors._id: 67871e6ef492fb2235af89b8
paper_authors.name: Xun Zou
paper_authors._id: 67871e6ef492fb2235af89b9
paper_authors.name: Xuyang Shen
paper_authors._id: 67871e6ef492fb2235af89ba
paper_authors.name: Yan Gong
paper_authors._id: 67871e6ef492fb2235af89bb
paper_authors.name: Yingjie Zhu
paper_authors._id: 67871e6ef492fb2235af89bc
paper_authors.name: Yipeng Zhou
paper_authors._id: 67871e6ef492fb2235af89bd
paper_authors.name: Yiran Zhong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T10:26:51.780Z
paper_authors._id: 64c525e4d68946edad6c7067
paper_authors.avatarUrl: /avatars/1b108661634af602717a4ab4b66a151f.svg
paper_authors.fullname: Yiran Zhong
paper_authors.type: user
paper_authors.user: IanZhong
paper_authors._id: 67871e6ef492fb2235af89be
paper_authors.name: Yongyi Hu
paper_authors._id: 67871e6ef492fb2235af89bf
paper_authors.name: Yuanxiang Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:00:47.428Z
paper_authors._id: 64813e4b3fb124fc98503a7e
paper_authors.avatarUrl: /avatars/f871b7d84f04f827041d4a23cb1cdc9f.svg
paper_authors.fullname: Yuanxiang Fan
paper_authors.type: user
paper_authors.user: ShiroFFF
paper_authors._id: 67871e6ef492fb2235af89c0
paper_authors.name: Yue Yu
paper_authors._id: 67871e6ef492fb2235af89c1
paper_authors.name: Yufeng Yang
paper_authors._id: 67871e6ef492fb2235af89c2
paper_authors.name: Yuhao Li
paper_authors._id: 67871e6ef492fb2235af89c3
paper_authors.name: Yunan Huang
paper_authors._id: 67871e6ef492fb2235af89c4
paper_authors.name: Yunji Li
paper_authors._id: 67871e6ef492fb2235af89c5
paper_authors.name: Yunpeng Huang
paper_authors._id: 67871e6ef492fb2235af89c6
paper_authors.name: Yunzhi Xu
paper_authors._id: 67871e6ef492fb2235af89c7
paper_authors.name: Yuxin Mao
paper_authors._id: 67871e6ef492fb2235af89c8
paper_authors.name: Zehan Li
paper_authors._id: 67871e6ef492fb2235af89c9
paper_authors.name: Zekang Li
paper_authors._id: 67871e6ef492fb2235af89ca
paper_authors.name: Zewei Tao
paper_authors._id: 67871e6ef492fb2235af89cb
paper_authors.name: Zewen Ying
paper_authors._id: 67871e6ef492fb2235af89cc
paper_authors.name: Zhaoyang Cong
paper_authors._id: 67871e6ef492fb2235af89cd
paper_authors.name: Zhen Qin
paper_authors._id: 67871e6ef492fb2235af89ce
paper_authors.name: Zhenhua Fan
paper_authors._id: 67871e6ef492fb2235af89cf
paper_authors.name: Zhihang Yu
paper_authors._id: 67871e6ef492fb2235af89d0
paper_authors.name: Zhuo Jiang
paper_authors._id: 67871e6ef492fb2235af89d1
paper_authors.name: Zijia Wu
paper_title: MiniMax-01: Scaling Foundation Models with Lightning Attention
paper_summary: We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01,
  which are comparable to top-tier models while offering superior capabilities in
  processing longer contexts. The core lies in lightning attention and its
  efficient scaling. To maximize computational capacity, we integrate it with
  Mixture of Experts (MoE), creating a model with 32 experts and 456 billion
  total parameters, of which 45.9 billion are activated for each token. We
  develop an optimized parallel strategy and highly efficient
  computation-communication overlap techniques for MoE and lightning attention.
  This approach enables us to conduct efficient training and inference on models
  with hundreds of billions of parameters across contexts spanning millions of
  tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens
  during training and extrapolate to 4 million tokens during inference at an
  affordable cost. Our vision-language model, MiniMax-VL-01 is built through
  continued training with 512 billion vision-language tokens. Experiments on both
  standard and in-house benchmarks show that our models match the performance of
  state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32
  times longer context window. We publicly release MiniMax-01 at
  https://github.com/MiniMax-AI.
paper_ai_keywords: lightning attention Mixture of Experts (MoE) context window computation-communication overlap efficient training inference vision-language model dense models