title: Qwen2.5-1M Technical Report
summary: We introduce Qwen2.5-1M, a series of models that extend the context length to
  1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series
  have significantly enhanced long-context capabilities through long-context
  pre-training and post-training. Key techniques such as long data synthesis,
  progressive pre-training, and multi-stage supervised fine-tuning are employed
  to effectively enhance long-context performance while reducing training costs.
    To promote the use of long-context models among a broader user base, we
  present and open-source our inference framework. This framework includes a
  length extrapolation method that can expand the model context lengths by at
  least four times, or even more, without additional training. To reduce
  inference costs, we implement a sparse attention method along with chunked
  prefill optimization for deployment scenarios and a sparsity refinement method
  to improve precision. Additionally, we detail our optimizations in the
  inference engine, including kernel optimization, pipeline parallelism, and
  scheduling optimization, which significantly enhance overall inference
  performance. By leveraging our inference framework, the Qwen2.5-1M models
  achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million
  tokens of context. This framework provides an efficient and powerful solution
  for developing applications that require long-context processing using
  open-source models.
    The Qwen2.5-1M series currently includes the open-source models
  Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed
  model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly
  improved in long-context tasks without compromising performance in
  short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model
  significantly outperforms GPT-4o-mini in long-context tasks and supports
  contexts eight times longer.
paper_authors._id: 67986c83b5e71350993d28eb
paper_authors.name: An Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T06:43:08.373Z
paper_authors._id: 62088594a5943c8a8fc94560
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
paper_authors.fullname: An Yang
paper_authors.type: user
paper_authors.user: yangapku
paper_authors._id: 67986c83b5e71350993d28ec
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:56:10.598Z
paper_authors._id: 6583ab7983a9e1460c67d876
paper_authors.avatarUrl: /avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
paper_authors.fullname: bowen
paper_authors.type: user
paper_authors.user: bowenYu
paper_authors._id: 67986c83b5e71350993d28ed
paper_authors.name: Chengyuan Li
paper_authors._id: 67986c83b5e71350993d28ee
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:56:44.491Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 67986c83b5e71350993d28ef
paper_authors.name: Fei Huang
paper_authors._id: 67986c83b5e71350993d28f0
paper_authors.name: Haoyan Huang
paper_authors._id: 67986c83b5e71350993d28f1
paper_authors.name: Jiandong Jiang
paper_authors._id: 67986c83b5e71350993d28f2
paper_authors.name: Jianhong Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:58:35.581Z
paper_authors._id: 654bead777401b47e6424f88
paper_authors.avatarUrl: /avatars/7bcbdbb051c93b004f0dc3ad36c4a0ce.svg
paper_authors.fullname: Jianhong Tu
paper_authors.type: user
paper_authors.user: ToviTu
paper_authors._id: 67986c83b5e71350993d28f3
paper_authors.name: Jianwei Zhang
paper_authors._id: 67986c83b5e71350993d28f4
paper_authors.name: Jingren Zhou
paper_authors._id: 67986c83b5e71350993d28f5
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:57:31.261Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 67986c83b5e71350993d28f6
paper_authors.name: Kai Dang
paper_authors._id: 67986c83b5e71350993d28f7
paper_authors.name: Kexin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:57:56.435Z
paper_authors._id: 65b0b3957e5d5a4ecc750de0
paper_authors.avatarUrl: /avatars/e0d79d3265ca4ad5c5411feb01043fb4.svg
paper_authors.fullname: Kexin Yang
paper_authors.type: user
paper_authors.user: dawn0929
paper_authors._id: 67986c83b5e71350993d28f8
paper_authors.name: Le Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:48:00.054Z
paper_authors._id: 6455056fe4952d1c6cb45e49
paper_authors.avatarUrl: /avatars/730f61ebc4f2df7489499ad7471a426e.svg
paper_authors.fullname: Le Yu
paper_authors.type: user
paper_authors.user: vanillaOVO
paper_authors._id: 67986c83b5e71350993d28f9
paper_authors.name: Mei Li
paper_authors._id: 67986c83b5e71350993d28fa
paper_authors.name: Minmin Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:58:27.795Z
paper_authors._id: 636a390037d9329b4a007009
paper_authors.avatarUrl: /avatars/a3c9117e104d4667e39e20ec83dc5cd6.svg
paper_authors.fullname: Minmin Sun
paper_authors.type: user
paper_authors.user: minminsun
paper_authors._id: 67986c83b5e71350993d28fb
paper_authors.name: Qin Zhu
paper_authors._id: 67986c83b5e71350993d28fc
paper_authors.name: Rui Men
paper_authors._id: 67986c83b5e71350993d28fd
paper_authors.name: Tao He
paper_authors._id: 67986c83b5e71350993d28fe
paper_authors.name: Weijia Xu
paper_authors._id: 67986c83b5e71350993d28ff
paper_authors.name: Wenbiao Yin
paper_authors._id: 67986c83b5e71350993d2900
paper_authors.name: Wenyuan Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:58:11.021Z
paper_authors._id: 63f4c99721eb234ab73dd112
paper_authors.avatarUrl: /avatars/162e92d7aeb7de1c6ebf4d6e2bff33f5.svg
paper_authors.fullname: yu wenyuan
paper_authors.type: user
paper_authors.user: liuxinyijian
paper_authors._id: 67986c83b5e71350993d2901
paper_authors.name: Xiafei Qiu
paper_authors._id: 67986c83b5e71350993d2902
paper_authors.name: Xingzhang Ren
paper_authors._id: 67986c83b5e71350993d2903
paper_authors.name: Xinlong Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T14:48:49.659Z
paper_authors._id: 6493eb93f453c3a7a2828362
paper_authors.avatarUrl: /avatars/0a0e1f010465ab2042e532fc1f5b8053.svg
paper_authors.fullname: Yang Xinlong
paper_authors.type: user
paper_authors.user: Yangyy666
paper_authors._id: 67986c83b5e71350993d2904
paper_authors.name: Yong Li
paper_authors._id: 67986c83b5e71350993d2905
paper_authors.name: Zhiying Xu
paper_authors._id: 67986c83b5e71350993d2906
paper_authors.name: Zipeng Zhang
paper_title: Qwen2.5-1M Technical Report
paper_summary: We introduce Qwen2.5-1M, a series of models that extend the context length to
  1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series
  have significantly enhanced long-context capabilities through long-context
  pre-training and post-training. Key techniques such as long data synthesis,
  progressive pre-training, and multi-stage supervised fine-tuning are employed
  to effectively enhance long-context performance while reducing training costs.
    To promote the use of long-context models among a broader user base, we
  present and open-source our inference framework. This framework includes a
  length extrapolation method that can expand the model context lengths by at
  least four times, or even more, without additional training. To reduce
  inference costs, we implement a sparse attention method along with chunked
  prefill optimization for deployment scenarios and a sparsity refinement method
  to improve precision. Additionally, we detail our optimizations in the
  inference engine, including kernel optimization, pipeline parallelism, and
  scheduling optimization, which significantly enhance overall inference
  performance. By leveraging our inference framework, the Qwen2.5-1M models
  achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million
  tokens of context. This framework provides an efficient and powerful solution
  for developing applications that require long-context processing using
  open-source models.
    The Qwen2.5-1M series currently includes the open-source models
  Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed
  model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly
  improved in long-context tasks without compromising performance in
  short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model
  significantly outperforms GPT-4o-mini in long-context tasks and supports
  contexts eight times longer.
paper_ai_summary: The Qwen2.5-1M series models extend context length to 1 million tokens with enhanced long-context capabilities, employing techniques like long data synthesis and progressive pre-training, and are supported by an open-source inference framework with sparse attention and kernel optimizations.
paper_ai_keywords: long-context pre-training long data synthesis progressive pre-training multi-stage supervised fine-tuning length extrapolation sparse attention chunked prefill optimization sparsity refinement kernel optimization pipeline parallelism scheduling optimization