title: Qwen2.5 Technical Report
summary: In this report, we introduce Qwen2.5, a comprehensive series of large
  language models (LLMs) designed to meet diverse needs. Compared to previous
  iterations, Qwen 2.5 has been significantly improved during both the
  pre-training and post-training stages. In terms of pre-training, we have scaled
  the high-quality pre-training datasets from the previous 7 trillion tokens to
  18 trillion tokens. This provides a strong foundation for common sense, expert
  knowledge, and reasoning capabilities. In terms of post-training, we implement
  intricate supervised finetuning with over 1 million samples, as well as
  multistage reinforcement learning. Post-training techniques enhance human
  preference, and notably improve long text generation, structural data analysis,
  and instruction following. To handle diverse and varied use cases effectively,
  we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base
  and instruction-tuned models, with quantized versions available. In addition,
  for hosted solutions, the proprietary models currently include two
  mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both
  available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier
  performance on a wide range of benchmarks evaluating language understanding,
  reasoning, mathematics, coding, human preference alignment, etc. Specifically,
  the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and
  proprietary models and demonstrates competitive performance to the
  state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5
  times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness
  while performing competitively against GPT-4o-mini and GPT-4o respectively.
  Additionally, as the foundation, Qwen2.5 models have been instrumental in
  training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and
  multimodal models.
paper_authors._id: 6764dda159e2ef96ec013c07
paper_authors.name: Qwen
paper_authors._id: 6764dda159e2ef96ec013c09
paper_authors.name: An Yang
paper_authors._id: 6764dda159e2ef96ec013c0a
paper_authors.name: Baosong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:42:54.655Z
paper_authors._id: 64b0a77df12b47366663884c
paper_authors.avatarUrl: /avatars/a212ea862abb5966060e439dd0e7656f.svg
paper_authors.fullname: Baosong Yang
paper_authors.type: user
paper_authors.user: Baosong
paper_authors._id: 6764dda159e2ef96ec013c0b
paper_authors.name: Beichen Zhang
paper_authors._id: 6764dda159e2ef96ec013c0c
paper_authors.name: Binyuan Hui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:32:55.260Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 6764dda159e2ef96ec013c0d
paper_authors.name: Bo Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:15:45.070Z
paper_authors._id: 62c695ad5aae1c624ca992e2
paper_authors.avatarUrl: /avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
paper_authors.fullname: Bo Zheng
paper_authors.type: user
paper_authors.user: bzheng
paper_authors._id: 6764dda159e2ef96ec013c0e
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:43:56.820Z
paper_authors._id: 6583ab7983a9e1460c67d876
paper_authors.avatarUrl: /avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
paper_authors.fullname: bowen
paper_authors.type: user
paper_authors.user: bowenYu
paper_authors._id: 6764dda159e2ef96ec013c0f
paper_authors.name: Chengyuan Li
paper_authors._id: 6764dda159e2ef96ec013c10
paper_authors.name: Dayiheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:32:51.398Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 6764dda159e2ef96ec013c11
paper_authors.name: Fei Huang
paper_authors._id: 6764dda159e2ef96ec013c12
paper_authors.name: Haoran Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:44:49.831Z
paper_authors._id: 6436618aeef1f55654a9f458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
paper_authors.fullname: Haoran Wei
paper_authors.type: user
paper_authors.user: HaoranWei
paper_authors._id: 6764dda159e2ef96ec013c13
paper_authors.name: Huan Lin
paper_authors._id: 6764dda159e2ef96ec013c14
paper_authors.name: Jian Yang
paper_authors._id: 6764dda159e2ef96ec013c15
paper_authors.name: Jianhong Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:45:05.864Z
paper_authors._id: 654bead777401b47e6424f88
paper_authors.avatarUrl: /avatars/7bcbdbb051c93b004f0dc3ad36c4a0ce.svg
paper_authors.fullname: Jianhong Tu
paper_authors.type: user
paper_authors.user: ToviTu
paper_authors._id: 6764dda159e2ef96ec013c16
paper_authors.name: Jianwei Zhang
paper_authors._id: 6764dda159e2ef96ec013c17
paper_authors.name: Jianxin Yang
paper_authors._id: 6764dda159e2ef96ec013c18
paper_authors.name: Jiaxi Yang
paper_authors._id: 6764dda159e2ef96ec013c19
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:45:52.281Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_authors._id: 6764dda159e2ef96ec013c1a
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:45:59.082Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 6764dda159e2ef96ec013c1b
paper_authors.name: Kai Dang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T15:42:29.190Z
paper_authors._id: 6712930f0fac3235c56edf5b
paper_authors.avatarUrl: /avatars/cafe7cb56ce7c3b2572f5f2d0b89357a.svg
paper_authors.fullname: kai dang
paper_authors.type: user
paper_authors.user: 1vk5i
paper_authors._id: 6764dda159e2ef96ec013c1c
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:46:07.173Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 6764dda159e2ef96ec013c1d
paper_authors.name: Keqin Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:42:00.963Z
paper_authors._id: 6154629f6181394cc00cb98b
paper_authors.avatarUrl: /avatars/de2d5cb6fee9c03a99289d03272eddeb.svg
paper_authors.fullname: Keqin Bao
paper_authors.type: user
paper_authors.user: morningStar
paper_authors._id: 6764dda159e2ef96ec013c1e
paper_authors.name: Kexin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:46:29.866Z
paper_authors._id: 65b0b3957e5d5a4ecc750de0
paper_authors.avatarUrl: /avatars/e0d79d3265ca4ad5c5411feb01043fb4.svg
paper_authors.fullname: Kexin Yang
paper_authors.type: user
paper_authors.user: dawn0929
paper_authors._id: 6764dda159e2ef96ec013c1f
paper_authors.name: Le Yu
paper_authors._id: 6764dda159e2ef96ec013c20
paper_authors.name: Mei Li
paper_authors._id: 6764dda159e2ef96ec013c21
paper_authors.name: Mingfeng Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:46:45.959Z
paper_authors._id: 5f8946925d083370c711f296
paper_authors.avatarUrl: /avatars/14246aae3b1f8b7ad050f8ff2c8b260e.svg
paper_authors.fullname: Mingfeng Xue
paper_authors.type: user
paper_authors.user: mingfengxue
paper_authors._id: 6764dda159e2ef96ec013c22
paper_authors.name: Pei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:47:05.519Z
paper_authors._id: 634872721df68daaa0125e08
paper_authors.avatarUrl: /avatars/c091b70448a4e1d43fce3a31a7beddf3.svg
paper_authors.fullname: Pei Zhang
paper_authors.type: user
paper_authors.user: peizhang
paper_authors._id: 6764dda159e2ef96ec013c23
paper_authors.name: Qin Zhu
paper_authors._id: 6764dda159e2ef96ec013c24
paper_authors.name: Rui Men
paper_authors._id: 6764dda159e2ef96ec013c25
paper_authors.name: Runji Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:32:53.455Z
paper_authors._id: 649a52e5de0fb7f3f499e583
paper_authors.avatarUrl: /avatars/25f6106fa168ae57ad5cd8ef55c70d31.svg
paper_authors.fullname: Runji Lin
paper_authors.type: user
paper_authors.user: RunjiLin
paper_authors._id: 6764dda159e2ef96ec013c26
paper_authors.name: Tianhao Li
paper_authors._id: 6764dda159e2ef96ec013c27
paper_authors.name: Tingyu Xia
paper_authors._id: 6764dda159e2ef96ec013c28
paper_authors.name: Xingzhang Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:49.873Z
paper_authors._id: 645c82c4081be4b32049633a
paper_authors.avatarUrl: /avatars/e5a08cf0ec5a04bd9d66111382ce0508.svg
paper_authors.fullname: xzhren
paper_authors.type: user
paper_authors.user: xingzhang
paper_authors._id: 6764dda159e2ef96ec013c29
paper_authors.name: Xuancheng Ren
paper_authors._id: 6764dda159e2ef96ec013c2a
paper_authors.name: Yang Fan
paper_authors._id: 6764dda159e2ef96ec013c2b
paper_authors.name: Yang Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T13:55:55.786Z
paper_authors._id: 63481fea27bd5edd982328f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665670948064-noauth.jpeg
paper_authors.fullname: Yang Su
paper_authors.type: user
paper_authors.user: yang-su2000
paper_authors._id: 6764dda159e2ef96ec013c2c
paper_authors.name: Yichang Zhang
paper_authors._id: 6764dda159e2ef96ec013c2d
paper_authors.name: Yu Wan
paper_authors._id: 6764dda159e2ef96ec013c2e
paper_authors.name: Yuqiong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:14:40.709Z
paper_authors._id: 666aacfb918ba11c7c598194
paper_authors.avatarUrl: /avatars/45bee8f1fdbdd256ee47d25e4bf01a7a.svg
paper_authors.fullname: Yuqiong Liu
paper_authors.type: user
paper_authors.user: lyq333
paper_authors._id: 6764dda159e2ef96ec013c2f
paper_authors.name: Zeyu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:14:34.133Z
paper_authors._id: 672c25ca8cfb61188128eb6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
paper_authors.fullname: Zeyu Cui
paper_authors.type: user
paper_authors.user: misakamage
paper_authors._id: 6764dda159e2ef96ec013c30
paper_authors.name: Zhenru Zhang
paper_authors._id: 6764dda159e2ef96ec013c31
paper_authors.name: Zihan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T08:41:53.155Z
paper_authors._id: 647ccbd6e07cf9bb2d485244
paper_authors.avatarUrl: /avatars/e8915abaff04f6762247e196b7cf84df.svg
paper_authors.fullname: Zihan Qiu
paper_authors.type: user
paper_authors.user: QwQZh
paper_title: Qwen2.5 Technical Report
paper_summary: In this report, we introduce Qwen2.5, a comprehensive series of large
  language models (LLMs) designed to meet diverse needs. Compared to previous
  iterations, Qwen 2.5 has been significantly improved during both the
  pre-training and post-training stages. In terms of pre-training, we have scaled
  the high-quality pre-training datasets from the previous 7 trillion tokens to
  18 trillion tokens. This provides a strong foundation for common sense, expert
  knowledge, and reasoning capabilities. In terms of post-training, we implement
  intricate supervised finetuning with over 1 million samples, as well as
  multistage reinforcement learning. Post-training techniques enhance human
  preference, and notably improve long text generation, structural data analysis,
  and instruction following. To handle diverse and varied use cases effectively,
  we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base
  and instruction-tuned models, with quantized versions available. In addition,
  for hosted solutions, the proprietary models currently include two
  mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both
  available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier
  performance on a wide range of benchmarks evaluating language understanding,
  reasoning, mathematics, coding, human preference alignment, etc. Specifically,
  the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and
  proprietary models and demonstrates competitive performance to the
  state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5
  times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness
  while performing competitively against GPT-4o-mini and GPT-4o respectively.
  Additionally, as the foundation, Qwen2.5 models have been instrumental in
  training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and
  multimodal models.
paper_ai_keywords: large language models (LLMs) pre-training high-quality pre-training datasets supervised fine-tuning reinforcement learning mixture-of-experts (MoE) open-weight offerings instruction-tuned models quantized versions language understanding reasoning capabilities coding human preference alignment benchmarks Llama-3-405B-Instruct GPT-4o-mini GPT-4o specialized models multimodal models average word perplexity (AWP)