title: Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and
    Beyond
summary: This paper presents our work on the Light-R1 series, with models, data, and
  code all released.
    We first focus on training long COT models from scratch, specifically
  starting from models initially lacking long COT capabilities. Using a
  curriculum training recipe consisting of two-stage SFT and semi-on-policy DPO,
  we train our model Light-R1-32B from Qwen2.5-32B-Instruct, resulting in
  superior math performance compared to DeepSeek-R1-Distill-Qwen-32B. Despite
  being trained exclusively on math data, Light-R1-32B shows strong
  generalization across other domains. In the subsequent phase of this work, we
  highlight the significant benefit of the 3k dataset constructed for the second
  SFT stage on enhancing other models. By fine-tuning DeepSeek-R1-Distilled
  models using this dataset, we obtain new SOTA models in 7B and 14B, while the
  32B model, Light-R1-32B-DS performed comparably to QwQ-32B and DeepSeek-R1.
    Furthermore, we extend our work by applying reinforcement learning,
  specifically GRPO, on long-COT models to further improve reasoning performance.
  We successfully train our final Light-R1-14B-DS with RL, achieving SOTA
  performance among 14B parameter models in math. With AIME24 & 25 scores of 74.0
  and 60.2 respectively, Light-R1-14B-DS surpasses even many 32B models and
  DeepSeek-R1-Distill-Llama-70B. Its RL training also exhibits well expected
  behavior, showing simultaneous increase in response length and reward score.
    The Light-R1 series of work validates training long-COT models from scratch,
  showcases the art in SFT data and releases SOTA models from RL.
paper_authors._id: 67d3a87df6ea55297c3cd071
paper_authors.name: Liang Wen
paper_authors._id: 67d3a87df6ea55297c3cd072
paper_authors.name: Yunke Cai
paper_authors._id: 67d3a87df6ea55297c3cd073
paper_authors.name: Fenrui Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:07:18.770Z
paper_authors._id: 643cbf3aae8d93dc395383e3
paper_authors.avatarUrl: /avatars/3e719374cc14483e9cdfb174ab99ad26.svg
paper_authors.fullname: Fenrui Xiao
paper_authors.type: user
paper_authors.user: cizhenshi
paper_authors._id: 67d3a87df6ea55297c3cd074
paper_authors.name: Xin He
paper_authors._id: 67d3a87df6ea55297c3cd075
paper_authors.name: Qi An
paper_authors._id: 67d3a87df6ea55297c3cd076
paper_authors.name: Zhenyu Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:07:33.157Z
paper_authors._id: 646c2f4d628e5b50b2e22511
paper_authors.avatarUrl: /avatars/da64cfd3320f00a1718ab40165ed35dd.svg
paper_authors.fullname: zhenyu duan
paper_authors.type: user
paper_authors.user: duan901010
paper_authors._id: 67d3a87df6ea55297c3cd077
paper_authors.name: Yimin Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:07:12.002Z
paper_authors._id: 66fe5e256990f417165835e8
paper_authors.avatarUrl: /avatars/8d3d0a2922f2e17610cb36e84af73584.svg
paper_authors.fullname: Yimin DU
paper_authors.type: user
paper_authors.user: yyy99
paper_authors._id: 67d3a87df6ea55297c3cd078
paper_authors.name: Junchen Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:07:42.920Z
paper_authors._id: 663a9ae58cf658ffaf3d02e5
paper_authors.avatarUrl: /avatars/aa35668dc088e675e794b9ceb935c56f.svg
paper_authors.fullname: Junchen Liu
paper_authors.type: user
paper_authors.user: JunchenLiu
paper_authors._id: 67d3a87df6ea55297c3cd079
paper_authors.name: Lifu Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:07:54.728Z
paper_authors._id: 641d1c7045487810d137866c
paper_authors.avatarUrl: /avatars/dc1140875c23213db4991714bdb1a3a7.svg
paper_authors.fullname: tlf
paper_authors.type: user
paper_authors.user: tanglifu
paper_authors._id: 67d3a87df6ea55297c3cd07a
paper_authors.name: Xiaowei Lv
paper_authors._id: 67d3a87df6ea55297c3cd07b
paper_authors.name: Haosheng Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:06:40.468Z
paper_authors._id: 65685c676c1ae87d4ee8dcfb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65685c676c1ae87d4ee8dcfb/YjUejWm43ODaiA4gAxvp1.jpeg
paper_authors.fullname: Haosheng Zou
paper_authors.type: user
paper_authors.user: zhs12
paper_authors._id: 67d3a87df6ea55297c3cd07c
paper_authors.name: Yongchao Deng
paper_authors._id: 67d3a87df6ea55297c3cd07d
paper_authors.name: Shousheng Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:06:25.241Z
paper_authors._id: 660a339940e346fba5b66f1d
paper_authors.avatarUrl: /avatars/ae8e29f0ab259257af3e888553ef5946.svg
paper_authors.fullname: jiashousheng
paper_authors.type: user
paper_authors.user: shousheng
paper_authors._id: 67d3a87df6ea55297c3cd07e
paper_authors.name: Xiangzheng Zhang
paper_title: Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and
    Beyond
paper_summary: This paper presents our work on the Light-R1 series, with models, data, and
  code all released.
    We first focus on training long COT models from scratch, specifically
  starting from models initially lacking long COT capabilities. Using a
  curriculum training recipe consisting of two-stage SFT and semi-on-policy DPO,
  we train our model Light-R1-32B from Qwen2.5-32B-Instruct, resulting in
  superior math performance compared to DeepSeek-R1-Distill-Qwen-32B. Despite
  being trained exclusively on math data, Light-R1-32B shows strong
  generalization across other domains. In the subsequent phase of this work, we
  highlight the significant benefit of the 3k dataset constructed for the second
  SFT stage on enhancing other models. By fine-tuning DeepSeek-R1-Distilled
  models using this dataset, we obtain new SOTA models in 7B and 14B, while the
  32B model, Light-R1-32B-DS performed comparably to QwQ-32B and DeepSeek-R1.
    Furthermore, we extend our work by applying reinforcement learning,
  specifically GRPO, on long-COT models to further improve reasoning performance.
  We successfully train our final Light-R1-14B-DS with RL, achieving SOTA
  performance among 14B parameter models in math. With AIME24 & 25 scores of 74.0
  and 60.2 respectively, Light-R1-14B-DS surpasses even many 32B models and
  DeepSeek-R1-Distill-Llama-70B. Its RL training also exhibits well expected
  behavior, showing simultaneous increase in response length and reward score.
    The Light-R1 series of work validates training long-COT models from scratch,
  showcases the art in SFT data and releases SOTA models from RL.
paper_ai_keywords: COT models curriculum training two-stage SFT semi-on-policy DPO long COT capabilities SOTA generalization 3k dataset fine-tuning GRPO reasoning performance AIME24 & 25 scores response length reward score