Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning
  for LLMs
We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model
optimized via reinforcement learning (RL) to achieve efficient and robust
reasoning capabilities. Built upon the publicly available Ling-lite model, a
16.8 billion parameter model with 2.75 billion activated parameters, our
approach matches the performance of state-of-the-art (SOTA) small-scale
reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench,
GPQA-Diamond) while activating only one-third of the parameters required by
comparable models. To accomplish this, we introduce a joint training pipeline
integrating distillation with RL, revealing undocumented challenges in MoE RL
training. First, we identify optimization instability during RL training, and
we propose Constrained Contextual Computation Policy Optimization(C3PO), a
novel approach that enhances training stability and improves computational
throughput via algorithm-system co-design methodology. Second, we empirically
demonstrate that selecting distillation checkpoints based on entropy loss for
RL training, rather than validation metrics, yields superior
performance-efficiency trade-offs in subsequent RL training. Finally, we
develop a two-stage training paradigm to harmonize multi-domain data
integration, addressing domain conflicts that arise in training with mixed
dataset. We will release the model, dataset, and code.
685236ac0164cd1316710512
Ring Team
685236ac0164cd1316710513
Bin Hu
685236ac0164cd1316710514
Cai Chen
685236ac0164cd1316710515
Deng Zhao
685236ac0164cd1316710516
Ding Liu
685236ac0164cd1316710517
Dingnan Jin
685236ac0164cd1316710518
Feng Zhu
685236ac0164cd1316710519
Hao Dai
685236ac0164cd131671051a
Hongzhi Luan
685236ac0164cd131671051b
Jia Guo
685236ac0164cd131671051c
Jiaming Liu
685236ac0164cd131671051d
Jiewei Wu
685236ac0164cd131671051e
Jun Mei
685236ac0164cd131671051f
Jun Zhou
685236ac0164cd1316710520
Junbo Zhao
685236ac0164cd1316710521
Junwu Xiong
claimed_verified
2025-06-20T12:19:55.087Z
63ec94fe54bba67815158b5b
/avatars/66c7bbc6224bda796d8e8c22fbb46a17.svg
Xiong Jun Wu
user
junwux
685236ac0164cd1316710522
Kaihong Zhang
685236ac0164cd1316710523
Kuan Xu
685236ac0164cd1316710524
Lei Liang
685236ac0164cd1316710525
Liang Jiang
685236ac0164cd1316710526
Liangcheng Fu
685236ac0164cd1316710527
Longfei Zheng
685236ac0164cd1316710528
Qiang Gao
685236ac0164cd1316710529
Qing Cui
685236ac0164cd131671052a
Quan Wan
685236ac0164cd131671052b
Shaomian Zheng
685236ac0164cd131671052c
Shuaicheng Li
685236ac0164cd131671052d
Tongkai Yang
685236ac0164cd131671052e
Wang Ren
685236ac0164cd131671052f
Xiaodong Yan
685236ac0164cd1316710530
Xiaopei Wan
685236ac0164cd1316710531
Xiaoyun Feng
685236ac0164cd1316710532
Xin Zhao
685236ac0164cd1316710533
Xinxing Yang
685236ac0164cd1316710534
Xinyu Kong
685236ac0164cd1316710535
Xuemin Yang
685236ac0164cd1316710536
Yang Li
685236ac0164cd1316710537
Yingting Wu
685236ac0164cd1316710538
Yongkang Liu
685236ac0164cd1316710539
Zhankai Xu
685236ac0164cd131671053a
Zhenduo Zhang
685236ac0164cd131671053b
Zhenglei Zhou
685236ac0164cd131671053c
Zhenyu Huang
685236ac0164cd131671053d
Zhiqiang Zhang
685236ac0164cd131671053e
Zihao Wang
685236ac0164cd131671053f
Zujie Wen
claimed_verified
2025-06-24T08:10:13.798Z
6859242154dec71b4c7cf0c2
/avatars/c14398864ba3d9c3e3ad5e6c80237fa2.svg
zujie wen
user
ZjWen
Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning
  for LLMs
We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model
optimized via reinforcement learning (RL) to achieve efficient and robust
reasoning capabilities. Built upon the publicly available Ling-lite model, a
16.8 billion parameter model with 2.75 billion activated parameters, our
approach matches the performance of state-of-the-art (SOTA) small-scale
reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench,
GPQA-Diamond) while activating only one-third of the parameters required by
comparable models. To accomplish this, we introduce a joint training pipeline
integrating distillation with RL, revealing undocumented challenges in MoE RL
training. First, we identify optimization instability during RL training, and
we propose Constrained Contextual Computation Policy Optimization(C3PO), a
novel approach that enhances training stability and improves computational
throughput via algorithm-system co-design methodology. Second, we empirically
demonstrate that selecting distillation checkpoints based on entropy loss for
RL training, rather than validation metrics, yields superior
performance-efficiency trade-offs in subsequent RL training. Finally, we
develop a two-stage training paradigm to harmonize multi-domain data
integration, addressing domain conflicts that arise in training with mixed
dataset. We will release the model, dataset, and code.
Ring-lite uses a MoE architecture and reinforcement learning to efficiently match SOTA reasoning models while activating fewer parameters and addressing challenges specific to MoE training.
Mixture-of-Experts (MoE) reinforcement learning (RL) Ling-lite AIME LiveCodeBench GPQA-Diamond Constrained Contextual Computation Policy Optimization(C3PO) entropy loss two-stage training paradigm