title: Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's
    Own Signals
summary: Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models
  (LLMs) suffer from sample inefficiency due to the redundant exposure of
  identical queries under uniform data sampling. While previous work has explored
  curriculum learning via heuristic difficulty metrics, these strategies exhibit
  limitations by neglecting the intrinsic learning signals generated by the model
  itself, thus leading to suboptimal training regimes. In this paper, we identify
  a model-inherent signal termed angle concentration that effectively reflects an
  LLM's capacity to learn from specific data. We theoretically and empirically
  demonstrate a correlation between the angular distribution of token hidden
  state vectors and the resulting gradient, revealing a learning preference for
  data exhibiting higher angle concentration. Inspired by this finding, we
  propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By
  leveraging the model's intrinsic angle concentration signal, GAIN-RL
  dynamically selects training data in each epoch, ensuring consistently
  impactful gradient updates and thus significantly enhancing overall training
  efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5x
  acceleration in training efficiency across diverse mathematical and coding
  tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient
  sampling yields data-efficient training, achieving better performance with half
  the original data compared to vanilla GRPO with full training data. Code is
  realsed at https://github.com/wangqinsi1/GAINRL/tree/main.
paper_authors._id: 6840c71b2361fd58c84588b1
paper_authors.name: Qinsi Wang
paper_authors._id: 6840c71b2361fd58c84588b2
paper_authors.name: Jinghan Ke
paper_authors._id: 6840c71b2361fd58c84588b3
paper_authors.name: Hancheng Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:33:36.043Z
paper_authors._id: 6597f70d260bcd5fb5529613
paper_authors.avatarUrl: /avatars/8fd7a210f3cb0de1229b49a3cd1ad8a0.svg
paper_authors.fullname: Hancheng Ye
paper_authors.type: user
paper_authors.user: HankYe
paper_authors._id: 6840c71b2361fd58c84588b4
paper_authors.name: Yueqian Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:33:42.287Z
paper_authors._id: 64b5198c25882acb62fb77ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b5198c25882acb62fb77ef/HX9pfMEPQlfjvSAgSLplY.png
paper_authors.fullname: Yueqian Lin
paper_authors.type: user
paper_authors.user: linyueqian
paper_authors._id: 6840c71b2361fd58c84588b5
paper_authors.name: Yuzhe Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:33:49.125Z
paper_authors._id: 66d9052a60730595a867e58f
paper_authors.avatarUrl: /avatars/3a84156253153150a573d374b731a286.svg
paper_authors.fullname: Yuzhe Fu
paper_authors.type: user
paper_authors.user: YuzheFu
paper_authors._id: 6840c71b2361fd58c84588b6
paper_authors.name: Jianyi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:33:56.480Z
paper_authors._id: 676d402ad886f8125af92360
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Pby6l-_ymnvPZSAAWWeS_.png
paper_authors.fullname: jianyi.zhang
paper_authors.type: user
paper_authors.user: jianyi0115
paper_authors._id: 6840c71b2361fd58c84588b7
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:34:01.864Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 6840c71b2361fd58c84588b8
paper_authors.name: Chenfeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:34:10.768Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_authors._id: 6840c71b2361fd58c84588b9
paper_authors.name: Yiran Chen
paper_title: Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's
    Own Signals
paper_summary: Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models
  (LLMs) suffer from sample inefficiency due to the redundant exposure of
  identical queries under uniform data sampling. While previous work has explored
  curriculum learning via heuristic difficulty metrics, these strategies exhibit
  limitations by neglecting the intrinsic learning signals generated by the model
  itself, thus leading to suboptimal training regimes. In this paper, we identify
  a model-inherent signal termed angle concentration that effectively reflects an
  LLM's capacity to learn from specific data. We theoretically and empirically
  demonstrate a correlation between the angular distribution of token hidden
  state vectors and the resulting gradient, revealing a learning preference for
  data exhibiting higher angle concentration. Inspired by this finding, we
  propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By
  leveraging the model's intrinsic angle concentration signal, GAIN-RL
  dynamically selects training data in each epoch, ensuring consistently
  impactful gradient updates and thus significantly enhancing overall training
  efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5x
  acceleration in training efficiency across diverse mathematical and coding
  tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient
  sampling yields data-efficient training, achieving better performance with half
  the original data compared to vanilla GRPO with full training data. Code is
  realsed at https://github.com/wangqinsi1/GAINRL/tree/main.
paper_ai_summary: GAIN-RL leverages angle concentration signals to improve training efficiency and data efficiency in Reinforcement Fine-tuning of Large Language Models.
paper_ai_keywords: Reinforcement Fine-tuning Large Language Models sample inefficiency curriculum learning angle concentration gradient updates token hidden state vectors GAIN-RL Gradient-driven Angle-Informed Navigated RL