unalignment/toxic-dpo-v0.2
Viewer
•
Updated
•
541
•
188
•
132
Nikita Kezins
entfane
AI & ML interests
LLM post-training, adversarial training, safety, knowledge transfer
Recent Activity
liked
a dataset
4 days ago
unalignment/toxic-dpo-v0.2
liked
a dataset
4 days ago
prometheus-eval/Feedback-Collection
liked
a dataset
4 days ago
prometheus-eval/Preference-Collection
Organizations
liked
3
datasets
4 days ago
french to sql model
1
#2 opened 28 days ago
by
Dieuveil
upvoted
an
article
6 days ago
Article
Training and Finetuning Reranker Models with Sentence Transformers v4
By
•
•
151
liked
2
models
7 days ago
reranker0.6b and embedding0.6b are the same model weights？
2
#6 opened about 2 months ago
by
chaochaoli
liked
a
dataset
7 days ago
Thank you very much!
reacted to
Kseniase's
post with 👍
7 days ago
Post
4857
9 new policy optimization techniques
Reinforcement Learning (RL) won't stuck in the same old PPO loop - in the last two months alone, researchers have introduced a new wave of techniques, reshaping how we train and fine-tune LLMs, VLMs, and agents.
Here are 9 fresh policy optimization techniques worth knowing:
1. GSPO: Group Sequence Policy Optimization → Group Sequence Policy Optimization (2507.18071)
Shifts from token-level to sequence-level optimization, clipping, and rewarding to capture the full picture and increase stability compared to GRPO. GSPO-token variation also allows token-level fine-tuning.
2. LAPO: Length-Adaptive Policy Optimization → LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization (2507.15758)
A two-stage RL framework that trains models to adaptively control reasoning length by learning typical solution lengths for shorter and more efficient reasoning.
3. HBPO: Hierarchical Budget Policy Optimization → Hierarchical Budget Policy Optimization for Adaptive Reasoning (2507.15844)
This one trains model to adapt reasoning depth based on problem complexity. It divides training samples into subgroups with different token budgets, using budget-aware rewards to align reasoning effort with task difficulty.
4. SOPHIA: Semi-off-policy reinforcement learning → Semi-off-Policy Reinforcement Learning for Vision-Language Slow-thinking Reasoning (2507.16814)
Combines on-policy visual understanding from the Vision Language Models (VLMs) with off-policy reasoning from an LM, assigning outcome-based rewards and propagating visual rewards backward through the reasoning steps.
5. RePO: Replay-Enhanced Policy Optimization → RePO: Replay-Enhanced Policy Optimization (2506.09340)
Introduces a replay buffer into on-policy RL for LLMs, retrieving diverse off-policy samples for each prompt to broaden the training data per prompt
Read further below ⬇️
If you like it, also subscribe to the Turing Post: https://www.turingpost.com/subscribe
Reinforcement Learning (RL) won't stuck in the same old PPO loop - in the last two months alone, researchers have introduced a new wave of techniques, reshaping how we train and fine-tune LLMs, VLMs, and agents.
Here are 9 fresh policy optimization techniques worth knowing:
1. GSPO: Group Sequence Policy Optimization → Group Sequence Policy Optimization (2507.18071)
Shifts from token-level to sequence-level optimization, clipping, and rewarding to capture the full picture and increase stability compared to GRPO. GSPO-token variation also allows token-level fine-tuning.
2. LAPO: Length-Adaptive Policy Optimization → LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization (2507.15758)
A two-stage RL framework that trains models to adaptively control reasoning length by learning typical solution lengths for shorter and more efficient reasoning.
3. HBPO: Hierarchical Budget Policy Optimization → Hierarchical Budget Policy Optimization for Adaptive Reasoning (2507.15844)
This one trains model to adapt reasoning depth based on problem complexity. It divides training samples into subgroups with different token budgets, using budget-aware rewards to align reasoning effort with task difficulty.
4. SOPHIA: Semi-off-policy reinforcement learning → Semi-off-Policy Reinforcement Learning for Vision-Language Slow-thinking Reasoning (2507.16814)
Combines on-policy visual understanding from the Vision Language Models (VLMs) with off-policy reasoning from an LM, assigning outcome-based rewards and propagating visual rewards backward through the reasoning steps.
5. RePO: Replay-Enhanced Policy Optimization → RePO: Replay-Enhanced Policy Optimization (2506.09340)
Introduces a replay buffer into on-policy RL for LLMs, retrieving diverse off-policy samples for each prompt to broaden the training data per prompt
Read further below ⬇️
If you like it, also subscribe to the Turing Post: https://www.turingpost.com/subscribe
- 1 reply
Seems to be very interesting, is there any paper published?
Benchmarks
1
#1 opened 7 days ago
by
entfane
liked
a
model
7 days ago
Release of SFT tuned model
👍
2
3
#8 opened 7 months ago
by
yakazimir
liked
a
model
9 days ago
liked
a
dataset
11 days ago
upvoted
an
article
13 days ago
Article
Illustrating Reinforcement Learning from Human Feedback (RLHF)
By
and 3 others
•
•
313
upvoted
an
article
18 days ago
Article
KV Caching Explained: Optimizing Transformer Inference Efficiency
By
•
•
107
updated
a
model
19 days ago