DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware
  Regressive GRPO
Recent works have demonstrated the effectiveness of reinforcement learning
(RL)-based post-training in enhancing the reasoning capabilities of large
language models (LLMs). In particular, Group Relative Policy Optimization
(GRPO) has shown impressive success by employing a PPO-style reinforcement
algorithm with group-based normalized rewards. However, the application of GRPO
to Video Large Language Models (Video LLMs) has been less studied. In this
paper, we explore GRPO for video LLMs and identify two primary issues that
impede its effective learning: (1) reliance on safeguards, and (2) the
vanishing advantage problem. To mitigate these challenges, we propose
DeepVideo-R1, a video large language model trained with our proposed Reg-GRPO
(Regressive GRPO) and difficulty-aware data augmentation strategy. Reg-GRPO
reformulates the GRPO objective as a regression task, directly predicting the
advantage in GRPO. This design eliminates the need for safeguards like clipping
and min functions, thereby facilitating more direct policy guidance by aligning
the model with the advantage values. We also design the difficulty-aware data
augmentation strategy that dynamically augments training samples at solvable
difficulty levels, fostering diverse and informative reward signals. Our
comprehensive experiments show that DeepVideo-R1 significantly improves video
reasoning performance across multiple video reasoning benchmarks.
684fd9a160b4a34dbe007b93
Jinyoung Park
claimed_verified
2025-06-18T12:17:05.214Z
64b6eae88ba7d6c922c0434a
/avatars/6adac8242106ab12abeaa3584346c0cd.svg
Jinyoung Park
user
jinypark
684fd9a160b4a34dbe007b94
Jeehye Na
684fd9a160b4a34dbe007b95
Jinyoung Kim
684fd9a160b4a34dbe007b96
Hyunwoo J. Kim
claimed_verified
2025-06-16T15:50:47.946Z
63183a30a760ff8cff318be5
/avatars/1b314afd8d3319423915ffc2ae56d567.svg
Hyunwoo Kim
user
hkimcvml
DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware
  Regressive GRPO
Recent works have demonstrated the effectiveness of reinforcement learning
(RL)-based post-training in enhancing the reasoning capabilities of large
language models (LLMs). In particular, Group Relative Policy Optimization
(GRPO) has shown impressive success by employing a PPO-style reinforcement
algorithm with group-based normalized rewards. However, the application of GRPO
to Video Large Language Models (Video LLMs) has been less studied. In this
paper, we explore GRPO for video LLMs and identify two primary issues that
impede its effective learning: (1) reliance on safeguards, and (2) the
vanishing advantage problem. To mitigate these challenges, we propose
DeepVideo-R1, a video large language model trained with our proposed Reg-GRPO
(Regressive GRPO) and difficulty-aware data augmentation strategy. Reg-GRPO
reformulates the GRPO objective as a regression task, directly predicting the
advantage in GRPO. This design eliminates the need for safeguards like clipping
and min functions, thereby facilitating more direct policy guidance by aligning
the model with the advantage values. We also design the difficulty-aware data
augmentation strategy that dynamically augments training samples at solvable
difficulty levels, fostering diverse and informative reward signals. Our
comprehensive experiments show that DeepVideo-R1 significantly improves video
reasoning performance across multiple video reasoning benchmarks.
DeepVideo-R1 enhances video reasoning performance using Reg-GRPO, a regression-based GRPO approach, and difficulty-aware data augmentation for video large language models.
reinforcement learning Group Relative Policy Optimization GRPO Policy Optimization PPO Video Large Language Models Video LLMs DeepVideo-R1 Reg-GRPO regression task advantage values difficulty-aware data augmentation video reasoning benchmarks