title: DanceGRPO: Unleashing GRPO on Visual Generation
summary: Recent breakthroughs in generative models-particularly diffusion models and
  rectified flows-have revolutionized visual content creation, yet aligning model
  outputs with human preferences remains a critical challenge. Existing
  reinforcement learning (RL)-based methods for visual generation face critical
  limitations: incompatibility with modern Ordinary Differential Equations
  (ODEs)-based sampling paradigms, instability in large-scale training, and lack
  of validation for video generation. This paper introduces DanceGRPO, the first
  unified framework to adapt Group Relative Policy Optimization (GRPO) to visual
  generation paradigms, unleashing one unified RL algorithm across two generative
  paradigms (diffusion models and rectified flows), three tasks (text-to-image,
  text-to-video, image-to-video), four foundation models (Stable Diffusion,
  HunyuanVideo, FLUX, SkyReel-I2V), and five reward models (image/video
  aesthetics, text-image alignment, video motion quality, and binary reward). To
  our knowledge, DanceGRPO is the first RL-based unified framework capable of
  seamless adaptation across diverse generative paradigms, tasks, foundational
  models, and reward models. DanceGRPO demonstrates consistent and substantial
  improvements, which outperform baselines by up to 181% on benchmarks such as
  HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Notably, DanceGRPO not only can
  stabilize policy optimization for complex video generation, but also enables
  generative policy to better capture denoising trajectories for Best-of-N
  inference scaling and learn from sparse binary feedback. Our results establish
  DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning
  from Human Feedback (RLHF) tasks in visual generation, offering new insights
  into harmonizing reinforcement learning and visual synthesis. The code will be
  released.
paper_authors._id: 6822b53e731ca42bb4ba7d2f
paper_authors.name: Zeyue Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:33:44.632Z
paper_authors._id: 63721f5ada3183d9d53cfe1f
paper_authors.avatarUrl: /avatars/593c14c907848da7dbc9e5418751bd94.svg
paper_authors.fullname: Xue Zeyue
paper_authors.type: user
paper_authors.user: xzyhku
paper_authors._id: 6822b53e731ca42bb4ba7d30
paper_authors.name: Jie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:12:02.685Z
paper_authors._id: 6381c5d63680a7cf34e08ca9
paper_authors.avatarUrl: /avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
paper_authors.fullname: wujie10558@gmail.com
paper_authors.type: user
paper_authors.user: wujie10
paper_authors._id: 6822b53e731ca42bb4ba7d31
paper_authors.name: Yu Gao
paper_authors._id: 6822b53e731ca42bb4ba7d32
paper_authors.name: Fangyuan Kong
paper_authors._id: 6822b53e731ca42bb4ba7d33
paper_authors.name: Lingting Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:34:00.874Z
paper_authors._id: 6423f5e6774cc34079730f31
paper_authors.avatarUrl: /avatars/7ebeb1f623c86b1a676c95bb67572f8b.svg
paper_authors.fullname: Lingting Zhu
paper_authors.type: user
paper_authors.user: ltzhu
paper_authors._id: 6822b53e731ca42bb4ba7d34
paper_authors.name: Mengzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:34:06.563Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 6822b53e731ca42bb4ba7d35
paper_authors.name: Zhiheng Liu
paper_authors._id: 6822b53e731ca42bb4ba7d36
paper_authors.name: Wei Liu
paper_authors._id: 6822b53e731ca42bb4ba7d37
paper_authors.name: Qiushan Guo
paper_authors._id: 6822b53e731ca42bb4ba7d38
paper_authors.name: Weilin Huang
paper_authors._id: 6822b53e731ca42bb4ba7d39
paper_authors.name: Ping Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:35:40.333Z
paper_authors._id: 67cb7d55560c3dcbb1adeaa3
paper_authors.avatarUrl: /avatars/0b616d3655b0b54a621c2608b2f14379.svg
paper_authors.fullname: Ping Luo
paper_authors.type: user
paper_authors.user: appleluo
paper_title: DanceGRPO: Unleashing GRPO on Visual Generation
paper_summary: Recent breakthroughs in generative models-particularly diffusion models and
  rectified flows-have revolutionized visual content creation, yet aligning model
  outputs with human preferences remains a critical challenge. Existing
  reinforcement learning (RL)-based methods for visual generation face critical
  limitations: incompatibility with modern Ordinary Differential Equations
  (ODEs)-based sampling paradigms, instability in large-scale training, and lack
  of validation for video generation. This paper introduces DanceGRPO, the first
  unified framework to adapt Group Relative Policy Optimization (GRPO) to visual
  generation paradigms, unleashing one unified RL algorithm across two generative
  paradigms (diffusion models and rectified flows), three tasks (text-to-image,
  text-to-video, image-to-video), four foundation models (Stable Diffusion,
  HunyuanVideo, FLUX, SkyReel-I2V), and five reward models (image/video
  aesthetics, text-image alignment, video motion quality, and binary reward). To
  our knowledge, DanceGRPO is the first RL-based unified framework capable of
  seamless adaptation across diverse generative paradigms, tasks, foundational
  models, and reward models. DanceGRPO demonstrates consistent and substantial
  improvements, which outperform baselines by up to 181% on benchmarks such as
  HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Notably, DanceGRPO not only can
  stabilize policy optimization for complex video generation, but also enables
  generative policy to better capture denoising trajectories for Best-of-N
  inference scaling and learn from sparse binary feedback. Our results establish
  DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning
  from Human Feedback (RLHF) tasks in visual generation, offering new insights
  into harmonizing reinforcement learning and visual synthesis. The code will be
  released.
paper_ai_summary: DanceGRPO is a unified RL framework that enhances visual generation across different paradigms, tasks, models, and reward systems, outperforming baselines in benchmarks and improving video generation stability.
paper_ai_keywords: diffusion models rectified flows reinforcement learning Group Relative Policy Optimization (GRPO) text-to-image text-to-video image-to-video Stable Diffusion HunyuanVideo FLUX SkyReel-I2V image/video aesthetics text-image alignment video motion quality binary reward Best-of-N inference Reinforcement Learning from Human Feedback (RLHF)