DanceGRPO: Unleashing GRPO on Visual Generation
Recent breakthroughs in generative models-particularly diffusion models and
rectified flows-have revolutionized visual content creation, yet aligning model
outputs with human preferences remains a critical challenge. Existing
reinforcement learning (RL)-based methods for visual generation face critical
limitations: incompatibility with modern Ordinary Differential Equations
(ODEs)-based sampling paradigms, instability in large-scale training, and lack
of validation for video generation. This paper introduces DanceGRPO, the first
unified framework to adapt Group Relative Policy Optimization (GRPO) to visual
generation paradigms, unleashing one unified RL algorithm across two generative
paradigms (diffusion models and rectified flows), three tasks (text-to-image,
text-to-video, image-to-video), four foundation models (Stable Diffusion,
HunyuanVideo, FLUX, SkyReel-I2V), and five reward models (image/video
aesthetics, text-image alignment, video motion quality, and binary reward). To
our knowledge, DanceGRPO is the first RL-based unified framework capable of
seamless adaptation across diverse generative paradigms, tasks, foundational
models, and reward models. DanceGRPO demonstrates consistent and substantial
improvements, which outperform baselines by up to 181% on benchmarks such as
HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Notably, DanceGRPO not only can
stabilize policy optimization for complex video generation, but also enables
generative policy to better capture denoising trajectories for Best-of-N
inference scaling and learn from sparse binary feedback. Our results establish
DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning
from Human Feedback (RLHF) tasks in visual generation, offering new insights
into harmonizing reinforcement learning and visual synthesis. The code will be
released.
6822b53e731ca42bb4ba7d2f
Zeyue Xue
admin_assigned
2025-05-13T08:33:44.632Z
63721f5ada3183d9d53cfe1f
/avatars/593c14c907848da7dbc9e5418751bd94.svg
Xue Zeyue
user
xzyhku
6822b53e731ca42bb4ba7d30
Jie Wu
claimed_verified
2025-05-13T08:12:02.685Z
6381c5d63680a7cf34e08ca9
/avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
wujie10558@gmail.com
user
wujie10
6822b53e731ca42bb4ba7d31
Yu Gao
6822b53e731ca42bb4ba7d32
Fangyuan Kong
6822b53e731ca42bb4ba7d33
Lingting Zhu
admin_assigned
2025-05-13T08:34:00.874Z
6423f5e6774cc34079730f31
/avatars/7ebeb1f623c86b1a676c95bb67572f8b.svg
Lingting Zhu
user
ltzhu
6822b53e731ca42bb4ba7d34
Mengzhao Chen
admin_assigned
2025-05-13T08:34:06.563Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
6822b53e731ca42bb4ba7d35
Zhiheng Liu
6822b53e731ca42bb4ba7d36
Wei Liu
6822b53e731ca42bb4ba7d37
Qiushan Guo
6822b53e731ca42bb4ba7d38
Weilin Huang
6822b53e731ca42bb4ba7d39
Ping Luo
admin_assigned
2025-05-13T08:35:40.333Z
67cb7d55560c3dcbb1adeaa3
/avatars/0b616d3655b0b54a621c2608b2f14379.svg
Ping Luo
user
appleluo
DanceGRPO: Unleashing GRPO on Visual Generation
Recent breakthroughs in generative models-particularly diffusion models and
rectified flows-have revolutionized visual content creation, yet aligning model
outputs with human preferences remains a critical challenge. Existing
reinforcement learning (RL)-based methods for visual generation face critical
limitations: incompatibility with modern Ordinary Differential Equations
(ODEs)-based sampling paradigms, instability in large-scale training, and lack
of validation for video generation. This paper introduces DanceGRPO, the first
unified framework to adapt Group Relative Policy Optimization (GRPO) to visual
generation paradigms, unleashing one unified RL algorithm across two generative
paradigms (diffusion models and rectified flows), three tasks (text-to-image,
text-to-video, image-to-video), four foundation models (Stable Diffusion,
HunyuanVideo, FLUX, SkyReel-I2V), and five reward models (image/video
aesthetics, text-image alignment, video motion quality, and binary reward). To
our knowledge, DanceGRPO is the first RL-based unified framework capable of
seamless adaptation across diverse generative paradigms, tasks, foundational
models, and reward models. DanceGRPO demonstrates consistent and substantial
improvements, which outperform baselines by up to 181% on benchmarks such as
HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Notably, DanceGRPO not only can
stabilize policy optimization for complex video generation, but also enables
generative policy to better capture denoising trajectories for Best-of-N
inference scaling and learn from sparse binary feedback. Our results establish
DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning
from Human Feedback (RLHF) tasks in visual generation, offering new insights
into harmonizing reinforcement learning and visual synthesis. The code will be
released.
DanceGRPO is a unified RL framework that enhances visual generation across different paradigms, tasks, models, and reward systems, outperforming baselines in benchmarks and improving video generation stability.
diffusion models rectified flows reinforcement learning Group Relative Policy Optimization (GRPO) text-to-image text-to-video image-to-video Stable Diffusion HunyuanVideo FLUX SkyReel-I2V image/video aesthetics text-image alignment video motion quality binary reward Best-of-N inference Reinforcement Learning from Human Feedback (RLHF)