title: Flow-GRPO: Training Flow Matching Models via Online RL
summary: We propose Flow-GRPO, the first method integrating online reinforcement
  learning (RL) into flow matching models. Our approach uses two key strategies:
  (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary
  Differential Equation (ODE) into an equivalent Stochastic Differential Equation
  (SDE) that matches the original model's marginal distribution at all timesteps,
  enabling statistical sampling for RL exploration; and (2) a Denoising Reduction
  strategy that reduces training denoising steps while retaining the original
  inference timestep number, significantly improving sampling efficiency without
  performance degradation. Empirically, Flow-GRPO is effective across multiple
  text-to-image tasks. For complex compositions, RL-tuned SD3.5 generates nearly
  perfect object counts, spatial relations, and fine-grained attributes, boosting
  GenEval accuracy from 63% to 95%. In visual text rendering, its accuracy
  improves from 59% to 92%, significantly enhancing text generation.
  Flow-GRPO also achieves substantial gains in human preference alignment.
  Notably, little to no reward hacking occurred, meaning rewards did not increase
  at the cost of image quality or diversity, and both remained stable in our
  experiments.
paper_authors._id: 681d9829edf34a77aab565eb
paper_authors.name: Jie Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T13:54:46.048Z
paper_authors._id: 639be86b59473c6ae02ef9c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
paper_authors.fullname: Jie Liu
paper_authors.type: user
paper_authors.user: jieliu
paper_authors._id: 681d9829edf34a77aab565ec
paper_authors.name: Gongye Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:41:47.403Z
paper_authors._id: 6553316bf151de82f6a23e1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
paper_authors.fullname: Gongye Liu
paper_authors.type: user
paper_authors.user: liuhuohuo
paper_authors._id: 681d9829edf34a77aab565ed
paper_authors.name: Jiajun Liang
paper_authors._id: 681d9829edf34a77aab565ee
paper_authors.name: Yangguang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:43:44.697Z
paper_authors._id: 64d71083a787c9bc7b9f1238
paper_authors.avatarUrl: /avatars/d0b0546dec7fc5792921154bec41385a.svg
paper_authors.fullname: Yangguang Li
paper_authors.type: user
paper_authors.user: Lp256
paper_authors._id: 681d9829edf34a77aab565ef
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:45:07.297Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 681d9829edf34a77aab565f0
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:42:27.977Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 681d9829edf34a77aab565f1
paper_authors.name: Pengfei Wan
paper_authors._id: 681d9829edf34a77aab565f2
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:43:01.366Z
paper_authors._id: 644c8324f02250233d0d67d9
paper_authors.avatarUrl: /avatars/feb39d281457c1750f3eada3c060a23e.svg
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: dizhang
paper_authors._id: 681d9829edf34a77aab565f3
paper_authors.name: Wanli Ouyang
paper_title: Flow-GRPO: Training Flow Matching Models via Online RL
paper_summary: We propose Flow-GRPO, the first method integrating online reinforcement
  learning (RL) into flow matching models. Our approach uses two key strategies:
  (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary
  Differential Equation (ODE) into an equivalent Stochastic Differential Equation
  (SDE) that matches the original model's marginal distribution at all timesteps,
  enabling statistical sampling for RL exploration; and (2) a Denoising Reduction
  strategy that reduces training denoising steps while retaining the original
  inference timestep number, significantly improving sampling efficiency without
  performance degradation. Empirically, Flow-GRPO is effective across multiple
  text-to-image tasks. For complex compositions, RL-tuned SD3.5 generates nearly
  perfect object counts, spatial relations, and fine-grained attributes, boosting
  GenEval accuracy from 63% to 95%. In visual text rendering, its accuracy
  improves from 59% to 92%, significantly enhancing text generation.
  Flow-GRPO also achieves substantial gains in human preference alignment.
  Notably, little to no reward hacking occurred, meaning rewards did not increase
  at the cost of image quality or diversity, and both remained stable in our
  experiments.
paper_ai_summary: Flow-GRPO combines online reinforcement learning with flow matching models through an ODE-to-SDE conversion and denoising reduction, improving sampling efficiency and performance across text-to-image tasks.
paper_ai_keywords: Flow-GRPO online reinforcement learning flow matching models ODE-to-SDE conversion Stochastic Differential Equation Denoising Reduction text-to-image tasks RL-tuned SD3.5 GenEval visual text rendering human preference alignment