title: VARD: Efficient and Dense Fine-Tuning for Diffusion Models with
    Value-based RL
summary: Diffusion models have emerged as powerful generative tools across various
  domains, yet tailoring pre-trained models to exhibit specific desirable
  properties remains challenging. While reinforcement learning (RL) offers a
  promising solution,current methods struggle to simultaneously achieve stable,
  efficient fine-tuning and support non-differentiable rewards. Furthermore,
  their reliance on sparse rewards provides inadequate supervision during
  intermediate steps, often resulting in suboptimal generation quality. To
  address these limitations, dense and differentiable signals are required
  throughout the diffusion process. Hence, we propose VAlue-based Reinforced
  Diffusion (VARD): a novel approach that first learns a value function
  predicting expection of rewards from intermediate states, and subsequently uses
  this value function with KL regularization to provide dense supervision
  throughout the generation process. Our method maintains proximity to the
  pretrained model while enabling effective and stable training via
  backpropagation. Experimental results demonstrate that our approach facilitates
  better trajectory guidance, improves training efficiency and extends the
  applicability of RL to diffusion models optimized for complex,
  non-differentiable reward functions.
paper_authors._id: 682e8e9f1d7637a2584b6b8c
paper_authors.name: Fengyuan Dai
paper_authors._id: 682e8e9f1d7637a2584b6b8d
paper_authors.name: Zifeng Zhuang
paper_authors._id: 682e8e9f1d7637a2584b6b8e
paper_authors.name: Yufei Huang
paper_authors._id: 682e8e9f1d7637a2584b6b8f
paper_authors.name: Siteng Huang
paper_authors._id: 682e8e9f1d7637a2584b6b90
paper_authors.name: Bangyan Liao
paper_authors._id: 682e8e9f1d7637a2584b6b91
paper_authors.name: Donglin Wang
paper_authors._id: 682e8e9f1d7637a2584b6b92
paper_authors.name: Fajie Yuan
paper_title: VARD: Efficient and Dense Fine-Tuning for Diffusion Models with
    Value-based RL
paper_summary: Diffusion models have emerged as powerful generative tools across various
  domains, yet tailoring pre-trained models to exhibit specific desirable
  properties remains challenging. While reinforcement learning (RL) offers a
  promising solution,current methods struggle to simultaneously achieve stable,
  efficient fine-tuning and support non-differentiable rewards. Furthermore,
  their reliance on sparse rewards provides inadequate supervision during
  intermediate steps, often resulting in suboptimal generation quality. To
  address these limitations, dense and differentiable signals are required
  throughout the diffusion process. Hence, we propose VAlue-based Reinforced
  Diffusion (VARD): a novel approach that first learns a value function
  predicting expection of rewards from intermediate states, and subsequently uses
  this value function with KL regularization to provide dense supervision
  throughout the generation process. Our method maintains proximity to the
  pretrained model while enabling effective and stable training via
  backpropagation. Experimental results demonstrate that our approach facilitates
  better trajectory guidance, improves training efficiency and extends the
  applicability of RL to diffusion models optimized for complex,
  non-differentiable reward functions.
paper_ai_summary: VARD introduces a value function based reinforcement learning approach to enhance diffusion models with dense and differentiable supervision, improving training efficiency and handling non-differentiable rewards.
paper_ai_keywords: diffusion models reinforcement learning stable fine-tuning non-differentiable rewards sparse rewards value function dense supervision KL regularization backpropagation trajectory guidance training efficiency