title: Free^2Guide: Gradient-Free Path Integral Control for Enhancing
    Text-to-Video Generation with Large Vision-Language Models
summary: Diffusion models have achieved impressive results in generative tasks like
  text-to-image (T2I) and text-to-video (T2V) synthesis. However, achieving
  accurate text alignment in T2V generation remains challenging due to the
  complex temporal dependency across frames. Existing reinforcement learning
  (RL)-based approaches to enhance text alignment often require differentiable
  reward functions or are constrained to limited prompts, hindering their
  scalability and applicability. In this paper, we propose Free^2Guide, a novel
  gradient-free framework for aligning generated videos with text prompts without
  requiring additional model training. Leveraging principles from path integral
  control, Free^2Guide approximates guidance for diffusion models using
  non-differentiable reward functions, thereby enabling the integration of
  powerful black-box Large Vision-Language Models (LVLMs) as reward model.
  Additionally, our framework supports the flexible ensembling of multiple reward
  models, including large-scale image-based models, to synergistically enhance
  alignment without incurring substantial computational overhead. We demonstrate
  that Free^2Guide significantly improves text alignment across various
  dimensions and enhances the overall quality of generated videos.
paper_authors._id: 674952f5ad6772de5d595ae8
paper_authors.name: Jaemin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-29T09:33:20.413Z
paper_authors._id: 64c3732de6c3860fba66ceb0
paper_authors.avatarUrl: /avatars/785783ca08687923053eac641326281f.svg
paper_authors.fullname: JaeminKim
paper_authors.type: user
paper_authors.user: kjm981995
paper_authors._id: 674952f5ad6772de5d595ae9
paper_authors.name: Bryan S Kim
paper_authors._id: 674952f5ad6772de5d595aea
paper_authors.name: Jong Chul Ye
paper_title: Free^2Guide: Gradient-Free Path Integral Control for Enhancing
    Text-to-Video Generation with Large Vision-Language Models
paper_summary: Diffusion models have achieved impressive results in generative tasks like
  text-to-image (T2I) and text-to-video (T2V) synthesis. However, achieving
  accurate text alignment in T2V generation remains challenging due to the
  complex temporal dependency across frames. Existing reinforcement learning
  (RL)-based approaches to enhance text alignment often require differentiable
  reward functions or are constrained to limited prompts, hindering their
  scalability and applicability. In this paper, we propose Free^2Guide, a novel
  gradient-free framework for aligning generated videos with text prompts without
  requiring additional model training. Leveraging principles from path integral
  control, Free^2Guide approximates guidance for diffusion models using
  non-differentiable reward functions, thereby enabling the integration of
  powerful black-box Large Vision-Language Models (LVLMs) as reward model.
  Additionally, our framework supports the flexible ensembling of multiple reward
  models, including large-scale image-based models, to synergistically enhance
  alignment without incurring substantial computational overhead. We demonstrate
  that Free^2Guide significantly improves text alignment across various
  dimensions and enhances the overall quality of generated videos.
paper_ai_keywords: diffusion models text-to-image (T2I) text-to-video (T2V) synthesis reinforcement learning (RL) differentiable reward functions path integral control