title: D-Flow: Differentiating through Flows for Controlled Generation
summary: Taming the generation outcome of state of the art Diffusion and Flow-Matching
  (FM) models without having to re-train a task-specific model unlocks a powerful
  tool for solving inverse problems, conditional generation, and controlled
  generation in general. In this work we introduce D-Flow, a simple framework for
  controlling the generation process by differentiating through the flow,
  optimizing for the source (noise) point. We motivate this framework by our key
  observation stating that for Diffusion/FM models trained with Gaussian
  probability paths, differentiating through the generation process projects
  gradient on the data manifold, implicitly injecting the prior into the
  optimization process. We validate our framework on linear and non-linear
  controlled generation problems including: image and audio inverse problems and
  conditional molecule generation reaching state of the art performance across
  all.
paper_authors._id: 65d7424a052013f1dff1000a
paper_authors.name: Heli Ben-Hamu
paper_authors._id: 65d7424a052013f1dff1000b
paper_authors.name: Omri Puny
paper_authors._id: 65d7424a052013f1dff1000c
paper_authors.name: Itai Gat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-22T22:27:05.289Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 65d7424a052013f1dff1000d
paper_authors.name: Brian Karrer
paper_authors._id: 65d7424a052013f1dff1000e
paper_authors.name: Uriel Singer
paper_authors._id: 65d7424a052013f1dff1000f
paper_authors.name: Yaron Lipman
paper_title: D-Flow: Differentiating through Flows for Controlled Generation
paper_summary: Taming the generation outcome of state of the art Diffusion and Flow-Matching
  (FM) models without having to re-train a task-specific model unlocks a powerful
  tool for solving inverse problems, conditional generation, and controlled
  generation in general. In this work we introduce D-Flow, a simple framework for
  controlling the generation process by differentiating through the flow,
  optimizing for the source (noise) point. We motivate this framework by our key
  observation stating that for Diffusion/FM models trained with Gaussian
  probability paths, differentiating through the generation process projects
  gradient on the data manifold, implicitly injecting the prior into the
  optimization process. We validate our framework on linear and non-linear
  controlled generation problems including: image and audio inverse problems and
  conditional molecule generation reaching state of the art performance across
  all.
paper_ai_summary: A framework that controls generation in Diffusion and Flow-Matching models by differentiating through the flow enables solving inverse and conditional generation problems with state-of-the-art performance.
paper_ai_keywords: Diffusion models Flow-Matching models D-Flow framework Gaussian probability paths data manifold inverse problems conditional generation