D-Flow: Differentiating through Flows for Controlled Generation
Taming the generation outcome of state of the art Diffusion and Flow-Matching
(FM) models without having to re-train a task-specific model unlocks a powerful
tool for solving inverse problems, conditional generation, and controlled
generation in general. In this work we introduce D-Flow, a simple framework for
controlling the generation process by differentiating through the flow,
optimizing for the source (noise) point. We motivate this framework by our key
observation stating that for Diffusion/FM models trained with Gaussian
probability paths, differentiating through the generation process projects
gradient on the data manifold, implicitly injecting the prior into the
optimization process. We validate our framework on linear and non-linear
controlled generation problems including: image and audio inverse problems and
conditional molecule generation reaching state of the art performance across
all.
65d7424a052013f1dff1000a
Heli Ben-Hamu
65d7424a052013f1dff1000b
Omri Puny
65d7424a052013f1dff1000c
Itai Gat
claimed_verified
2024-02-22T22:27:05.289Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
65d7424a052013f1dff1000d
Brian Karrer
65d7424a052013f1dff1000e
Uriel Singer
65d7424a052013f1dff1000f
Yaron Lipman
D-Flow: Differentiating through Flows for Controlled Generation
Taming the generation outcome of state of the art Diffusion and Flow-Matching
(FM) models without having to re-train a task-specific model unlocks a powerful
tool for solving inverse problems, conditional generation, and controlled
generation in general. In this work we introduce D-Flow, a simple framework for
controlling the generation process by differentiating through the flow,
optimizing for the source (noise) point. We motivate this framework by our key
observation stating that for Diffusion/FM models trained with Gaussian
probability paths, differentiating through the generation process projects
gradient on the data manifold, implicitly injecting the prior into the
optimization process. We validate our framework on linear and non-linear
controlled generation problems including: image and audio inverse problems and
conditional molecule generation reaching state of the art performance across
all.
A framework that controls generation in Diffusion and Flow-Matching models by differentiating through the flow enables solving inverse and conditional generation problems with state-of-the-art performance.
Diffusion models Flow-Matching models D-Flow framework Gaussian probability paths data manifold inverse problems conditional generation