Steering Rectified Flow Models in the Vector Field for Controlled Image
  Generation
Diffusion models (DMs) excel in photorealism, image editing, and solving
inverse problems, aided by classifier-free guidance and image inversion
techniques. However, rectified flow models (RFMs) remain underexplored for
these tasks. Existing DM-based methods often require additional training, lack
generalization to pretrained latent models, underperform, and demand
significant computational resources due to extensive backpropagation through
ODE solvers and inversion processes. In this work, we first develop a
theoretical and empirical understanding of the vector field dynamics of RFMs in
efficiently guiding the denoising trajectory. Our findings reveal that we can
navigate the vector field in a deterministic and gradient-free manner.
Utilizing this property, we propose FlowChef, which leverages the vector field
to steer the denoising trajectory for controlled image generation tasks,
facilitated by gradient skipping. FlowChef is a unified framework for
controlled image generation that, for the first time, simultaneously addresses
classifier guidance, linear inverse problems, and image editing without the
need for extra training, inversion, or intensive backpropagation. Finally, we
perform extensive evaluations and show that FlowChef significantly outperforms
baselines in terms of performance, memory, and time requirements, achieving new
state-of-the-art results. Project Page: https://flowchef.github.io.
674e89d1ec384b629bba5e61
Maitreya Patel
claimed_verified
2024-12-03T11:14:47.767Z
622d2ff38d04fd29a9ccf1a7
https://cdn-avatars.huggingface.co/v1/production/uploads/622d2ff38d04fd29a9ccf1a7/ORpGrlU8Lm_oSEVftcZEK.jpeg
Maitreya Patel
user
mpatel57
674e89d1ec384b629bba5e62
Song Wen
claimed_verified
2024-12-04T09:46:25.505Z
638d79bf5869d78eb7e8ccec
/avatars/c047dc15cd650f91a276115d8d1ad10a.svg
Song Wen
user
swen9
674e89d1ec384b629bba5e63
Dimitris N. Metaxas
674e89d1ec384b629bba5e64
Yezhou Yang
claimed_verified
2024-12-03T18:43:20.762Z
6486a850903d5878eb98110f
https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
'YZ' Yezhou Yang
user
AiWithoutBorders
Steering Rectified Flow Models in the Vector Field for Controlled Image
  Generation
Diffusion models (DMs) excel in photorealism, image editing, and solving
inverse problems, aided by classifier-free guidance and image inversion
techniques. However, rectified flow models (RFMs) remain underexplored for
these tasks. Existing DM-based methods often require additional training, lack
generalization to pretrained latent models, underperform, and demand
significant computational resources due to extensive backpropagation through
ODE solvers and inversion processes. In this work, we first develop a
theoretical and empirical understanding of the vector field dynamics of RFMs in
efficiently guiding the denoising trajectory. Our findings reveal that we can
navigate the vector field in a deterministic and gradient-free manner.
Utilizing this property, we propose FlowChef, which leverages the vector field
to steer the denoising trajectory for controlled image generation tasks,
facilitated by gradient skipping. FlowChef is a unified framework for
controlled image generation that, for the first time, simultaneously addresses
classifier guidance, linear inverse problems, and image editing without the
need for extra training, inversion, or intensive backpropagation. Finally, we
perform extensive evaluations and show that FlowChef significantly outperforms
baselines in terms of performance, memory, and time requirements, achieving new
state-of-the-art results. Project Page: https://flowchef.github.io.
FlowChef is a unified framework for controlled image generation using rectified flow models, achieving improvements over diffusion models in photorealism, image editing, and inverse problems without additional training or intensive backpropagation.
diffusion models rectified flow models vector field dynamics denoising trajectory gradient skipping classifier guidance linear inverse problems image editing