Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control
<|reference_start|>Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control: Dynamical generative models that produce samples through an iterative process, such as Flow Matching and denoising diffusion models, have seen widespread use, but there has not been many theoretically-sound methods for improving these models with reward fine-tuning. In this work, we cast reward fine-tuning as stochastic optimal control (SOC). Critically, we prove that a very specific memoryless noise schedule must be enforced during fine-tuning, in order to account for the dependency between the noise variable and the generated samples. We also propose a new algorithm named Adjoint Matching which outperforms existing SOC algorithms, by casting SOC problems as a regression problem. We find that our approach significantly improves over existing methods for reward fine-tuning, achieving better consistency, realism, and generalization to unseen human preference reward models, while retaining sample diversity.<|reference_end|>
@article{domingo-enrich2024adjoint,
    title={Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with
  Memoryless Stochastic Optimal Control},
    author={Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, Ricky T. Q. Chen},
    journal={arXiv preprint arXiv:2409.08861},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.08861},
    primaryClass={cs.LG math.OC stat.ML}
}