SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flow
<|reference_start|>SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flow: Diffusion models excel in high-quality generation but suffer from slow inference due to iterative sampling. While recent methods have successfully transformed diffusion models into one-step generators, they neglect model size reduction, limiting their applicability in compute-constrained scenarios. This paper aims to develop small, efficient one-step diffusion models based on the powerful rectified flow framework, by exploring joint compression of inference steps and model size. The rectified flow framework trains one-step generative models using two operations, reflow and distillation. Compared with the original framework, squeezing the model size brings two new challenges: (1) the initialization mismatch between large teachers and small students during reflow; (2) the underperformance of naive distillation on small student models. To overcome these issues, we propose Annealing Reflow and Flow-Guided Distillation, which together comprise our SlimFlow framework. With our novel framework, we train a one-step diffusion model with an FID of 5.02 and 15.7M parameters, outperforming the previous state-of-the-art one-step diffusion model (FID=6.47, 19.4M parameters) on CIFAR10. On ImageNet 64$\times$64 and FFHQ 64$\times$64, our method yields small one-step diffusion models that are comparable to larger models, showcasing the effectiveness of our method in creating compact, efficient one-step diffusion models.<|reference_end|>
@article{zhu2024slimflow:,
    title={SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flow},
    author={Yuanzhi Zhu, Xingchao Liu, Qiang Liu},
    journal={arXiv preprint arXiv:2407.12718},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12718},
    primaryClass={cs.CV}
}