Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
<|reference_start|>Diffusion Soup: Model Merging for Text-to-Image Diffusion Models: We present Diffusion Soup, a compartmentalization method for Text-to-Image Generation that averages the weights of diffusion models trained on sharded data. By construction, our approach enables training-free continual learning and unlearning with no additional memory or inference costs, since models corresponding to data shards can be added or removed by re-averaging. We show that Diffusion Soup samples from a point in weight space that approximates the geometric mean of the distributions of constituent datasets, which offers anti-memorization guarantees and enables zero-shot style mixing. Empirically, Diffusion Soup outperforms a paragon model trained on the union of all data shards and achieves a 30% improvement in Image Reward (.34 $\to$ .44) on domain sharded data, and a 59% improvement in IR (.37 $\to$ .59) on aesthetic data. In both cases, souping also prevails in TIFA score (respectively, 85.5 $\to$ 86.5 and 85.6 $\to$ 86.8). We demonstrate robust unlearning -- removing any individual domain shard only lowers performance by 1% in IR (.45 $\to$ .44) -- and validate our theoretical insights on anti-memorization using real data. Finally, we showcase Diffusion Soup's ability to blend the distinct styles of models finetuned on different shards, resulting in the zero-shot generation of hybrid styles.<|reference_end|>
@article{biggs2024diffusion,
    title={Diffusion Soup: Model Merging for Text-to-Image Diffusion Models},
    author={Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar,
  Yusheng Xie, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto},
    journal={arXiv preprint arXiv:2406.08431},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.08431},
    primaryClass={cs.CV cs.AI cs.CR cs.LG}
}