Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Models
<|reference_start|>Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Models: We propose a Bayesian framework for fine-tuning large diffusion models with a novel network structure called Bayesian Power Steering (BPS). We clarify the meaning behind adaptation from a \textit{large probability space} to a \textit{small probability space} and explore the task of fine-tuning pre-trained models using learnable modules from a Bayesian perspective. BPS extracts task-specific knowledge from a pre-trained model's learned prior distribution. It efficiently leverages large diffusion models, differentially intervening different hidden features with a head-heavy and foot-light configuration. Experiments highlight the superiority of BPS over contemporary methods across a range of tasks even with limited amount of data. Notably, BPS attains an FID score of 10.49 under the sketch condition on the COCO17 dataset.<|reference_end|>
@article{huang2024bayesian,
    title={Bayesian Power Steering: An Effective Approach for Domain Adaptation of
  Diffusion Models},
    author={Ding Huang and Ting Li and Jian Huang},
    journal={arXiv preprint arXiv:2406.03683},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03683},
    primaryClass={cs.LG stat.ML}
}