Information Theoretic Text-to-Image Alignment
<|reference_start|>Information Theoretic Text-to-Image Alignment: Diffusion models for Text-to-Image (T2I) conditional generation have seen tremendous success recently. Despite their success, accurately capturing user intentions with these models still requires a laborious trial and error process. This challenge is commonly identified as a model alignment problem, an issue that has attracted considerable attention by the research community. Instead of relying on fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-language models to steer image generation, in this work we present a novel method that relies on an information-theoretic alignment measure. In a nutshell, our method uses self-supervised fine-tuning and relies on point-wise mutual information between prompts and images to define a synthetic training set to induce model alignment. Our comparative analysis shows that our method is on-par or superior to the state-of-the-art, yet requires nothing but a pre-trained denoising network to estimate MI and a lightweight fine-tuning strategy.<|reference_end|>
@article{wang2024information,
    title={Information Theoretic Text-to-Image Alignment},
    author={Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, Pietro
  Michiardi},
    journal={arXiv preprint arXiv:2405.20759},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.20759},
    primaryClass={cs.LG cs.CV}
}