MinorityPrompt: Text to Minority Image Generation via Prompt Optimization
<|reference_start|>MinorityPrompt: Text to Minority Image Generation via Prompt Optimization: We investigate the generation of minority samples using pretrained text-to-image (T2I) latent diffusion models. Minority instances, in the context of T2I generation, can be defined as ones living on low-density regions of text-conditional data distributions. They are valuable for various applications of modern T2I generators, such as data augmentation and creative AI. Unfortunately, existing pretrained T2I diffusion models primarily focus on high-density regions, largely due to the influence of guided samplers (like CFG) that are essential for producing high-quality generations. To address this, we present a novel framework to counter the high-density-focus of T2I diffusion models. Specifically, we first develop an online prompt optimization framework that can encourage the emergence of desired properties during inference while preserving semantic contents of user-provided prompts. We subsequently tailor this generic prompt optimizer into a specialized solver that promotes the generation of minority features by incorporating a carefully-crafted likelihood objective. Our comprehensive experiments, conducted across various types of T2I models, demonstrate that our approach significantly enhances the capability to produce high-quality minority instances compared to existing samplers.<|reference_end|>
@article{um2024minorityprompt:,
    title={MinorityPrompt: Text to Minority Image Generation via Prompt
  Optimization},
    author={Soobin Um, Jong Chul Ye},
    journal={arXiv preprint arXiv:2410.07838},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07838},
    primaryClass={cs.CV cs.AI cs.LG}
}