ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation
<|reference_start|>ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation: The practical use of text-to-image generation has evolved from simple, monolithic models to complex workflows that combine multiple specialized components. While workflow-based approaches can lead to improved image quality, crafting effective workflows requires significant expertise, owing to the large number of available components, their complex inter-dependence, and their dependence on the generation prompt. Here, we introduce the novel task of prompt-adaptive workflow generation, where the goal is to automatically tailor a workflow to each user prompt. We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Both approaches lead to improved image quality when compared to monolithic models or generic, prompt-independent workflows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.<|reference_end|>
@article{gal2024comfygen:,
    title={ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation},
    author={Rinon Gal, Adi Haviv, Yuval Alaluf, Amit H. Bermano, Daniel Cohen-Or,
  Gal Chechik},
    journal={arXiv preprint arXiv:2410.01731},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01731},
    primaryClass={cs.CV cs.CL cs.GR}
}