Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language
<|reference_start|>Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language: Diffusion-based models have shown great potential in generating high-quality images with various layouts, which can benefit downstream perception tasks. However, a fully automatic layout generation driven only by language and a suitable metric for measuring multiple generated instances has not been well explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework that generates high-quality multi-modal training examples to augment perception and multi-modal training. Starting with a simple list of natural language concepts, we prompt large language models (LLMs) to generate a detailed description and design reasonable layouts. Next, we use an off-the-shelf text-to-image model to generate multiple images. Then, the generated data are refined using a comprehensively designed metric to ensure quality. In particular, we present a new metric, Composite Layout and Image Score (CLIS), to evaluate the generated images fairly. Our synthetic high-quality examples boost performance in various scenarios by customizing the initial concept list, especially in addressing challenges associated with long-tailed distribution and imbalanced datasets. Experiment results on downstream tasks demonstrate that Auto Cherry-Picker can significantly improve the performance of existing models. In addition, we have thoroughly investigated the correlation between CLIS and performance gains in downstream tasks, and we find that a better CLIS score results in better performance. This finding shows the potential for evaluation metrics as the role for various visual perception and MLLM tasks. Code will be available.<|reference_end|>
@article{chen2024auto,
    title={Auto Cherry-Picker: Learning from High-quality Generative Data Driven by
  Language},
    author={Yicheng Chen, Xiangtai Li, Yining Li, Yanhong Zeng, Jianzong Wu,
  Xiangyu Zhao, Kai Chen},
    journal={arXiv preprint arXiv:2406.20085},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.20085},
    primaryClass={cs.CV}
}