title: Auto Cherry-Picker: Learning from High-quality Generative Data Driven by
    Language
summary: Diffusion-based models have shown great potential in generating high-quality
  images with various layouts, which can benefit downstream perception tasks.
  However, a fully automatic layout generation driven only by language and a
  suitable metric for measuring multiple generated instances has not been well
  explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework
  that generates high-quality multi-modal training examples to augment perception
  and multi-modal training. Starting with a simple list of natural language
  concepts, we prompt large language models (LLMs) to generate a detailed
  description and design reasonable layouts. Next, we use an off-the-shelf
  text-to-image model to generate multiple images. Then, the generated data are
  refined using a comprehensively designed metric to ensure quality. In
  particular, we present a new metric, Composite Layout and Image Score (CLIS),
  to evaluate the generated images fairly. Our synthetic high-quality examples
  boost performance in various scenarios by customizing the initial concept list,
  especially in addressing challenges associated with long-tailed distribution
  and imbalanced datasets. Experiment results on downstream tasks demonstrate
  that Auto Cherry-Picker can significantly improve the performance of existing
  models. In addition, we have thoroughly investigated the correlation between
  CLIS and performance gains in downstream tasks, and we find that a better CLIS
  score results in better performance. This finding shows the potential for
  evaluation metrics as the role for various visual perception and MLLM tasks.
  Code will be available.
paper_authors._id: 6683d7d517811efbf6741fe9
paper_authors.name: Yicheng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:19:22.420Z
paper_authors._id: 664e7d6d1b242206a6a2b73f
paper_authors.avatarUrl: /avatars/e7377fde18eb89fbf037daaed0c055e8.svg
paper_authors.fullname: Yicheng Chen
paper_authors.type: user
paper_authors.user: cyczzz
paper_authors._id: 6683d7d517811efbf6741fea
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:19:13.197Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 6683d7d517811efbf6741feb
paper_authors.name: Yining Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:19:07.539Z
paper_authors._id: 6570542c91da380dfbdd2d36
paper_authors.avatarUrl: /avatars/cf3e7f7c5bf790f93fc6b4cb5d72c915.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ANTiSOC1AL
paper_authors._id: 6683d7d517811efbf6741fec
paper_authors.name: Yanhong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:18:59.658Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 6683d7d517811efbf6741fed
paper_authors.name: Jianzong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:20:13.413Z
paper_authors._id: 657a6eed1ccc3c2a5ea7b585
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RIQIF-JJdNI0SwJEq_9z7.jpeg
paper_authors.fullname: Jianzong Wu
paper_authors.type: user
paper_authors.user: jianzongwu
paper_authors._id: 6683d7d517811efbf6741fee
paper_authors.name: Xiangyu Zhao
paper_authors._id: 6683d7d517811efbf6741fef
paper_authors.name: Kai Chen
paper_title: Auto Cherry-Picker: Learning from High-quality Generative Data Driven by
    Language
paper_summary: Diffusion-based models have shown great potential in generating high-quality
  images with various layouts, which can benefit downstream perception tasks.
  However, a fully automatic layout generation driven only by language and a
  suitable metric for measuring multiple generated instances has not been well
  explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework
  that generates high-quality multi-modal training examples to augment perception
  and multi-modal training. Starting with a simple list of natural language
  concepts, we prompt large language models (LLMs) to generate a detailed
  description and design reasonable layouts. Next, we use an off-the-shelf
  text-to-image model to generate multiple images. Then, the generated data are
  refined using a comprehensively designed metric to ensure quality. In
  particular, we present a new metric, Composite Layout and Image Score (CLIS),
  to evaluate the generated images fairly. Our synthetic high-quality examples
  boost performance in various scenarios by customizing the initial concept list,
  especially in addressing challenges associated with long-tailed distribution
  and imbalanced datasets. Experiment results on downstream tasks demonstrate
  that Auto Cherry-Picker can significantly improve the performance of existing
  models. In addition, we have thoroughly investigated the correlation between
  CLIS and performance gains in downstream tasks, and we find that a better CLIS
  score results in better performance. This finding shows the potential for
  evaluation metrics as the role for various visual perception and MLLM tasks.
  Code will be available.
paper_ai_keywords: diffusion-based models high-quality images perception tasks layout generation language-driven multi-modal training examples AutoCherry-Picker (ACP) natural language concepts large language models (LLMs) text-to-image model composite layout and image score (CLIS) long-tailed distribution imbalanced scales downstream tasks performance gains