title: EvolveDirector: Approaching Advanced Text-to-Image Generation with Large
    Vision-Language Models
summary: Recent advancements in generation models have showcased remarkable
  capabilities in generating fantastic content. However, most of them are trained
  on proprietary high-quality data, and some models withhold their parameters and
  only provide accessible application programming interfaces (APIs), limiting
  their benefits for downstream tasks. To explore the feasibility of training a
  text-to-image generation model comparable to advanced models using publicly
  available resources, we introduce EvolveDirector. This framework interacts with
  advanced models through their public APIs to obtain text-image data pairs to
  train a base model. Our experiments with extensive data indicate that the model
  trained on generated data of the advanced model can approximate its generation
  capability. However, it requires large-scale samples of 10 million or more.
  This incurs significant expenses in time, computational resources, and
  especially the costs associated with calling fee-based APIs. To address this
  problem, we leverage pre-trained large vision-language models (VLMs) to guide
  the evolution of the base model. VLM continuously evaluates the base model
  during training and dynamically updates and refines the training dataset by the
  discrimination, expansion, deletion, and mutation operations. Experimental
  results show that this paradigm significantly reduces the required data volume.
  Furthermore, when approaching multiple advanced models, EvolveDirector can
  select the best samples generated by them to learn powerful and balanced
  abilities. The final trained model Edgen is demonstrated to outperform these
  advanced models. The code and model weights are available at
  https://github.com/showlab/EvolveDirector.
paper_authors._id: 67074fe901f0e95af7736253
paper_authors.name: Rui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:48:43.532Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 67074fe901f0e95af7736254
paper_authors.name: Hangjie Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T08:34:30.810Z
paper_authors._id: 649d54b314afbb10ce2a9eeb
paper_authors.avatarUrl: /avatars/15c325d8c2273ff63569f23015e98486.svg
paper_authors.fullname: Hangjie Yuan
paper_authors.type: user
paper_authors.user: JacobYuan
paper_authors._id: 67074fe901f0e95af7736255
paper_authors.name: Yujie Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:21:30.590Z
paper_authors._id: 637f70d6fab5db9101c3dfc8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
paper_authors.fullname: Yujie Wei
paper_authors.type: user
paper_authors.user: weilllllls
paper_authors._id: 67074fe901f0e95af7736256
paper_authors.name: Shiwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:21:57.406Z
paper_authors._id: 63a1200045edac9f7508bae9
paper_authors.avatarUrl: /avatars/e84f0b045f32c5b8b4da43458650b925.svg
paper_authors.fullname: Shiwei Zhang
paper_authors.type: user
paper_authors.user: StevenZhang
paper_authors._id: 67074fe901f0e95af7736257
paper_authors.name: Yuchao Gu
paper_authors._id: 67074fe901f0e95af7736258
paper_authors.name: Lingmin Ran
paper_authors._id: 67074fe901f0e95af7736259
paper_authors.name: Xiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:19:58.663Z
paper_authors._id: 65fca775fa59bdf4737b1a84
paper_authors.avatarUrl: /avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
paper_authors.fullname: Xiang Wang
paper_authors.type: user
paper_authors.user: xiangwang1223
paper_authors._id: 67074fe901f0e95af773625a
paper_authors.name: Zhangjie Wu
paper_authors._id: 67074fe901f0e95af773625b
paper_authors.name: Junhao Zhang
paper_authors._id: 67074fe901f0e95af773625c
paper_authors.name: Yingya Zhang
paper_authors._id: 67074fe901f0e95af773625d
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:03:10.342Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: EvolveDirector: Approaching Advanced Text-to-Image Generation with Large
    Vision-Language Models
paper_summary: Recent advancements in generation models have showcased remarkable
  capabilities in generating fantastic content. However, most of them are trained
  on proprietary high-quality data, and some models withhold their parameters and
  only provide accessible application programming interfaces (APIs), limiting
  their benefits for downstream tasks. To explore the feasibility of training a
  text-to-image generation model comparable to advanced models using publicly
  available resources, we introduce EvolveDirector. This framework interacts with
  advanced models through their public APIs to obtain text-image data pairs to
  train a base model. Our experiments with extensive data indicate that the model
  trained on generated data of the advanced model can approximate its generation
  capability. However, it requires large-scale samples of 10 million or more.
  This incurs significant expenses in time, computational resources, and
  especially the costs associated with calling fee-based APIs. To address this
  problem, we leverage pre-trained large vision-language models (VLMs) to guide
  the evolution of the base model. VLM continuously evaluates the base model
  during training and dynamically updates and refines the training dataset by the
  discrimination, expansion, deletion, and mutation operations. Experimental
  results show that this paradigm significantly reduces the required data volume.
  Furthermore, when approaching multiple advanced models, EvolveDirector can
  select the best samples generated by them to learn powerful and balanced
  abilities. The final trained model Edgen is demonstrated to outperform these
  advanced models. The code and model weights are available at
  https://github.com/showlab/EvolveDirector.
paper_ai_summary: EvolveDirector leverages pre-trained vision-language models to guide and refine a text-to-image generation model, reducing data requirements and improving performance compared to advanced models.
paper_ai_keywords: text-to-image generation model pre-trained large vision-language models VLMs discrimination expansion deletion mutation Edgen