EvolveDirector: Approaching Advanced Text-to-Image Generation with Large
  Vision-Language Models
Recent advancements in generation models have showcased remarkable
capabilities in generating fantastic content. However, most of them are trained
on proprietary high-quality data, and some models withhold their parameters and
only provide accessible application programming interfaces (APIs), limiting
their benefits for downstream tasks. To explore the feasibility of training a
text-to-image generation model comparable to advanced models using publicly
available resources, we introduce EvolveDirector. This framework interacts with
advanced models through their public APIs to obtain text-image data pairs to
train a base model. Our experiments with extensive data indicate that the model
trained on generated data of the advanced model can approximate its generation
capability. However, it requires large-scale samples of 10 million or more.
This incurs significant expenses in time, computational resources, and
especially the costs associated with calling fee-based APIs. To address this
problem, we leverage pre-trained large vision-language models (VLMs) to guide
the evolution of the base model. VLM continuously evaluates the base model
during training and dynamically updates and refines the training dataset by the
discrimination, expansion, deletion, and mutation operations. Experimental
results show that this paradigm significantly reduces the required data volume.
Furthermore, when approaching multiple advanced models, EvolveDirector can
select the best samples generated by them to learn powerful and balanced
abilities. The final trained model Edgen is demonstrated to outperform these
advanced models. The code and model weights are available at
https://github.com/showlab/EvolveDirector.
67074fe901f0e95af7736253
Rui Zhao
claimed_verified
2024-10-14T07:48:43.532Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
67074fe901f0e95af7736254
Hangjie Yuan
admin_assigned
2024-10-14T08:34:30.810Z
649d54b314afbb10ce2a9eeb
/avatars/15c325d8c2273ff63569f23015e98486.svg
Hangjie Yuan
user
JacobYuan
67074fe901f0e95af7736255
Yujie Wei
admin_assigned
2024-10-14T12:21:30.590Z
637f70d6fab5db9101c3dfc8
https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
Yujie Wei
user
weilllllls
67074fe901f0e95af7736256
Shiwei Zhang
admin_assigned
2024-10-14T12:21:57.406Z
63a1200045edac9f7508bae9
/avatars/e84f0b045f32c5b8b4da43458650b925.svg
Shiwei Zhang
user
StevenZhang
67074fe901f0e95af7736257
Yuchao Gu
67074fe901f0e95af7736258
Lingmin Ran
67074fe901f0e95af7736259
Xiang Wang
admin_assigned
2024-10-14T12:19:58.663Z
65fca775fa59bdf4737b1a84
/avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
Xiang Wang
user
xiangwang1223
67074fe901f0e95af773625a
Zhangjie Wu
67074fe901f0e95af773625b
Junhao Zhang
67074fe901f0e95af773625c
Yingya Zhang
67074fe901f0e95af773625d
Mike Zheng Shou
admin_assigned
2024-10-14T12:03:10.342Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
EvolveDirector: Approaching Advanced Text-to-Image Generation with Large
  Vision-Language Models
Recent advancements in generation models have showcased remarkable
capabilities in generating fantastic content. However, most of them are trained
on proprietary high-quality data, and some models withhold their parameters and
only provide accessible application programming interfaces (APIs), limiting
their benefits for downstream tasks. To explore the feasibility of training a
text-to-image generation model comparable to advanced models using publicly
available resources, we introduce EvolveDirector. This framework interacts with
advanced models through their public APIs to obtain text-image data pairs to
train a base model. Our experiments with extensive data indicate that the model
trained on generated data of the advanced model can approximate its generation
capability. However, it requires large-scale samples of 10 million or more.
This incurs significant expenses in time, computational resources, and
especially the costs associated with calling fee-based APIs. To address this
problem, we leverage pre-trained large vision-language models (VLMs) to guide
the evolution of the base model. VLM continuously evaluates the base model
during training and dynamically updates and refines the training dataset by the
discrimination, expansion, deletion, and mutation operations. Experimental
results show that this paradigm significantly reduces the required data volume.
Furthermore, when approaching multiple advanced models, EvolveDirector can
select the best samples generated by them to learn powerful and balanced
abilities. The final trained model Edgen is demonstrated to outperform these
advanced models. The code and model weights are available at
https://github.com/showlab/EvolveDirector.
EvolveDirector leverages pre-trained vision-language models to guide and refine a text-to-image generation model, reducing data requirements and improving performance compared to advanced models.
text-to-image generation model pre-trained large vision-language models VLMs discrimination expansion deletion mutation Edgen