Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven
  Text-to-Image Generation
In subject-driven text-to-image generation, recent works have achieved
superior performance by training the model on synthetic datasets containing
numerous image pairs. Trained on these datasets, generative models can produce
text-aligned images for specific subject from arbitrary testing image in a
zero-shot manner. They even outperform methods which require additional
fine-tuning on testing images. However, the cost of creating such datasets is
prohibitive for most researchers. To generate a single training pair, current
methods fine-tune a pre-trained text-to-image model on the subject image to
capture fine-grained details, then use the fine-tuned model to create images
for the same subject based on creative text prompts. Consequently, constructing
a large-scale dataset with millions of subjects can require hundreds of
thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient
method to construct datasets for subject-driven editing and generation.
Specifically, our dataset construction does not need any subject-level
fine-tuning. After pre-training two generative models, we are able to generate
infinite number of high-quality samples. We construct the first large-scale
dataset for subject-driven image editing and generation, which contains 5
million image pairs, text prompts, and masks. Our dataset is 5 times the size
of previous largest dataset, yet our cost is tens of thousands of GPU hours
lower. To test the proposed dataset, we also propose a model which is capable
of both subject-driven image editing and generation. By simply training the
model on our proposed dataset, it obtains competitive results, illustrating the
effectiveness of the proposed dataset construction framework.
666be5c51dcf1f59373e26b5
Yufan Zhou
claimed_verified
2024-06-14T07:21:28.121Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
666be5c51dcf1f59373e26b6
Ruiyi Zhang
admin_assigned
2024-06-14T11:16:54.638Z
64668f982da1abc242355cbb
/avatars/c9a248b8d70b1a8f7b78265c98690570.svg
Ruiyi Zhang
user
zhangry868
666be5c51dcf1f59373e26b7
Kaizhi Zheng
666be5c51dcf1f59373e26b8
Nanxuan Zhao
666be5c51dcf1f59373e26b9
Jiuxiang Gu
666be5c51dcf1f59373e26ba
Zichao Wang
666be5c51dcf1f59373e26bb
Xin Eric Wang
claimed_verified
2024-06-14T15:40:41.214Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
666be5c51dcf1f59373e26bc
Tong Sun
Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven
  Text-to-Image Generation
In subject-driven text-to-image generation, recent works have achieved
superior performance by training the model on synthetic datasets containing
numerous image pairs. Trained on these datasets, generative models can produce
text-aligned images for specific subject from arbitrary testing image in a
zero-shot manner. They even outperform methods which require additional
fine-tuning on testing images. However, the cost of creating such datasets is
prohibitive for most researchers. To generate a single training pair, current
methods fine-tune a pre-trained text-to-image model on the subject image to
capture fine-grained details, then use the fine-tuned model to create images
for the same subject based on creative text prompts. Consequently, constructing
a large-scale dataset with millions of subjects can require hundreds of
thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient
method to construct datasets for subject-driven editing and generation.
Specifically, our dataset construction does not need any subject-level
fine-tuning. After pre-training two generative models, we are able to generate
infinite number of high-quality samples. We construct the first large-scale
dataset for subject-driven image editing and generation, which contains 5
million image pairs, text prompts, and masks. Our dataset is 5 times the size
of previous largest dataset, yet our cost is tens of thousands of GPU hours
lower. To test the proposed dataset, we also propose a model which is capable
of both subject-driven image editing and generation. By simply training the
model on our proposed dataset, it obtains competitive results, illustrating the
effectiveness of the proposed dataset construction framework.
Toffee efficiently constructs large-scale synthetic datasets for subject-driven text-to-image generation and editing without subject-level fine-tuning, enabling competitive model performance with reduced computational cost.
generative models subject-driven text-to-image generation fine-tuning dataset construction image editing high-quality samples