title: Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven
    Text-to-Image Generation
summary: In subject-driven text-to-image generation, recent works have achieved
  superior performance by training the model on synthetic datasets containing
  numerous image pairs. Trained on these datasets, generative models can produce
  text-aligned images for specific subject from arbitrary testing image in a
  zero-shot manner. They even outperform methods which require additional
  fine-tuning on testing images. However, the cost of creating such datasets is
  prohibitive for most researchers. To generate a single training pair, current
  methods fine-tune a pre-trained text-to-image model on the subject image to
  capture fine-grained details, then use the fine-tuned model to create images
  for the same subject based on creative text prompts. Consequently, constructing
  a large-scale dataset with millions of subjects can require hundreds of
  thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient
  method to construct datasets for subject-driven editing and generation.
  Specifically, our dataset construction does not need any subject-level
  fine-tuning. After pre-training two generative models, we are able to generate
  infinite number of high-quality samples. We construct the first large-scale
  dataset for subject-driven image editing and generation, which contains 5
  million image pairs, text prompts, and masks. Our dataset is 5 times the size
  of previous largest dataset, yet our cost is tens of thousands of GPU hours
  lower. To test the proposed dataset, we also propose a model which is capable
  of both subject-driven image editing and generation. By simply training the
  model on our proposed dataset, it obtains competitive results, illustrating the
  effectiveness of the proposed dataset construction framework.
paper_authors._id: 666be5c51dcf1f59373e26b5
paper_authors.name: Yufan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:21:28.121Z
paper_authors._id: 635c2c2a7a165601151d3f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
paper_authors.fullname: Yufan Zhou
paper_authors.type: user
paper_authors.user: YfZ
paper_authors._id: 666be5c51dcf1f59373e26b6
paper_authors.name: Ruiyi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:16:54.638Z
paper_authors._id: 64668f982da1abc242355cbb
paper_authors.avatarUrl: /avatars/c9a248b8d70b1a8f7b78265c98690570.svg
paper_authors.fullname: Ruiyi Zhang
paper_authors.type: user
paper_authors.user: zhangry868
paper_authors._id: 666be5c51dcf1f59373e26b7
paper_authors.name: Kaizhi Zheng
paper_authors._id: 666be5c51dcf1f59373e26b8
paper_authors.name: Nanxuan Zhao
paper_authors._id: 666be5c51dcf1f59373e26b9
paper_authors.name: Jiuxiang Gu
paper_authors._id: 666be5c51dcf1f59373e26ba
paper_authors.name: Zichao Wang
paper_authors._id: 666be5c51dcf1f59373e26bb
paper_authors.name: Xin Eric Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T15:40:41.214Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_authors._id: 666be5c51dcf1f59373e26bc
paper_authors.name: Tong Sun
paper_title: Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven
    Text-to-Image Generation
paper_summary: In subject-driven text-to-image generation, recent works have achieved
  superior performance by training the model on synthetic datasets containing
  numerous image pairs. Trained on these datasets, generative models can produce
  text-aligned images for specific subject from arbitrary testing image in a
  zero-shot manner. They even outperform methods which require additional
  fine-tuning on testing images. However, the cost of creating such datasets is
  prohibitive for most researchers. To generate a single training pair, current
  methods fine-tune a pre-trained text-to-image model on the subject image to
  capture fine-grained details, then use the fine-tuned model to create images
  for the same subject based on creative text prompts. Consequently, constructing
  a large-scale dataset with millions of subjects can require hundreds of
  thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient
  method to construct datasets for subject-driven editing and generation.
  Specifically, our dataset construction does not need any subject-level
  fine-tuning. After pre-training two generative models, we are able to generate
  infinite number of high-quality samples. We construct the first large-scale
  dataset for subject-driven image editing and generation, which contains 5
  million image pairs, text prompts, and masks. Our dataset is 5 times the size
  of previous largest dataset, yet our cost is tens of thousands of GPU hours
  lower. To test the proposed dataset, we also propose a model which is capable
  of both subject-driven image editing and generation. By simply training the
  model on our proposed dataset, it obtains competitive results, illustrating the
  effectiveness of the proposed dataset construction framework.
paper_ai_keywords: generative models text-to-image generation subject-driven text-to-image generation synthetic datasets image pairs zero-shot manner fine-tuning text-aligned images pre-trained text-to-image model creative text prompts dataset construction magic Toffee high-quality samples subject-driven image editing masks deep learning models