title: Gen4Gen: Generative Data Pipeline for Generative Multi-Concept
    Composition
summary: Recent text-to-image diffusion models are able to learn and synthesize images
  containing novel, personalized concepts (e.g., their own pets or specific
  items) with just a few examples for training. This paper tackles two
  interconnected issues within this realm of personalizing text-to-image
  diffusion models. First, current personalization techniques fail to reliably
  extend to multiple concepts -- we hypothesize this to be due to the mismatch
  between complex scenes and simple text descriptions in the pre-training dataset
  (e.g., LAION). Second, given an image containing multiple personalized
  concepts, there lacks a holistic metric that evaluates performance on not just
  the degree of resemblance of personalized concepts, but also whether all
  concepts are present in the image and whether the image accurately reflects the
  overall text description. To address these issues, we introduce Gen4Gen, a
  semi-automated dataset creation pipeline utilizing generative models to combine
  personalized concepts into complex compositions along with text-descriptions.
  Using this, we create a dataset called MyCanvas, that can be used to benchmark
  the task of multi-concept personalization. In addition, we design a
  comprehensive metric comprising two scores (CP-CLIP and TI-CLIP) for better
  quantifying the performance of multi-concept, personalized text-to-image
  diffusion methods. We provide a simple baseline built on top of Custom
  Diffusion with empirical prompting strategies for future researchers to
  evaluate on MyCanvas. We show that by improving data quality and prompting
  strategies, we can significantly increase multi-concept personalized image
  generation quality, without requiring any modifications to model architecture
  or training algorithms.
paper_authors._id: 65dbf547f386d08eb0298b35
paper_authors.name: Chun-Hsiao Yeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-26T09:24:24.689Z
paper_authors._id: 6499eca0685215f7247bd5ce
paper_authors.avatarUrl: /avatars/b6fea0c33c3c930c7314b99b414072a9.svg
paper_authors.fullname: Chun-Hsiao Yeh
paper_authors.type: user
paper_authors.user: danielchyeh
paper_authors._id: 65dbf547f386d08eb0298b36
paper_authors.name: Ta-Ying Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:20:41.984Z
paper_authors._id: 6442b024e255a338677d78fa
paper_authors.avatarUrl: /avatars/516769c6c3bc7a0d416924181c6866dc.svg
paper_authors.fullname: Ta-Ying Cheng
paper_authors.type: user
paper_authors.user: chengtim
paper_authors._id: 65dbf547f386d08eb0298b37
paper_authors.name: He-Yen Hsieh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-28T08:28:27.412Z
paper_authors._id: 65de5db9f546556af598426a
paper_authors.avatarUrl: /avatars/6f179ea76af9edc995d494517c3086ac.svg
paper_authors.fullname: Hsieh
paper_authors.type: user
paper_authors.user: He-Yen
paper_authors._id: 65dbf547f386d08eb0298b38
paper_authors.name: Chuan-En Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:20:54.997Z
paper_authors._id: 617e396ba7481f8efe4c82fd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/617e396ba7481f8efe4c82fd/SaxzURusTKUubf0sFazBd.png
paper_authors.fullname: David Chuan-En Lin
paper_authors.type: user
paper_authors.user: chuanenlin
paper_authors._id: 65dbf547f386d08eb0298b39
paper_authors.name: Yi Ma
paper_authors._id: 65dbf547f386d08eb0298b3a
paper_authors.name: Andrew Markham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:21:53.883Z
paper_authors._id: 63fca1aa9df782461c7c6367
paper_authors.avatarUrl: /avatars/d61de110a4e87da80a5ca5820921773a.svg
paper_authors.fullname: Andrew Markham
paper_authors.type: user
paper_authors.user: markham
paper_authors._id: 65dbf547f386d08eb0298b3b
paper_authors.name: Niki Trigoni
paper_authors._id: 65dbf547f386d08eb0298b3c
paper_authors.name: H. T. Kung
paper_authors._id: 65dbf547f386d08eb0298b3d
paper_authors.name: Yubei Chen
paper_title: Gen4Gen: Generative Data Pipeline for Generative Multi-Concept
    Composition
paper_summary: Recent text-to-image diffusion models are able to learn and synthesize images
  containing novel, personalized concepts (e.g., their own pets or specific
  items) with just a few examples for training. This paper tackles two
  interconnected issues within this realm of personalizing text-to-image
  diffusion models. First, current personalization techniques fail to reliably
  extend to multiple concepts -- we hypothesize this to be due to the mismatch
  between complex scenes and simple text descriptions in the pre-training dataset
  (e.g., LAION). Second, given an image containing multiple personalized
  concepts, there lacks a holistic metric that evaluates performance on not just
  the degree of resemblance of personalized concepts, but also whether all
  concepts are present in the image and whether the image accurately reflects the
  overall text description. To address these issues, we introduce Gen4Gen, a
  semi-automated dataset creation pipeline utilizing generative models to combine
  personalized concepts into complex compositions along with text-descriptions.
  Using this, we create a dataset called MyCanvas, that can be used to benchmark
  the task of multi-concept personalization. In addition, we design a
  comprehensive metric comprising two scores (CP-CLIP and TI-CLIP) for better
  quantifying the performance of multi-concept, personalized text-to-image
  diffusion methods. We provide a simple baseline built on top of Custom
  Diffusion with empirical prompting strategies for future researchers to
  evaluate on MyCanvas. We show that by improving data quality and prompting
  strategies, we can significantly increase multi-concept personalized image
  generation quality, without requiring any modifications to model architecture
  or training algorithms.
paper_ai_summary: Gen4Gen addresses data and metric challenges in multi-concept personalization for text-to-image diffusion models, enhancing image generation quality through improved data creation and empirical prompting strategies.
paper_ai_keywords: text-to-image diffusion models personalized concepts MyCanvas CP-CLIP TI-CLIP Custom Diffusion empirical prompting strategies multi-concept personalization