title: λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion
    Models by Leveraging CLIP Latent Space
summary: Despite the recent advances in personalized text-to-image (P-T2I) generative
  models, subject-driven T2I remains challenging. The primary bottlenecks include
  1) Intensive training resource requirements, 2) Hyper-parameter sensitivity
  leading to inconsistent outputs, and 3) Balancing the intricacies of novel
  visual concept and composition alignment. We start by re-iterating the core
  philosophy of T2I diffusion models to address the above limitations.
  Predominantly, contemporary subject-driven T2I approaches hinge on Latent
  Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention
  layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the
  latent space of these diffusion models significantly escalates resource
  demands, leading to inconsistent results and necessitating numerous iterations
  for a single desired image. Recently, ECLIPSE has demonstrated a more
  resource-efficient pathway for training UnCLIP-based T2I models, circumventing
  the need for diffusion text-to-image priors. Building on this, we introduce
  lambda-ECLIPSE. Our method illustrates that effective P-T2I does not
  necessarily depend on the latent space of diffusion models. lambda-ECLIPSE
  achieves single, multi-subject, and edge-guided T2I personalization with just
  34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text
  interleaved data. Through extensive experiments, we also establish that
  lambda-ECLIPSE surpasses existing baselines in composition alignment while
  preserving concept alignment performance, even with significantly lower
  resource utilization.
paper_authors._id: 65c59e6d73a47fa85dbf34fa
paper_authors.name: Maitreya Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:05:02.871Z
paper_authors._id: 622d2ff38d04fd29a9ccf1a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/622d2ff38d04fd29a9ccf1a7/ORpGrlU8Lm_oSEVftcZEK.jpeg
paper_authors.fullname: Maitreya Patel
paper_authors.type: user
paper_authors.user: mpatel57
paper_authors._id: 65c59e6d73a47fa85dbf34fb
paper_authors.name: Sangmin Jung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:03:11.118Z
paper_authors._id: 650f3acee14eeb01d45353cd
paper_authors.avatarUrl: /avatars/2f3dbc30318da10eb5c73f7cd3e8ee68.svg
paper_authors.fullname: Sangmin Jung
paper_authors.type: user
paper_authors.user: apexmin
paper_authors._id: 65c59e6d73a47fa85dbf34fc
paper_authors.name: Chitta Baral
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:03:18.091Z
paper_authors._id: 62687430dc80d589a9f6cd45
paper_authors.avatarUrl: /avatars/fe56b8c92657e4f210c86cc097d42813.svg
paper_authors.fullname: Chitta Baral
paper_authors.type: user
paper_authors.user: cogint-asu
paper_authors._id: 65c59e6d73a47fa85dbf34fd
paper_authors.name: Yezhou Yang
paper_title: λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion
    Models by Leveraging CLIP Latent Space
paper_summary: Despite the recent advances in personalized text-to-image (P-T2I) generative
  models, subject-driven T2I remains challenging. The primary bottlenecks include
  1) Intensive training resource requirements, 2) Hyper-parameter sensitivity
  leading to inconsistent outputs, and 3) Balancing the intricacies of novel
  visual concept and composition alignment. We start by re-iterating the core
  philosophy of T2I diffusion models to address the above limitations.
  Predominantly, contemporary subject-driven T2I approaches hinge on Latent
  Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention
  layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the
  latent space of these diffusion models significantly escalates resource
  demands, leading to inconsistent results and necessitating numerous iterations
  for a single desired image. Recently, ECLIPSE has demonstrated a more
  resource-efficient pathway for training UnCLIP-based T2I models, circumventing
  the need for diffusion text-to-image priors. Building on this, we introduce
  lambda-ECLIPSE. Our method illustrates that effective P-T2I does not
  necessarily depend on the latent space of diffusion models. lambda-ECLIPSE
  achieves single, multi-subject, and edge-guided T2I personalization with just
  34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text
  interleaved data. Through extensive experiments, we also establish that
  lambda-ECLIPSE surpasses existing baselines in composition alignment while
  preserving concept alignment performance, even with significantly lower
  resource utilization.
paper_ai_keywords: text-to-image (P-T2I) generative models adversarial adaptations hyper-parameter LDMs (Latent Diffusion Models) cross-attention layers diffusion models latent space ECLIPSE UnCLIP-based T2I $\lambda$-ECLIPSE concept alignment composition alignment