λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion
  Models by Leveraging CLIP Latent Space
Despite the recent advances in personalized text-to-image (P-T2I) generative
models, subject-driven T2I remains challenging. The primary bottlenecks include
1) Intensive training resource requirements, 2) Hyper-parameter sensitivity
leading to inconsistent outputs, and 3) Balancing the intricacies of novel
visual concept and composition alignment. We start by re-iterating the core
philosophy of T2I diffusion models to address the above limitations.
Predominantly, contemporary subject-driven T2I approaches hinge on Latent
Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention
layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the
latent space of these diffusion models significantly escalates resource
demands, leading to inconsistent results and necessitating numerous iterations
for a single desired image. Recently, ECLIPSE has demonstrated a more
resource-efficient pathway for training UnCLIP-based T2I models, circumventing
the need for diffusion text-to-image priors. Building on this, we introduce
lambda-ECLIPSE. Our method illustrates that effective P-T2I does not
necessarily depend on the latent space of diffusion models. lambda-ECLIPSE
achieves single, multi-subject, and edge-guided T2I personalization with just
34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text
interleaved data. Through extensive experiments, we also establish that
lambda-ECLIPSE surpasses existing baselines in composition alignment while
preserving concept alignment performance, even with significantly lower
resource utilization.
65c59e6d73a47fa85dbf34fa
Maitreya Patel
claimed_verified
2024-02-09T08:05:02.871Z
622d2ff38d04fd29a9ccf1a7
https://cdn-avatars.huggingface.co/v1/production/uploads/622d2ff38d04fd29a9ccf1a7/ORpGrlU8Lm_oSEVftcZEK.jpeg
Maitreya Patel
user
mpatel57
65c59e6d73a47fa85dbf34fb
Sangmin Jung
admin_assigned
2024-02-09T09:03:11.118Z
650f3acee14eeb01d45353cd
/avatars/2f3dbc30318da10eb5c73f7cd3e8ee68.svg
Sangmin Jung
user
apexmin
65c59e6d73a47fa85dbf34fc
Chitta Baral
admin_assigned
2024-02-09T09:03:18.091Z
62687430dc80d589a9f6cd45
/avatars/fe56b8c92657e4f210c86cc097d42813.svg
Chitta Baral
user
cogint-asu
65c59e6d73a47fa85dbf34fd
Yezhou Yang
λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion
  Models by Leveraging CLIP Latent Space
Despite the recent advances in personalized text-to-image (P-T2I) generative
models, subject-driven T2I remains challenging. The primary bottlenecks include
1) Intensive training resource requirements, 2) Hyper-parameter sensitivity
leading to inconsistent outputs, and 3) Balancing the intricacies of novel
visual concept and composition alignment. We start by re-iterating the core
philosophy of T2I diffusion models to address the above limitations.
Predominantly, contemporary subject-driven T2I approaches hinge on Latent
Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention
layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the
latent space of these diffusion models significantly escalates resource
demands, leading to inconsistent results and necessitating numerous iterations
for a single desired image. Recently, ECLIPSE has demonstrated a more
resource-efficient pathway for training UnCLIP-based T2I models, circumventing
the need for diffusion text-to-image priors. Building on this, we introduce
lambda-ECLIPSE. Our method illustrates that effective P-T2I does not
necessarily depend on the latent space of diffusion models. lambda-ECLIPSE
achieves single, multi-subject, and edge-guided T2I personalization with just
34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text
interleaved data. Through extensive experiments, we also establish that
lambda-ECLIPSE surpasses existing baselines in composition alignment while
preserving concept alignment performance, even with significantly lower
resource utilization.
The new model $\lambda$-ECLIPSE achieves resource-efficient, high-quality personalized text-to-image generation with fewer parameters and training hours, outperforming existing methods in composition and concept alignment.
Latent Diffusion Models LDMs cross-attention layers UnCLIP-based T2I models $\lambda$-ECLIPSE single-subject multi-subject edge-guided T2I personalization composition alignment concept alignment