title: ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
summary: Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g.,
  DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional
  T2I benchmarks, at the cost of significant computational resources. The unCLIP
  stack comprises T2I prior and diffusion image decoder. The T2I prior model
  alone adds a billion parameters compared to the Latent Diffusion Models, which
  increases the computational and high-quality data requirements. We introduce
  ECLIPSE, a novel contrastive learning method that is both parameter and
  data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g.,
  CLIP) to distill the knowledge into the prior model. We demonstrate that the
  ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere
  2.8% of the data, surpasses the baseline T2I priors with an average of 71.6%
  preference score under resource-limited setting. It also attains performance on
  par with SOTA big models, achieving an average of 63.36% preference score in
  terms of the ability to follow the text compositions. Extensive experiments on
  two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE
  priors consistently deliver high performance while significantly reducing
  resource dependency.
paper_authors._id: 6576712c1345577b70071a54
paper_authors.name: Maitreya Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-11T08:15:44.751Z
paper_authors._id: 622d2ff38d04fd29a9ccf1a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/622d2ff38d04fd29a9ccf1a7/ORpGrlU8Lm_oSEVftcZEK.jpeg
paper_authors.fullname: Maitreya Patel
paper_authors.type: user
paper_authors.user: mpatel57
paper_authors._id: 6576712c1345577b70071a55
paper_authors.name: Changhoon Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:19:42.622Z
paper_authors._id: 634ad9bfd440b86afd4b045c
paper_authors.avatarUrl: /avatars/3cce546fc4ad6a9c0bcf891b7ab6530b.svg
paper_authors.fullname: Changhoon Kim
paper_authors.type: user
paper_authors.user: fltr
paper_authors._id: 6576712c1345577b70071a56
paper_authors.name: Sheng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-11T08:15:41.752Z
paper_authors._id: 63cca807336d9d56171486cf
paper_authors.avatarUrl: /avatars/6d8b8c26f7b04e7b4ac313d095ab9e3e.svg
paper_authors.fullname: Sheng Cheng
paper_authors.type: user
paper_authors.user: duoduo1133
paper_authors._id: 6576712c1345577b70071a57
paper_authors.name: Chitta Baral
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:19:49.555Z
paper_authors._id: 62687430dc80d589a9f6cd45
paper_authors.avatarUrl: /avatars/fe56b8c92657e4f210c86cc097d42813.svg
paper_authors.fullname: Chitta Baral
paper_authors.type: user
paper_authors.user: cogint-asu
paper_authors._id: 6576712c1345577b70071a58
paper_authors.name: Yezhou Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-12T08:09:17.694Z
paper_authors._id: 6486a850903d5878eb98110f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
paper_authors.fullname: 'YZ' Yezhou Yang
paper_authors.type: user
paper_authors.user: AiWithoutBorders
paper_title: ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
paper_summary: Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g.,
  DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional
  T2I benchmarks, at the cost of significant computational resources. The unCLIP
  stack comprises T2I prior and diffusion image decoder. The T2I prior model
  alone adds a billion parameters compared to the Latent Diffusion Models, which
  increases the computational and high-quality data requirements. We introduce
  ECLIPSE, a novel contrastive learning method that is both parameter and
  data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g.,
  CLIP) to distill the knowledge into the prior model. We demonstrate that the
  ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere
  2.8% of the data, surpasses the baseline T2I priors with an average of 71.6%
  preference score under resource-limited setting. It also attains performance on
  par with SOTA big models, achieving an average of 63.36% preference score in
  terms of the ability to follow the text compositions. Extensive experiments on
  two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE
  priors consistently deliver high performance while significantly reducing
  resource dependency.
paper_ai_summary: ECLIPSE is a parameter and data-efficient contrastive learning method for Text-to-Image models that outperforms larger models with fewer resources and maintains high preference scores.
paper_ai_keywords: T2I diffusion models unCLIP models DALL-E-2 T2I prior diffusion image decoder contrastive learning pre-trained vision-language models CLIP ECLIPSE Karlo Kandinsky preference score