title: Improving Text-to-Image Consistency via Automatic Prompt Optimization
summary: Impressive advances in text-to-image (T2I) generative models have yielded a
  plethora of high performing models which are able to generate aesthetically
  appealing, photorealistic images. Despite the progress, these models still
  struggle to produce images that are consistent with the input prompt,
  oftentimes failing to capture object quantities, relations and attributes
  properly. Existing solutions to improve prompt-image consistency suffer from
  the following challenges: (1) they oftentimes require model fine-tuning, (2)
  they only focus on nearby prompt samples, and (3) they are affected by
  unfavorable trade-offs among image quality, representation diversity, and
  prompt-image consistency. In this paper, we address these challenges and
  introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a
  large language model (LLM) to improve prompt-image consistency in T2I models.
  Our framework starts from a user prompt and iteratively generates revised
  prompts with the goal of maximizing a consistency score. Our extensive
  validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost
  the initial consistency score by up to 24.9% in terms of DSG score while
  preserving the FID and increasing the recall between generated and real data.
  Our work paves the way toward building more reliable and robust T2I systems by
  harnessing the power of LLMs.
paper_authors._id: 660394fdcabe90e0a5219cf7
paper_authors.name: Oscar Mañas
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-27T15:16:24.376Z
paper_authors._id: 6355d5c460c1b72f62693983
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6355d5c460c1b72f62693983/heg2B7OnZ9Yqs3j8g0uTW.jpeg
paper_authors.fullname: Oscar Mañas
paper_authors.type: user
paper_authors.user: oscmansan
paper_authors._id: 660394fdcabe90e0a5219cf8
paper_authors.name: Pietro Astolfi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:29:03.588Z
paper_authors._id: 646388dfd4d34b01f45d87c3
paper_authors.avatarUrl: /avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
paper_authors.fullname: Pietro Astolfi
paper_authors.type: user
paper_authors.user: pietroastolfi
paper_authors._id: 660394fdcabe90e0a5219cf9
paper_authors.name: Melissa Hall
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:09:07.130Z
paper_authors._id: 6697bfb4acbced03ebd133f8
paper_authors.avatarUrl: /avatars/a7d41367c6f7974303feb5af2a387ce4.svg
paper_authors.fullname: Melissa Hall
paper_authors.type: user
paper_authors.user: melissah14
paper_authors._id: 660394fdcabe90e0a5219cfa
paper_authors.name: Candace Ross
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:29:13.341Z
paper_authors._id: 628709d57cc3b7bc1b95fcf0
paper_authors.avatarUrl: /avatars/98e0d9695527125a0409d1098cdceaa4.svg
paper_authors.fullname: Candace Ross
paper_authors.type: user
paper_authors.user: candaceross
paper_authors._id: 660394fdcabe90e0a5219cfb
paper_authors.name: Jack Urbanek
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:29:19.881Z
paper_authors._id: 631106397680dc699b1b1ca1
paper_authors.avatarUrl: /avatars/2c6f31b5328309cf4b42262bbb1c6372.svg
paper_authors.fullname: Jack Urbanek
paper_authors.type: user
paper_authors.user: JackUrb
paper_authors._id: 660394fdcabe90e0a5219cfc
paper_authors.name: Adina Williams
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:29:27.621Z
paper_authors._id: 6487618920fb2e386b73e5d2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6487618920fb2e386b73e5d2/ZnmBCV-gL3eMeDGC38Rz8.jpeg
paper_authors.fullname: Adina Williams
paper_authors.type: user
paper_authors.user: adinawilliams2
paper_authors._id: 660394fdcabe90e0a5219cfd
paper_authors.name: Aishwarya Agrawal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:29:49.127Z
paper_authors._id: 644fff2d17b6189cda58cb0e
paper_authors.avatarUrl: /avatars/03b367b9033c655129e48bd94bd245ee.svg
paper_authors.fullname: Aishwarya Agrawal
paper_authors.type: user
paper_authors.user: aagrawal
paper_authors._id: 660394fdcabe90e0a5219cfe
paper_authors.name: Adriana Romero-Soriano
paper_authors._id: 660394fdcabe90e0a5219cff
paper_authors.name: Michal Drozdzal
paper_title: Improving Text-to-Image Consistency via Automatic Prompt Optimization
paper_summary: Impressive advances in text-to-image (T2I) generative models have yielded a
  plethora of high performing models which are able to generate aesthetically
  appealing, photorealistic images. Despite the progress, these models still
  struggle to produce images that are consistent with the input prompt,
  oftentimes failing to capture object quantities, relations and attributes
  properly. Existing solutions to improve prompt-image consistency suffer from
  the following challenges: (1) they oftentimes require model fine-tuning, (2)
  they only focus on nearby prompt samples, and (3) they are affected by
  unfavorable trade-offs among image quality, representation diversity, and
  prompt-image consistency. In this paper, we address these challenges and
  introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a
  large language model (LLM) to improve prompt-image consistency in T2I models.
  Our framework starts from a user prompt and iteratively generates revised
  prompts with the goal of maximizing a consistency score. Our extensive
  validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost
  the initial consistency score by up to 24.9% in terms of DSG score while
  preserving the FID and increasing the recall between generated and real data.
  Our work paves the way toward building more reliable and robust T2I systems by
  harnessing the power of LLMs.
paper_ai_keywords: text-to-image (T2I) generative models image-to-image models photorealistic images prompt-image consistency object quantities object relations object attributes model fine-tuning prompt samples image quality representation diversity optimization-by-prompting framework OPT2I large language model (LLM) consistency score DSG score FID (Fréchet Inception Distance) recall between generated and real data