Improving Text-to-Image Consistency via Automatic Prompt Optimization
Impressive advances in text-to-image (T2I) generative models have yielded a
plethora of high performing models which are able to generate aesthetically
appealing, photorealistic images. Despite the progress, these models still
struggle to produce images that are consistent with the input prompt,
oftentimes failing to capture object quantities, relations and attributes
properly. Existing solutions to improve prompt-image consistency suffer from
the following challenges: (1) they oftentimes require model fine-tuning, (2)
they only focus on nearby prompt samples, and (3) they are affected by
unfavorable trade-offs among image quality, representation diversity, and
prompt-image consistency. In this paper, we address these challenges and
introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a
large language model (LLM) to improve prompt-image consistency in T2I models.
Our framework starts from a user prompt and iteratively generates revised
prompts with the goal of maximizing a consistency score. Our extensive
validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost
the initial consistency score by up to 24.9% in terms of DSG score while
preserving the FID and increasing the recall between generated and real data.
Our work paves the way toward building more reliable and robust T2I systems by
harnessing the power of LLMs.
660394fdcabe90e0a5219cf7
Oscar Mañas
extracted_confirmed
2024-03-27T15:16:24.376Z
6355d5c460c1b72f62693983
https://cdn-avatars.huggingface.co/v1/production/uploads/6355d5c460c1b72f62693983/RA4FJCgPyqMXgTaSvBUGq.jpeg
Oscar Mañas
user
oscmansan
660394fdcabe90e0a5219cf8
Pietro Astolfi
admin_assigned
2024-03-27T09:29:03.588Z
646388dfd4d34b01f45d87c3
/avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
Pietro Astolfi
user
pietroastolfi
660394fdcabe90e0a5219cf9
Melissa Hall
claimed_verified
2024-07-18T09:09:07.130Z
6697bfb4acbced03ebd133f8
/avatars/a7d41367c6f7974303feb5af2a387ce4.svg
Melissa Hall
user
melissah14
660394fdcabe90e0a5219cfa
Candace Ross
admin_assigned
2024-03-27T09:29:13.341Z
628709d57cc3b7bc1b95fcf0
/avatars/98e0d9695527125a0409d1098cdceaa4.svg
Candace Ross
user
candaceross
660394fdcabe90e0a5219cfb
Jack Urbanek
admin_assigned
2024-03-27T09:29:19.881Z
631106397680dc699b1b1ca1
/avatars/2c6f31b5328309cf4b42262bbb1c6372.svg
Jack Urbanek
user
JackUrb
660394fdcabe90e0a5219cfc
Adina Williams
admin_assigned
2024-03-27T09:29:27.621Z
6487618920fb2e386b73e5d2
https://cdn-avatars.huggingface.co/v1/production/uploads/6487618920fb2e386b73e5d2/ZnmBCV-gL3eMeDGC38Rz8.jpeg
Adina Williams
user
adinawilliams2
660394fdcabe90e0a5219cfd
Aishwarya Agrawal
admin_assigned
2024-03-27T09:29:49.127Z
644fff2d17b6189cda58cb0e
/avatars/03b367b9033c655129e48bd94bd245ee.svg
Aishwarya Agrawal
user
aagrawal
660394fdcabe90e0a5219cfe
Adriana Romero-Soriano
660394fdcabe90e0a5219cff
Michal Drozdzal
Improving Text-to-Image Consistency via Automatic Prompt Optimization
Impressive advances in text-to-image (T2I) generative models have yielded a
plethora of high performing models which are able to generate aesthetically
appealing, photorealistic images. Despite the progress, these models still
struggle to produce images that are consistent with the input prompt,
oftentimes failing to capture object quantities, relations and attributes
properly. Existing solutions to improve prompt-image consistency suffer from
the following challenges: (1) they oftentimes require model fine-tuning, (2)
they only focus on nearby prompt samples, and (3) they are affected by
unfavorable trade-offs among image quality, representation diversity, and
prompt-image consistency. In this paper, we address these challenges and
introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a
large language model (LLM) to improve prompt-image consistency in T2I models.
Our framework starts from a user prompt and iteratively generates revised
prompts with the goal of maximizing a consistency score. Our extensive
validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost
the initial consistency score by up to 24.9% in terms of DSG score while
preserving the FID and increasing the recall between generated and real data.
Our work paves the way toward building more reliable and robust T2I systems by
harnessing the power of LLMs.
The OPT2I framework leverages a large language model to iteratively refine text-to-image prompts, enhancing consistency between prompts and generated images while maintaining image quality.
T2I large language model prompt-image consistency consistency score DSG score FID recall reliable T2I systems