title: Getting it Right: Improving Spatial Consistency in Text-to-Image Models
summary: One of the key shortcomings in current text-to-image (T2I) models is their
  inability to consistently generate images which faithfully follow the spatial
  relationships specified in the text prompt. In this paper, we offer a
  comprehensive investigation of this limitation, while also developing datasets
  and methods that achieve state-of-the-art performance. First, we find that
  current vision-language datasets do not represent spatial relationships well
  enough; to alleviate this bottleneck, we create SPRIGHT, the first
  spatially-focused, large scale dataset, by re-captioning 6 million images from
  4 widely used vision datasets. Through a 3-fold evaluation and analysis
  pipeline, we find that SPRIGHT largely improves upon existing datasets in
  capturing spatial relationships. To demonstrate its efficacy, we leverage only
  ~0.25% of SPRIGHT and achieve a 22% improvement in generating spatially
  accurate images while also improving the FID and CMMD scores. Secondly, we find
  that training on images containing a large number of objects results in
  substantial improvements in spatial consistency. Notably, we attain
  state-of-the-art on T2I-CompBench with a spatial score of 0.2133, by
  fine-tuning on <500 images. Finally, through a set of controlled experiments
  and ablations, we document multiple findings that we believe will enhance the
  understanding of factors that affect spatial consistency in text-to-image
  models. We publicly release our dataset and model to foster further research in
  this area.
paper_authors._id: 660b8580cafce26b3c53f6f3
paper_authors.name: Agneet Chatterjee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-02T06:49:43.337Z
paper_authors._id: 6320c537a023aad6a7680c8b
paper_authors.avatarUrl: /avatars/057dc492b8f756b83f12ced0b74fae65.svg
paper_authors.fullname: Agneet Chatterjee
paper_authors.type: user
paper_authors.user: agneet
paper_authors._id: 660b8580cafce26b3c53f6f4
paper_authors.name: Gabriela Ben Melech Stan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:27:03.353Z
paper_authors._id: 63174cfddc97a974718f373a
paper_authors.avatarUrl: /avatars/f5a1d5d90abf8aafe8864bc4f1b076cc.svg
paper_authors.fullname: gabriela ben melech
paper_authors.type: user
paper_authors.user: gabsm
paper_authors._id: 660b8580cafce26b3c53f6f5
paper_authors.name: Estelle Aflalo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:55:08.001Z
paper_authors._id: 64575266900b28f7841090e5
paper_authors.avatarUrl: /avatars/13dc0a370bb24271653c352897a2673e.svg
paper_authors.fullname: Estelle Aflalo
paper_authors.type: user
paper_authors.user: estellea
paper_authors._id: 660b8580cafce26b3c53f6f6
paper_authors.name: Sayak Paul
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:02:54.848Z
paper_authors._id: 5f7fbd813e94f16a85448745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
paper_authors.fullname: Sayak Paul
paper_authors.type: user
paper_authors.user: sayakpaul
paper_authors._id: 660b8580cafce26b3c53f6f7
paper_authors.name: Dhruba Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T19:35:04.166Z
paper_authors._id: 638683b581fe8c678a39e80e
paper_authors.avatarUrl: /avatars/4124b185c1abc6dfd30e1b9b529941f8.svg
paper_authors.fullname: Dhruba Ghosh
paper_authors.type: user
paper_authors.user: djghosh
paper_authors._id: 660b8580cafce26b3c53f6f8
paper_authors.name: Tejas Gokhale
paper_authors._id: 660b8580cafce26b3c53f6f9
paper_authors.name: Ludwig Schmidt
paper_authors._id: 660b8580cafce26b3c53f6fa
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 660b8580cafce26b3c53f6fb
paper_authors.name: Vasudev Lal
paper_authors._id: 660b8580cafce26b3c53f6fc
paper_authors.name: Chitta Baral
paper_authors._id: 660b8580cafce26b3c53f6fd
paper_authors.name: Yezhou Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-04T07:49:51.954Z
paper_authors._id: 6486a850903d5878eb98110f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
paper_authors.fullname: 'YZ' Yezhou Yang
paper_authors.type: user
paper_authors.user: AiWithoutBorders
paper_title: Getting it Right: Improving Spatial Consistency in Text-to-Image Models
paper_summary: One of the key shortcomings in current text-to-image (T2I) models is their
  inability to consistently generate images which faithfully follow the spatial
  relationships specified in the text prompt. In this paper, we offer a
  comprehensive investigation of this limitation, while also developing datasets
  and methods that achieve state-of-the-art performance. First, we find that
  current vision-language datasets do not represent spatial relationships well
  enough; to alleviate this bottleneck, we create SPRIGHT, the first
  spatially-focused, large scale dataset, by re-captioning 6 million images from
  4 widely used vision datasets. Through a 3-fold evaluation and analysis
  pipeline, we find that SPRIGHT largely improves upon existing datasets in
  capturing spatial relationships. To demonstrate its efficacy, we leverage only
  ~0.25% of SPRIGHT and achieve a 22% improvement in generating spatially
  accurate images while also improving the FID and CMMD scores. Secondly, we find
  that training on images containing a large number of objects results in
  substantial improvements in spatial consistency. Notably, we attain
  state-of-the-art on T2I-CompBench with a spatial score of 0.2133, by
  fine-tuning on <500 images. Finally, through a set of controlled experiments
  and ablations, we document multiple findings that we believe will enhance the
  understanding of factors that affect spatial consistency in text-to-image
  models. We publicly release our dataset and model to foster further research in
  this area.
paper_ai_summary: The paper introduces SPRIGHT, a dataset focused on spatial relationships, and fine-tuning methods that significantly improve spatial consistency in text-to-image models.
paper_ai_keywords: text-to-image SPRIGHT spatial relationships vision-language datasets spatially-focused FID CMMD T2I-CompBench spatial score fine-tuning