title: A Picture is Worth a Thousand Words: Principled Recaptioning Improves
    Image Generation
summary: Text-to-image diffusion models achieved a remarkable leap in capabilities
  over the last few years, enabling high-quality and diverse synthesis of images
  from a textual prompt. However, even the most advanced models often struggle to
  precisely follow all of the directions in their prompts. The vast majority of
  these models are trained on datasets consisting of (image, caption) pairs where
  the images often come from the web, and the captions are their HTML alternate
  text. A notable example is the LAION dataset, used by Stable Diffusion and
  other models. In this work we observe that these captions are often of low
  quality, and argue that this significantly affects the model's capability to
  understand nuanced semantics in the textual prompts. We show that by relabeling
  the corpus with a specialized automatic captioning model and training a
  text-to-image model on the recaptioned dataset, the model benefits
  substantially across the board. First, in overall image quality: e.g. FID 14.84
  vs. the baseline of 17.87, and 64.3% improvement in faithful image generation
  according to human evaluation. Second, in semantic alignment, e.g. semantic
  object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and
  positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the
  corpus and provide evidence that this technique, which we call RECAP, both
  reduces the train-inference discrepancy and provides the model with more
  information per example, increasing sample efficiency and allowing the model to
  better understand the relations between captions and images.
paper_authors._id: 6539d9cff2ef6c212318b9c0
paper_authors.name: Eyal Segalis
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-26T18:15:18.583Z
paper_authors._id: 653aac6a5750f0f7cabb23fb
paper_authors.avatarUrl: /avatars/f001fc0846452e5b393956b533b8a7c6.svg
paper_authors.fullname: Eyal Segalis
paper_authors.type: user
paper_authors.user: eyalis50
paper_authors._id: 6539d9cff2ef6c212318b9c1
paper_authors.name: Dani Valevski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T07:52:44.313Z
paper_authors._id: 632b5adf6110e37dba3e3ca3
paper_authors.avatarUrl: /avatars/2c59a399d9b5d01d116fcb7775761fc8.svg
paper_authors.fullname: Dani Valevski
paper_authors.type: user
paper_authors.user: daniva
paper_authors._id: 6539d9cff2ef6c212318b9c2
paper_authors.name: Danny Lumen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-26T03:15:29.038Z
paper_authors._id: 6488751b7e6b887ab77ce30d
paper_authors.avatarUrl: /avatars/7fb9ef379f22d0190daab1e5ac548bc4.svg
paper_authors.fullname: Danny Lumen
paper_authors.type: user
paper_authors.user: DannyWasserman
paper_authors._id: 6539d9cff2ef6c212318b9c3
paper_authors.name: Yossi Matias
paper_authors._id: 6539d9cff2ef6c212318b9c4
paper_authors.name: Yaniv Leviathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T07:52:59.535Z
paper_authors._id: 6394a2244db1a9bedd1f2229
paper_authors.avatarUrl: /avatars/f04496811eb6873781e2ba606a2f97f0.svg
paper_authors.fullname: Yaniv Leviathan
paper_authors.type: user
paper_authors.user: yanivle
paper_title: A Picture is Worth a Thousand Words: Principled Recaptioning Improves
    Image Generation
paper_summary: Text-to-image diffusion models achieved a remarkable leap in capabilities
  over the last few years, enabling high-quality and diverse synthesis of images
  from a textual prompt. However, even the most advanced models often struggle to
  precisely follow all of the directions in their prompts. The vast majority of
  these models are trained on datasets consisting of (image, caption) pairs where
  the images often come from the web, and the captions are their HTML alternate
  text. A notable example is the LAION dataset, used by Stable Diffusion and
  other models. In this work we observe that these captions are often of low
  quality, and argue that this significantly affects the model's capability to
  understand nuanced semantics in the textual prompts. We show that by relabeling
  the corpus with a specialized automatic captioning model and training a
  text-to-image model on the recaptioned dataset, the model benefits
  substantially across the board. First, in overall image quality: e.g. FID 14.84
  vs. the baseline of 17.87, and 64.3% improvement in faithful image generation
  according to human evaluation. Second, in semantic alignment, e.g. semantic
  object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and
  positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the
  corpus and provide evidence that this technique, which we call RECAP, both
  reduces the train-inference discrepancy and provides the model with more
  information per example, increasing sample efficiency and allowing the model to
  better understand the relations between captions and images.
paper_ai_keywords: text-to-image diffusion models synthesis textual prompt LAION dataset Stable Diffusion automatic captioning model FID faithful image generation semantic object accuracy counting alignment errors positional alignment sample efficiency