A Picture is Worth a Thousand Words: Principled Recaptioning Improves
  Image Generation
Text-to-image diffusion models achieved a remarkable leap in capabilities
over the last few years, enabling high-quality and diverse synthesis of images
from a textual prompt. However, even the most advanced models often struggle to
precisely follow all of the directions in their prompts. The vast majority of
these models are trained on datasets consisting of (image, caption) pairs where
the images often come from the web, and the captions are their HTML alternate
text. A notable example is the LAION dataset, used by Stable Diffusion and
other models. In this work we observe that these captions are often of low
quality, and argue that this significantly affects the model's capability to
understand nuanced semantics in the textual prompts. We show that by relabeling
the corpus with a specialized automatic captioning model and training a
text-to-image model on the recaptioned dataset, the model benefits
substantially across the board. First, in overall image quality: e.g. FID 14.84
vs. the baseline of 17.87, and 64.3% improvement in faithful image generation
according to human evaluation. Second, in semantic alignment, e.g. semantic
object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and
positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the
corpus and provide evidence that this technique, which we call RECAP, both
reduces the train-inference discrepancy and provides the model with more
information per example, increasing sample efficiency and allowing the model to
better understand the relations between captions and images.
6539d9cff2ef6c212318b9c0
Eyal Segalis
extracted_confirmed
2023-10-26T18:15:18.583Z
653aac6a5750f0f7cabb23fb
/avatars/f001fc0846452e5b393956b533b8a7c6.svg
Eyal Segalis
user
eyalis50
6539d9cff2ef6c212318b9c1
Dani Valevski
admin_assigned
2023-10-26T07:52:44.313Z
632b5adf6110e37dba3e3ca3
/avatars/2c59a399d9b5d01d116fcb7775761fc8.svg
Dani Valevski
user
daniva
6539d9cff2ef6c212318b9c2
Danny Lumen
extracted_pending
2023-10-26T03:15:29.038Z
6488751b7e6b887ab77ce30d
/avatars/7fb9ef379f22d0190daab1e5ac548bc4.svg
Danny Lumen
user
DannyWasserman
6539d9cff2ef6c212318b9c3
Yossi Matias
6539d9cff2ef6c212318b9c4
Yaniv Leviathan
admin_assigned
2023-10-26T07:52:59.535Z
6394a2244db1a9bedd1f2229
/avatars/f04496811eb6873781e2ba606a2f97f0.svg
Yaniv Leviathan
user
yanivle
A Picture is Worth a Thousand Words: Principled Recaptioning Improves
  Image Generation
Text-to-image diffusion models achieved a remarkable leap in capabilities
over the last few years, enabling high-quality and diverse synthesis of images
from a textual prompt. However, even the most advanced models often struggle to
precisely follow all of the directions in their prompts. The vast majority of
these models are trained on datasets consisting of (image, caption) pairs where
the images often come from the web, and the captions are their HTML alternate
text. A notable example is the LAION dataset, used by Stable Diffusion and
other models. In this work we observe that these captions are often of low
quality, and argue that this significantly affects the model's capability to
understand nuanced semantics in the textual prompts. We show that by relabeling
the corpus with a specialized automatic captioning model and training a
text-to-image model on the recaptioned dataset, the model benefits
substantially across the board. First, in overall image quality: e.g. FID 14.84
vs. the baseline of 17.87, and 64.3% improvement in faithful image generation
according to human evaluation. Second, in semantic alignment, e.g. semantic
object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and
positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the
corpus and provide evidence that this technique, which we call RECAP, both
reduces the train-inference discrepancy and provides the model with more
information per example, increasing sample efficiency and allowing the model to
better understand the relations between captions and images.
Relabeling a text-to-image dataset with a specialized automatic captioning model improves image quality and semantic alignment in diffusion models.
text-to-image diffusion models LAION dataset Stable Diffusion FID faithful image generation semantic object accuracy counting alignment errors positional alignment train-inference discrepancy sample efficiency