title: Revisit Large-Scale Image-Caption Data in Pre-training Multimodal
    Foundation Models
summary: Recent advancements in multimodal models highlight the value of rewritten
  captions for improving performance, yet key challenges remain. For example,
  while synthetic captions often provide superior quality and image-text
  alignment, it is not clear whether they can fully replace AltTexts: the role of
  synthetic captions and their interaction with original web-crawled AltTexts in
  pre-training is still not well understood. Moreover, different multimodal
  foundation models may have unique preferences for specific caption formats, but
  efforts to identify the optimal captions for each model remain limited. In this
  work, we propose a novel, controllable, and scalable captioning pipeline
  designed to generate diverse caption formats tailored to various multimodal
  models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic
  Captions (DSC+) as case studies, we systematically explore their effects and
  interactions with AltTexts across models such as CLIP, multimodal LLMs, and
  diffusion models. Our findings reveal that a hybrid approach that keeps both
  synthetic captions and AltTexts can outperform the use of synthetic captions
  alone, improving both alignment and performance, with each model demonstrating
  preferences for particular caption formats. This comprehensive analysis
  provides valuable insights into optimizing captioning strategies, thereby
  advancing the pre-training of multimodal foundation models.
paper_authors._id: 66ff559f1e0b212adc0da561
paper_authors.name: Zhengfeng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:00.892Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 66ff559f1e0b212adc0da562
paper_authors.name: Vasileios Saveris
paper_authors._id: 66ff559f1e0b212adc0da563
paper_authors.name: Chen Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:23:46.017Z
paper_authors._id: 65409e53d07194a81ecdd427
paper_authors.avatarUrl: /avatars/73a88ffe7f78f05b6fc8eb57e9f1e587.svg
paper_authors.fullname: Chen Chen
paper_authors.type: user
paper_authors.user: ultra7chen
paper_authors._id: 66ff559f1e0b212adc0da564
paper_authors.name: Hong-You Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:04.373Z
paper_authors._id: 66fb920d63d612c629bfea36
paper_authors.avatarUrl: /avatars/86f5ccd4a15e8394792b422fefa64b69.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Hong-You
paper_authors._id: 66ff559f1e0b212adc0da565
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:02.875Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 66ff559f1e0b212adc0da566
paper_authors.name: Bowen Zhang
paper_authors._id: 66ff559f1e0b212adc0da567
paper_authors.name: Juan Lao Tebar
paper_authors._id: 66ff559f1e0b212adc0da568
paper_authors.name: Wenze Hu
paper_authors._id: 66ff559f1e0b212adc0da569
paper_authors.name: Zhe Gan
paper_authors._id: 66ff559f1e0b212adc0da56a
paper_authors.name: Peter Grasch
paper_authors._id: 66ff559f1e0b212adc0da56b
paper_authors.name: Meng Cao
paper_authors._id: 66ff559f1e0b212adc0da56c
paper_authors.name: Yinfei Yang
paper_title: Revisit Large-Scale Image-Caption Data in Pre-training Multimodal
    Foundation Models
paper_summary: Recent advancements in multimodal models highlight the value of rewritten
  captions for improving performance, yet key challenges remain. For example,
  while synthetic captions often provide superior quality and image-text
  alignment, it is not clear whether they can fully replace AltTexts: the role of
  synthetic captions and their interaction with original web-crawled AltTexts in
  pre-training is still not well understood. Moreover, different multimodal
  foundation models may have unique preferences for specific caption formats, but
  efforts to identify the optimal captions for each model remain limited. In this
  work, we propose a novel, controllable, and scalable captioning pipeline
  designed to generate diverse caption formats tailored to various multimodal
  models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic
  Captions (DSC+) as case studies, we systematically explore their effects and
  interactions with AltTexts across models such as CLIP, multimodal LLMs, and
  diffusion models. Our findings reveal that a hybrid approach that keeps both
  synthetic captions and AltTexts can outperform the use of synthetic captions
  alone, improving both alignment and performance, with each model demonstrating
  preferences for particular caption formats. This comprehensive analysis
  provides valuable insights into optimizing captioning strategies, thereby
  advancing the pre-training of multimodal foundation models.
paper_ai_summary: A study examines the use of synthetic and AltText captions in multimodal models, finding that a hybrid approach using both types improves alignment and performance across various models.
paper_ai_keywords: multimodal models Short Synthetic Captions (SSC) Dense Synthetic Captions (DSC+) CLIP multimodal LLMs diffusion models