title: CapsFusion: Rethinking Image-Text Data at Scale
summary: Large multimodal models demonstrate remarkable generalist ability to perform
  diverse multimodal tasks in a zero-shot manner. Large-scale web-based
  image-text pairs contribute fundamentally to this success, but suffer from
  excessive noise. Recent studies use alternative captions synthesized by
  captioning models and have achieved notable benchmark performance. However, our
  experiments reveal significant Scalability Deficiency and World Knowledge Loss
  issues in models trained with synthetic captions, which have been largely
  obscured by their initial benchmark success. Upon closer examination, we
  identify the root cause as the overly-simplified language structure and lack of
  knowledge details in existing synthetic captions. To provide higher-quality and
  more scalable multimodal pretraining data, we propose CapsFusion, an advanced
  framework that leverages large language models to consolidate and refine
  information from both web-based image-text pairs and synthetic captions.
  Extensive experiments show that CapsFusion captions exhibit remarkable
  all-round superiority over existing captions in terms of model performance
  (e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
  efficiency (requiring 11-16 times less computation than baselines), world
  knowledge depth, and scalability. These effectiveness, efficiency and
  scalability advantages position CapsFusion as a promising candidate for future
  scaling of LMM training.
paper_authors._id: 6541c65111e4836f27d46e88
paper_authors.name: Qiying Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T14:26:16.051Z
paper_authors._id: 6341375be5071b0c5cc946a3
paper_authors.avatarUrl: /avatars/c26a32b54a3f7731854059d247b5c523.svg
paper_authors.fullname: Qiying Yu
paper_authors.type: user
paper_authors.user: qiying
paper_authors._id: 6541c65111e4836f27d46e89
paper_authors.name: Quan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T14:57:06.789Z
paper_authors._id: 630d7a8f81ef9b1772b67f4c
paper_authors.avatarUrl: /avatars/00757abd6e548ccebb5bfb233be129a2.svg
paper_authors.fullname: Quan Sun
paper_authors.type: user
paper_authors.user: QuanSun
paper_authors._id: 6541c65111e4836f27d46e8a
paper_authors.name: Xiaosong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T14:55:59.849Z
paper_authors._id: 64e473f0b78bc92221ab1883
paper_authors.avatarUrl: /avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
paper_authors.fullname: Xiaosong Zhang
paper_authors.type: user
paper_authors.user: zhangxiaosong18
paper_authors._id: 6541c65111e4836f27d46e8b
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T14:56:06.913Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 6541c65111e4836f27d46e8c
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:23:09.979Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 6541c65111e4836f27d46e8d
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T14:56:41.820Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_authors._id: 6541c65111e4836f27d46e8e
paper_authors.name: Jingjing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T14:56:50.187Z
paper_authors._id: 650bf18678d2dfbad85c0db1
paper_authors.avatarUrl: /avatars/07ac9d726dc2ccf433746ee858a8044f.svg
paper_authors.fullname: jin liu
paper_authors.type: user
paper_authors.user: jingjingliu
paper_title: CapsFusion: Rethinking Image-Text Data at Scale
paper_summary: Large multimodal models demonstrate remarkable generalist ability to perform
  diverse multimodal tasks in a zero-shot manner. Large-scale web-based
  image-text pairs contribute fundamentally to this success, but suffer from
  excessive noise. Recent studies use alternative captions synthesized by
  captioning models and have achieved notable benchmark performance. However, our
  experiments reveal significant Scalability Deficiency and World Knowledge Loss
  issues in models trained with synthetic captions, which have been largely
  obscured by their initial benchmark success. Upon closer examination, we
  identify the root cause as the overly-simplified language structure and lack of
  knowledge details in existing synthetic captions. To provide higher-quality and
  more scalable multimodal pretraining data, we propose CapsFusion, an advanced
  framework that leverages large language models to consolidate and refine
  information from both web-based image-text pairs and synthetic captions.
  Extensive experiments show that CapsFusion captions exhibit remarkable
  all-round superiority over existing captions in terms of model performance
  (e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
  efficiency (requiring 11-16 times less computation than baselines), world
  knowledge depth, and scalability. These effectiveness, efficiency and
  scalability advantages position CapsFusion as a promising candidate for future
  scaling of LMM training.
paper_ai_summary: CapsFusion is an advanced framework that improves multimodal pretraining data by combining web-based image-text pairs and synthetic captions, leading to enhanced model performance, sample efficiency, and scalability.
paper_ai_keywords: multimodal models zero-shot image-text pairs captioning models Scalability Deficiency World Knowledge Loss large language models CapsFusion CIDEr score sample efficiency LMM training