title: What If We Recaption Billions of Web Images with LLaMA-3?
summary: Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate
  that semantically aligning and enriching textual descriptions of these pairs
  can significantly enhance model training across various vision-language tasks,
  particularly text-to-image generation. However, large-scale investigations in
  this area remain predominantly closed-source. Our paper aims to bridge this
  community effort, leveraging the powerful and open-sourced LLaMA-3, a
  GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a
  LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images
  from the DataComp-1B dataset. Our empirical results confirm that this enhanced
  dataset, Recap-DataComp-1B, offers substantial benefits in training advanced
  vision-language models. For discriminative models like CLIP, we observe
  enhanced zero-shot performance in cross-modal retrieval tasks. For generative
  models like text-to-image Diffusion Transformers, the generated images exhibit
  a significant improvement in alignment with users' text instructions,
  especially in following complex queries. Our project page is
  https://www.haqtu.me/Recap-Datacomp-1B/
paper_authors._id: 666a525aad291b6fa2567224
paper_authors.name: Xianhang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:23:55.019Z
paper_authors._id: 645e72d4b5c9a8666d0ddc1c
paper_authors.avatarUrl: /avatars/f0f1684731eb5ae148eadf841cceb18e.svg
paper_authors.fullname: xhl
paper_authors.type: user
paper_authors.user: Xianhang
paper_authors._id: 666a525aad291b6fa2567225
paper_authors.name: Haoqin Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:53:20.158Z
paper_authors._id: 604ae011caabafacfa48e3de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
paper_authors.fullname: Haoqin Tu
paper_authors.type: user
paper_authors.user: PahaII
paper_authors._id: 666a525aad291b6fa2567226
paper_authors.name: Mude Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:53:12.102Z
paper_authors._id: 659f6f0de06dc8fc0e411224
paper_authors.avatarUrl: /avatars/e16170f36b949c8697f277f6b52c51c3.svg
paper_authors.fullname: MudeHui
paper_authors.type: user
paper_authors.user: MudeHui
paper_authors._id: 666a525aad291b6fa2567227
paper_authors.name: Zeyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:23:52.589Z
paper_authors._id: 6462879777f63463fc5f75f9
paper_authors.avatarUrl: /avatars/f19d14a009e940e3b565a6c62251599b.svg
paper_authors.fullname: Zeyu Wang
paper_authors.type: user
paper_authors.user: zw123
paper_authors._id: 666a525aad291b6fa2567228
paper_authors.name: Bingchen Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:52:43.007Z
paper_authors._id: 62dcd71075e9787ec5aa41ba
paper_authors.avatarUrl: /avatars/f37ce036b76180ed0fa004f9c8c09363.svg
paper_authors.fullname: Bingchen Zhao
paper_authors.type: user
paper_authors.user: tennant
paper_authors._id: 666a525aad291b6fa2567229
paper_authors.name: Junfei Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:52:36.151Z
paper_authors._id: 64b5ba6060274cbb296d6288
paper_authors.avatarUrl: /avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
paper_authors.fullname: Junfei Xiao
paper_authors.type: user
paper_authors.user: lambertxiao
paper_authors._id: 666a525aad291b6fa256722a
paper_authors.name: Sucheng Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:18:12.526Z
paper_authors._id: 65317ea1501804124f011950
paper_authors.avatarUrl: /avatars/b055c3aba0c65d5377c69472e4576480.svg
paper_authors.fullname: Ren
paper_authors.type: user
paper_authors.user: OliverRen
paper_authors._id: 666a525aad291b6fa256722b
paper_authors.name: Jieru Mei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:52:22.423Z
paper_authors._id: 652809fa7ad7a3460213ac0f
paper_authors.avatarUrl: /avatars/5fc01ac91df2f03d213d05fa48109c67.svg
paper_authors.fullname: Jieru Mei
paper_authors.type: user
paper_authors.user: meijieru
paper_authors._id: 666a525aad291b6fa256722c
paper_authors.name: Qing Liu
paper_authors._id: 666a525aad291b6fa256722d
paper_authors.name: Huangjie Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:52:05.940Z
paper_authors._id: 62f6dc90d278a8f3e7877dc0
paper_authors.avatarUrl: /avatars/bce37487397278d5daf6d95f4dc25221.svg
paper_authors.fullname: Huangjie Zheng
paper_authors.type: user
paper_authors.user: hjzheng
paper_authors._id: 666a525aad291b6fa256722e
paper_authors.name: Yuyin Zhou
paper_authors._id: 666a525aad291b6fa256722f
paper_authors.name: Cihang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:51:47.209Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_title: What If We Recaption Billions of Web Images with LLaMA-3?
paper_summary: Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate
  that semantically aligning and enriching textual descriptions of these pairs
  can significantly enhance model training across various vision-language tasks,
  particularly text-to-image generation. However, large-scale investigations in
  this area remain predominantly closed-source. Our paper aims to bridge this
  community effort, leveraging the powerful and open-sourced LLaMA-3, a
  GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a
  LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images
  from the DataComp-1B dataset. Our empirical results confirm that this enhanced
  dataset, Recap-DataComp-1B, offers substantial benefits in training advanced
  vision-language models. For discriminative models like CLIP, we observe
  enhanced zero-shot performance in cross-modal retrieval tasks. For generative
  models like text-to-image Diffusion Transformers, the generated images exhibit
  a significant improvement in alignment with users' text instructions,
  especially in following complex queries. Our project page is
  https://www.haqtu.me/Recap-Datacomp-1B/
paper_ai_keywords: semantically aligning enriching vision-language tasks text-to-image generation open-sourced LLaMA-3 GPT-4 fine-tune LLaVA-1.5 recaptioning DataComp-1B Recap-DataComp-1B discriminative models CLIP zero-shot performance cross-modal retrieval tasks generative models text-to-image Diffusion Transformers complex queries