title: BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
summary: We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that
  bridges the gap between descriptive synthetic captions and factual web-scale
  alt-text. KALE augments synthetic dense image captions with web-scale alt-text
  to generate factually grounded image captions. Our two-stage approach leverages
  large vision-language models and language models to create knowledge-augmented
  captions, which are then used to train a specialized VLM for scaling up the
  dataset. We train vision-language models on KALE and demonstrate improvements
  on vision-language tasks. Our experiments show the utility of KALE for training
  more capable and knowledgeable multimodal models. We release the KALE dataset
  at https://huggingface.co/datasets/Salesforce/blip3-kale
paper_authors._id: 6734bf91fa1009ea2b5a1534
paper_authors.name: Anas Awadalla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:22:30.456Z
paper_authors._id: 60f1ab3afe0d78e01037eeb1
paper_authors.avatarUrl: /avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
paper_authors.fullname: Anas Awadalla
paper_authors.type: user
paper_authors.user: anas-awadalla
paper_authors._id: 6734bf91fa1009ea2b5a1535
paper_authors.name: Le Xue
paper_authors._id: 6734bf91fa1009ea2b5a1536
paper_authors.name: Manli Shu
paper_authors._id: 6734bf91fa1009ea2b5a1537
paper_authors.name: An Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:22:28.494Z
paper_authors._id: 634dfb9bbe5a827d48749f39
paper_authors.avatarUrl: /avatars/a8df0eeb1df39f3935dd686a94768f30.svg
paper_authors.fullname: An Yan
paper_authors.type: user
paper_authors.user: zzxslp
paper_authors._id: 6734bf91fa1009ea2b5a1538
paper_authors.name: Jun Wang
paper_authors._id: 6734bf91fa1009ea2b5a1539
paper_authors.name: Senthil Purushwalkam
paper_authors._id: 6734bf91fa1009ea2b5a153a
paper_authors.name: Sheng Shen
paper_authors._id: 6734bf91fa1009ea2b5a153b
paper_authors.name: Hannah Lee
paper_authors._id: 6734bf91fa1009ea2b5a153c
paper_authors.name: Oscar Lo
paper_authors._id: 6734bf91fa1009ea2b5a153d
paper_authors.name: Jae Sung Park
paper_authors._id: 6734bf91fa1009ea2b5a153e
paper_authors.name: Etash Guha
paper_authors._id: 6734bf91fa1009ea2b5a153f
paper_authors.name: Silvio Savarese
paper_authors._id: 6734bf91fa1009ea2b5a1540
paper_authors.name: Ludwig Schmidt
paper_authors._id: 6734bf91fa1009ea2b5a1541
paper_authors.name: Yejin Choi
paper_authors._id: 6734bf91fa1009ea2b5a1542
paper_authors.name: Caiming Xiong
paper_authors._id: 6734bf91fa1009ea2b5a1543
paper_authors.name: Ran Xu
paper_title: BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
paper_summary: We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that
  bridges the gap between descriptive synthetic captions and factual web-scale
  alt-text. KALE augments synthetic dense image captions with web-scale alt-text
  to generate factually grounded image captions. Our two-stage approach leverages
  large vision-language models and language models to create knowledge-augmented
  captions, which are then used to train a specialized VLM for scaling up the
  dataset. We train vision-language models on KALE and demonstrate improvements
  on vision-language tasks. Our experiments show the utility of KALE for training
  more capable and knowledgeable multimodal models. We release the KALE dataset
  at https://huggingface.co/datasets/Salesforce/blip3-kale
paper_ai_keywords: vision-language models image-text pairs alt-text dense image captions factually grounded image captions two-stage approach knowledge-augmented captions specialized VLM multimodal models