title: Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
summary: Existing automatic captioning methods for visual content face challenges such
  as lack of detail, content hallucination, and poor instruction following. In
  this work, we propose VisualFactChecker (VFC), a flexible training-free
  pipeline that generates high-fidelity and detailed captions for both 2D images
  and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text
  captioning models propose multiple initial captions; 2) verification, where a
  large language model (LLM) utilizes tools such as object detection and VQA
  models to fact-check proposed captions; 3) captioning, where an LLM generates
  the final caption by summarizing caption proposals and the fact check
  verification results. In this step, VFC can flexibly generate captions in
  various styles following complex instructions. We conduct comprehensive
  captioning evaluations using four metrics: 1) CLIP-Score for image-text
  similarity; 2) CLIP-Image-Score for measuring the image-image similarity
  between the original and the reconstructed image generated by a text-to-image
  model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V
  for fine-grained evaluation. Evaluation results show that VFC outperforms
  state-of-the-art open-sourced captioning methods for 2D images on the COCO
  dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by
  combining open-source models into a pipeline, we can attain captioning
  capability comparable to proprietary models such as GPT-4V, despite being over
  10x smaller in model size.
paper_authors._id: 6631b6fd351231c42803242e
paper_authors.name: Yunhao Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-20T07:39:04.409Z
paper_authors._id: 6520e493b80dc49ba0f1e262
paper_authors.avatarUrl: /avatars/af21c1ee154b2c9a0d56e69a07508ccb.svg
paper_authors.fullname: Yunhao Ge
paper_authors.type: user
paper_authors.user: yunhaog
paper_authors._id: 6631b6fd351231c42803242f
paper_authors.name: Xiaohui Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-16T19:47:26.135Z
paper_authors._id: 63363cb8a8048332fdc268cd
paper_authors.avatarUrl: /avatars/a9bf6ce3394a8b76eade1a7517b37fa1.svg
paper_authors.fullname: xiaohui zeng
paper_authors.type: user
paper_authors.user: xiaohui2022
paper_authors._id: 6631b6fd351231c428032430
paper_authors.name: Jacob Samuel Huffman
paper_authors._id: 6631b6fd351231c428032431
paper_authors.name: Tsung-Yi Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:21:48.636Z
paper_authors._id: 63b738acbd2d1535227daa4c
paper_authors.avatarUrl: /avatars/23300548e5c50f44e95d63568221f47b.svg
paper_authors.fullname: Tsung-Yi Lin
paper_authors.type: user
paper_authors.user: tsungyi
paper_authors._id: 6631b6fd351231c428032432
paper_authors.name: Ming-Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:21:54.003Z
paper_authors._id: 62f049afdf4b93aad5c7f2d6
paper_authors.avatarUrl: /avatars/e272e58ad996733d7098e50248e5b57e.svg
paper_authors.fullname: Ming-Yu Liu
paper_authors.type: user
paper_authors.user: mingyuliutw
paper_authors._id: 6631b6fd351231c428032433
paper_authors.name: Yin Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:21:59.307Z
paper_authors._id: 649f05367b57fab3a5b27c8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
paper_authors.fullname: Yin Cui
paper_authors.type: user
paper_authors.user: richardaecn
paper_title: Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
paper_summary: Existing automatic captioning methods for visual content face challenges such
  as lack of detail, content hallucination, and poor instruction following. In
  this work, we propose VisualFactChecker (VFC), a flexible training-free
  pipeline that generates high-fidelity and detailed captions for both 2D images
  and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text
  captioning models propose multiple initial captions; 2) verification, where a
  large language model (LLM) utilizes tools such as object detection and VQA
  models to fact-check proposed captions; 3) captioning, where an LLM generates
  the final caption by summarizing caption proposals and the fact check
  verification results. In this step, VFC can flexibly generate captions in
  various styles following complex instructions. We conduct comprehensive
  captioning evaluations using four metrics: 1) CLIP-Score for image-text
  similarity; 2) CLIP-Image-Score for measuring the image-image similarity
  between the original and the reconstructed image generated by a text-to-image
  model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V
  for fine-grained evaluation. Evaluation results show that VFC outperforms
  state-of-the-art open-sourced captioning methods for 2D images on the COCO
  dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by
  combining open-source models into a pipeline, we can attain captioning
  capability comparable to proprietary models such as GPT-4V, despite being over
  10x smaller in model size.
paper_ai_summary: VisualFactChecker improves caption accuracy and detail through a three-step pipeline combining image-to-text models, fact-checking with large language models, and final summarization, outperforming existing methods on multiple datasets.
paper_ai_keywords: image-to-text captioning models large language model object detection VQA models image-text similarity CLIP-Score CLIP-Image-Score text-to-image model human study Amazon Mechanical Turk GPT-4V COCO dataset Objaverse dataset