Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
Existing automatic captioning methods for visual content face challenges such
as lack of detail, content hallucination, and poor instruction following. In
this work, we propose VisualFactChecker (VFC), a flexible training-free
pipeline that generates high-fidelity and detailed captions for both 2D images
and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text
captioning models propose multiple initial captions; 2) verification, where a
large language model (LLM) utilizes tools such as object detection and VQA
models to fact-check proposed captions; 3) captioning, where an LLM generates
the final caption by summarizing caption proposals and the fact check
verification results. In this step, VFC can flexibly generate captions in
various styles following complex instructions. We conduct comprehensive
captioning evaluations using four metrics: 1) CLIP-Score for image-text
similarity; 2) CLIP-Image-Score for measuring the image-image similarity
between the original and the reconstructed image generated by a text-to-image
model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V
for fine-grained evaluation. Evaluation results show that VFC outperforms
state-of-the-art open-sourced captioning methods for 2D images on the COCO
dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by
combining open-source models into a pipeline, we can attain captioning
capability comparable to proprietary models such as GPT-4V, despite being over
10x smaller in model size.
6631b6fd351231c42803242e
Yunhao Ge
claimed_verified
2024-05-20T07:39:04.409Z
6520e493b80dc49ba0f1e262
/avatars/af21c1ee154b2c9a0d56e69a07508ccb.svg
Yunhao Ge
user
yunhaog
6631b6fd351231c42803242f
Xiaohui Zeng
claimed_verified
2024-11-16T19:47:26.135Z
63363cb8a8048332fdc268cd
/avatars/a9bf6ce3394a8b76eade1a7517b37fa1.svg
xiaohui zeng
user
xiaohui2022
6631b6fd351231c428032430
Jacob Samuel Huffman
6631b6fd351231c428032431
Tsung-Yi Lin
admin_assigned
2024-05-02T08:21:48.636Z
63b738acbd2d1535227daa4c
/avatars/23300548e5c50f44e95d63568221f47b.svg
Tsung-Yi Lin
user
tsungyi
6631b6fd351231c428032432
Ming-Yu Liu
admin_assigned
2024-05-02T08:21:54.003Z
62f049afdf4b93aad5c7f2d6
/avatars/e272e58ad996733d7098e50248e5b57e.svg
Ming-Yu Liu
user
mingyuliutw
6631b6fd351231c428032433
Yin Cui
admin_assigned
2024-05-02T08:21:59.307Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
Existing automatic captioning methods for visual content face challenges such
as lack of detail, content hallucination, and poor instruction following. In
this work, we propose VisualFactChecker (VFC), a flexible training-free
pipeline that generates high-fidelity and detailed captions for both 2D images
and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text
captioning models propose multiple initial captions; 2) verification, where a
large language model (LLM) utilizes tools such as object detection and VQA
models to fact-check proposed captions; 3) captioning, where an LLM generates
the final caption by summarizing caption proposals and the fact check
verification results. In this step, VFC can flexibly generate captions in
various styles following complex instructions. We conduct comprehensive
captioning evaluations using four metrics: 1) CLIP-Score for image-text
similarity; 2) CLIP-Image-Score for measuring the image-image similarity
between the original and the reconstructed image generated by a text-to-image
model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V
for fine-grained evaluation. Evaluation results show that VFC outperforms
state-of-the-art open-sourced captioning methods for 2D images on the COCO
dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by
combining open-source models into a pipeline, we can attain captioning
capability comparable to proprietary models such as GPT-4V, despite being over
10x smaller in model size.
VisualFactChecker improves caption accuracy and detail through a three-step pipeline combining image-to-text models, fact-checking with large language models, and final summarization, outperforming existing methods on multiple datasets.
image-to-text captioning models large language model object detection VQA models image-text similarity CLIP-Score CLIP-Image-Score text-to-image model human study Amazon Mechanical Turk GPT-4V COCO dataset Objaverse dataset