RICO: Improving Accuracy and Completeness in Image Recaptioning via
  Visual Reconstruction
Image recaptioning is widely used to generate training datasets with enhanced
quality for various multimodal tasks. Existing recaptioning methods typically
rely on powerful multimodal large language models (MLLMs) to enhance textual
descriptions, but often suffer from inaccuracies due to hallucinations and
incompleteness caused by missing fine-grained details. To address these
limitations, we propose RICO, a novel framework that refines captions through
visual reconstruction. Specifically, we leverage a text-to-image model to
reconstruct a caption into a reference image, and prompt an MLLM to identify
discrepancies between the original and reconstructed images to refine the
caption. This process is performed iteratively, further progressively promoting
the generation of more faithful and comprehensive descriptions. To mitigate the
additional computational cost induced by the iterative process, we introduce
RICO-Flash, which learns to generate captions like RICO using DPO. Extensive
experiments demonstrate that our approach significantly improves caption
accuracy and completeness, outperforms most baselines by approximately 10% on
both CapsBench and CompreCap. Code released at
https://github.com/wangyuchi369/RICO.
6837d79d4d9866c160d8f43b
Yuchi Wang
6837d79d4d9866c160d8f43c
Yishuo Cai
6837d79d4d9866c160d8f43d
Shuhuai Ren
6837d79d4d9866c160d8f43e
Sihan Yang
6837d79d4d9866c160d8f43f
Linli Yao
6837d79d4d9866c160d8f440
Yuanxin Liu
6837d79d4d9866c160d8f441
Yuanxing Zhang
6837d79d4d9866c160d8f442
Pengfei Wan
6837d79d4d9866c160d8f443
Xu Sun
RICO: Improving Accuracy and Completeness in Image Recaptioning via
  Visual Reconstruction
Image recaptioning is widely used to generate training datasets with enhanced
quality for various multimodal tasks. Existing recaptioning methods typically
rely on powerful multimodal large language models (MLLMs) to enhance textual
descriptions, but often suffer from inaccuracies due to hallucinations and
incompleteness caused by missing fine-grained details. To address these
limitations, we propose RICO, a novel framework that refines captions through
visual reconstruction. Specifically, we leverage a text-to-image model to
reconstruct a caption into a reference image, and prompt an MLLM to identify
discrepancies between the original and reconstructed images to refine the
caption. This process is performed iteratively, further progressively promoting
the generation of more faithful and comprehensive descriptions. To mitigate the
additional computational cost induced by the iterative process, we introduce
RICO-Flash, which learns to generate captions like RICO using DPO. Extensive
experiments demonstrate that our approach significantly improves caption
accuracy and completeness, outperforms most baselines by approximately 10% on
both CapsBench and CompreCap. Code released at
https://github.com/wangyuchi369/RICO.
A novel iterative framework, RICO, improves image caption accuracy by using visual reconstruction and a text-to-image model to refine discrepancies, while RICO-Flash enhances efficiency using DPO.
multimodal large language models text-to-image model DPO CapsBench CompreCap