ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality
  Debiasing
This paper presents ScaleCap, an inference-time scalable image captioning
strategy that generates comprehensive and detailed image captions. The key
challenges of high-quality image captioning lie in the inherent biases of
LVLMs: multimodal bias resulting in imbalanced descriptive granularity,
offering detailed accounts of some elements while merely skimming over others;
linguistic bias leading to hallucinated descriptions of non-existent objects.
To address these issues, we propose a scalable debiased captioning strategy,
which continuously enriches and calibrates the caption with increased inference
budget. Specifically, we propose two novel components: heuristic question
answering and contrastive sentence rating. The former generates
content-specific questions based on the image and answers them to progressively
inject relevant information into the caption. The latter employs sentence-level
offline contrastive decoding to effectively identify and eliminate
hallucinations caused by linguistic biases. With increased inference cost, more
heuristic questions are raised by ScaleCap to progressively capture additional
visual details, generating captions that are more accurate, balanced, and
informative. Extensive modality alignment experiments demonstrate the
effectiveness of ScaleCap. Annotating 450K images with ScaleCap and using them
for LVLM pretraining leads to consistent performance gains across 11 widely
used benchmarks. Furthermore, ScaleCap showcases superb richness and fidelity
of generated captions with two additional tasks: replacing images with captions
in VQA task, and reconstructing images from captions to assess semantic
coverage. Code is available at https://github.com/Cooperx521/ScaleCap.
685b7cc2d2ee4fac76521e83
Long Xing
685b7cc2d2ee4fac76521e84
Qidong Huang
claimed_verified
2025-06-25T08:07:23.757Z
656f1b21b075b63c90ba02ee
/avatars/d6856815ef06261394178161e4d511b4.svg
Huang Qidong
user
shikiw
685b7cc2d2ee4fac76521e85
Xiaoyi Dong
685b7cc2d2ee4fac76521e86
Pan Zhang
685b7cc2d2ee4fac76521e87
Yuhang Zang
claimed_verified
2025-06-25T08:07:21.449Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
685b7cc2d2ee4fac76521e88
Yuhang Cao
685b7cc2d2ee4fac76521e89
Jinsong Li
685b7cc2d2ee4fac76521e8a
Shuangrui Ding
685b7cc2d2ee4fac76521e8b
Weiming Zhang
685b7cc2d2ee4fac76521e8c
Nenghai Yu
685b7cc2d2ee4fac76521e8d
Jiaqi Wang
685b7cc2d2ee4fac76521e8e
Feng Wu
685b7cc2d2ee4fac76521e8f
Dahua Lin
ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality
  Debiasing
This paper presents ScaleCap, an inference-time scalable image captioning
strategy that generates comprehensive and detailed image captions. The key
challenges of high-quality image captioning lie in the inherent biases of
LVLMs: multimodal bias resulting in imbalanced descriptive granularity,
offering detailed accounts of some elements while merely skimming over others;
linguistic bias leading to hallucinated descriptions of non-existent objects.
To address these issues, we propose a scalable debiased captioning strategy,
which continuously enriches and calibrates the caption with increased inference
budget. Specifically, we propose two novel components: heuristic question
answering and contrastive sentence rating. The former generates
content-specific questions based on the image and answers them to progressively
inject relevant information into the caption. The latter employs sentence-level
offline contrastive decoding to effectively identify and eliminate
hallucinations caused by linguistic biases. With increased inference cost, more
heuristic questions are raised by ScaleCap to progressively capture additional
visual details, generating captions that are more accurate, balanced, and
informative. Extensive modality alignment experiments demonstrate the
effectiveness of ScaleCap. Annotating 450K images with ScaleCap and using them
for LVLM pretraining leads to consistent performance gains across 11 widely
used benchmarks. Furthermore, ScaleCap showcases superb richness and fidelity
of generated captions with two additional tasks: replacing images with captions
in VQA task, and reconstructing images from captions to assess semantic
coverage. Code is available at https://github.com/Cooperx521/ScaleCap.
ScaleCap enhances image captioning by iteratively enriching and calibrating captions using heuristic question answering and contrastive sentence rating, addressing multimodal and linguistic biases to improve accuracy, balance, and informativeness.
LVLMs multimodal bias linguistic bias heuristic question answering contrastive sentence rating VQA task