BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues
<|reference_start|>BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues: Effectively aligning with human judgment when evaluating machine-generated image captions represents a complex yet intriguing challenge. Existing evaluation metrics like CIDEr or CLIP-Score fall short in this regard as they do not take into account the corresponding image or lack the capability of encoding fine-grained details and penalizing hallucinations. To overcome these issues, in this paper, we propose BRIDGE, a new learnable and reference-free image captioning metric that employs a novel module to map visual features into dense vectors and integrates them into multi-modal pseudo-captions which are built during the evaluation process. This approach results in a multimodal metric that properly incorporates information from the input image without relying on reference captions, bridging the gap between human judgment and machine-generated image captions. Experiments spanning several datasets demonstrate that our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores. Our source code and trained models are publicly available at: https://github.com/aimagelab/bridge-score.<|reference_end|>
@article{sarto2024bridge:,
    title={BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger
  Visual Cues},
    author={Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara},
    journal={arXiv preprint arXiv:2407.20341},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.20341},
    primaryClass={cs.CV cs.AI cs.CL cs.MM}
}