IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning
<|reference_start|>IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning: Recent advancements in image captioning have explored text-only training methods to overcome the limitations of paired image-text data. However, existing text-only training methods often overlook the modality gap between using text data during training and employing images during inference. To address this issue, we propose a novel approach called Image-like Retrieval, which aligns text features with visually relevant features to mitigate the modality gap. Our method further enhances the accuracy of generated captions by designing a Fusion Module that integrates retrieved captions with input features. Additionally, we introduce a Frequency-based Entity Filtering technique that significantly improves caption quality. We integrate these methods into a unified framework, which we refer to as IFCap ($\textbf{I}$mage-like Retrieval and $\textbf{F}$requency-based Entity Filtering for Zero-shot $\textbf{Cap}$tioning). Through extensive experimentation, our straightforward yet powerful approach has demonstrated its efficacy, outperforming the state-of-the-art methods by a significant margin in both image captioning and video captioning compared to zero-shot captioning based on text-only training.<|reference_end|>
@article{lee2024ifcap:,
    title={IFCap: Image-like Retrieval and Frequency-based Entity Filtering for
  Zero-shot Captioning},
    author={Soeun Lee, Si-Woo Kim, Taewhan Kim, Dong-Jin Kim},
    journal={arXiv preprint arXiv:2409.18046},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.18046},
    primaryClass={cs.CV cs.AI cs.CL cs.LG}
}