Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models
<|reference_start|>Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models: Modern retrieval systems often struggle with upgrading to new and more powerful models due to the incompatibility of embeddings between the old and new models. This necessitates a costly process known as backfilling, which involves re-computing the embeddings for a large number of data samples. In vision, Backward-compatible Training (BT) has been proposed to ensure that the new model aligns with the old model's embeddings. This paper extends the concept of vision-only BT to the field of cross-modal retrieval, marking the first attempt to address Cross-modal BT (XBT). Our goal is to achieve backward-compatibility between Vision-Language Pretraining (VLP) models, such as CLIP, for the cross-modal retrieval task. To address XBT challenges, we propose an efficient solution: a projection module that maps the new model's embeddings to those of the old model. This module, pretrained solely with text data, significantly reduces the number of image-text pairs required for XBT learning, and, once it is pretrained, it avoids using the old model during training. Furthermore, we utilize parameter-efficient training strategies that improve efficiency and preserve the off-the-shelf new model's knowledge by avoiding any modifications. Experimental results on cross-modal retrieval datasets demonstrate the effectiveness of XBT and its potential to enable backfill-free upgrades when a new VLP model emerges.<|reference_end|>
@article{jang2024towards,
    title={Towards Cross-modal Backward-compatible Representation Learning for
  Vision-Language Models},
    author={Young Kyun Jang, Ser-nam Lim},
    journal={arXiv preprint arXiv:2405.14715},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14715},
    primaryClass={cs.CV cs.AI}
}