Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving
  Vision-Linguistic Compositionality
In this paper, we propose a new method to enhance compositional understanding
in pre-trained vision and language models (VLMs) without sacrificing
performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches
often improve compositional reasoning at the cost of degrading multi-modal
capabilities, primarily due to the use of global hard negative (HN) loss, which
contrasts global representations of images and texts. This global HN loss
pushes HN texts that are highly similar to the original ones, damaging the
model's multi-modal representations. To overcome this limitation, we propose
Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard
negative loss and selective calibrated regularization. These innovations
provide fine-grained negative supervision while preserving the model's
representational integrity. Our extensive evaluations across diverse benchmarks
for both compositionality and multi-modal tasks show that FSC-CLIP not only
achieves compositionality on par with state-of-the-art models but also retains
strong multi-modal capabilities. Code is available at:
https://github.com/ytaek-oh/fsc-clip.
67073256584fdadcc36c7071
Youngtaek Oh
claimed_verified
2024-10-10T08:01:47.459Z
658cedf163d9c84928e6d295
/avatars/d03c3fb194da25263789f8ebbbbf4b59.svg
Youngtaek Oh
user
ytaek-oh
67073256584fdadcc36c7072
Jae Won Cho
67073256584fdadcc36c7073
Dong-Jin Kim
67073256584fdadcc36c7074
In So Kweon
67073256584fdadcc36c7075
Junmo Kim
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving
  Vision-Linguistic Compositionality
In this paper, we propose a new method to enhance compositional understanding
in pre-trained vision and language models (VLMs) without sacrificing
performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches
often improve compositional reasoning at the cost of degrading multi-modal
capabilities, primarily due to the use of global hard negative (HN) loss, which
contrasts global representations of images and texts. This global HN loss
pushes HN texts that are highly similar to the original ones, damaging the
model's multi-modal representations. To overcome this limitation, we propose
Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard
negative loss and selective calibrated regularization. These innovations
provide fine-grained negative supervision while preserving the model's
representational integrity. Our extensive evaluations across diverse benchmarks
for both compositionality and multi-modal tasks show that FSC-CLIP not only
achieves compositionality on par with state-of-the-art models but also retains
strong multi-modal capabilities. Code is available at:
https://github.com/ytaek-oh/fsc-clip.
FSC-CLIP enhances compositional understanding in pre-trained vision and language models without degrading multi-modal performance, using local hard negative loss and selective calibrated regularization.
vision and language models (VLMs) global hard negative (HN) loss Fine-grained Selective Calibrated CLIP (FSC-CLIP) local hard negative loss selective calibrated regularization compositional understanding multi-modal tasks