title: Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving
    Vision-Linguistic Compositionality
summary: In this paper, we propose a new method to enhance compositional understanding
  in pre-trained vision and language models (VLMs) without sacrificing
  performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches
  often improve compositional reasoning at the cost of degrading multi-modal
  capabilities, primarily due to the use of global hard negative (HN) loss, which
  contrasts global representations of images and texts. This global HN loss
  pushes HN texts that are highly similar to the original ones, damaging the
  model's multi-modal representations. To overcome this limitation, we propose
  Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard
  negative loss and selective calibrated regularization. These innovations
  provide fine-grained negative supervision while preserving the model's
  representational integrity. Our extensive evaluations across diverse benchmarks
  for both compositionality and multi-modal tasks show that FSC-CLIP not only
  achieves compositionality on par with state-of-the-art models but also retains
  strong multi-modal capabilities. Code is available at:
  https://github.com/ytaek-oh/fsc-clip.
paper_authors._id: 67073256584fdadcc36c7071
paper_authors.name: Youngtaek Oh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:01:47.459Z
paper_authors._id: 658cedf163d9c84928e6d295
paper_authors.avatarUrl: /avatars/d03c3fb194da25263789f8ebbbbf4b59.svg
paper_authors.fullname: Youngtaek Oh
paper_authors.type: user
paper_authors.user: ytaek-oh
paper_authors._id: 67073256584fdadcc36c7072
paper_authors.name: Jae Won Cho
paper_authors._id: 67073256584fdadcc36c7073
paper_authors.name: Dong-Jin Kim
paper_authors._id: 67073256584fdadcc36c7074
paper_authors.name: In So Kweon
paper_authors._id: 67073256584fdadcc36c7075
paper_authors.name: Junmo Kim
paper_title: Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving
    Vision-Linguistic Compositionality
paper_summary: In this paper, we propose a new method to enhance compositional understanding
  in pre-trained vision and language models (VLMs) without sacrificing
  performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches
  often improve compositional reasoning at the cost of degrading multi-modal
  capabilities, primarily due to the use of global hard negative (HN) loss, which
  contrasts global representations of images and texts. This global HN loss
  pushes HN texts that are highly similar to the original ones, damaging the
  model's multi-modal representations. To overcome this limitation, we propose
  Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard
  negative loss and selective calibrated regularization. These innovations
  provide fine-grained negative supervision while preserving the model's
  representational integrity. Our extensive evaluations across diverse benchmarks
  for both compositionality and multi-modal tasks show that FSC-CLIP not only
  achieves compositionality on par with state-of-the-art models but also retains
  strong multi-modal capabilities. Code is available at:
  https://github.com/ytaek-oh/fsc-clip.
paper_ai_keywords: Feder method compositional understanding pre-trained vision and language models (VLMs) zero-shot multi-modal tasks traditional fine-tuning approaches compositional reasoning multi-modal capabilities global hard negative (HN) loss global representations local hard negative loss selective calibrated regularization representation integrity compositionality multi-modal tasks