title: Decoupled Global-Local Alignment for Improving Compositional
    Understanding
summary: Contrastive Language-Image Pre-training (CLIP) has achieved success on
  multiple downstream tasks by aligning image and text modalities. However, the
  nature of global contrastive learning limits CLIP's ability to comprehend
  compositional concepts, such as relations and attributes. Although recent
  studies employ global hard negative samples to improve compositional
  understanding, these methods significantly compromise the model's inherent
  general capabilities by forcibly distancing textual negative samples from
  images in the embedding space. To overcome this limitation, we introduce a
  Decoupled Global-Local Alignment (DeGLA) framework that improves compositional
  understanding while substantially mitigating losses in general capabilities. To
  optimize the retention of the model's inherent capabilities, we incorporate a
  self-distillation mechanism within the global alignment process, aligning the
  learnable image-text encoder with a frozen teacher model derived from an
  exponential moving average. Under the constraint of self-distillation, it
  effectively mitigates the catastrophic forgetting of pretrained knowledge
  during fine-tuning. To improve compositional understanding, we first leverage
  the in-context learning capability of Large Language Models (LLMs) to construct
  about 2M high-quality negative captions across five types. Subsequently, we
  propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC)
  loss to enhance vision-language compositionally. Extensive experimental results
  demonstrate the effectiveness of the DeGLA framework. Compared to previous
  state-of-the-art methods, DeGLA achieves an average enhancement of 3.5% across
  the VALSE, SugarCrepe, and ARO benchmarks. Concurrently, it obtains an average
  performance improvement of 13.0% on zero-shot classification tasks across
  eleven datasets. Our code will be released at
  https://github.com/xiaoxing2001/DeGLA
paper_authors._id: 6809bebd0f6dfd7bd5159b76
paper_authors.name: Xiaoxing Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:50:22.326Z
paper_authors._id: 67223563fa69c82e19d2232c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1z_axjIty3uB4UDYa9JK4.png
paper_authors.fullname: xiaoxinghhh
paper_authors.type: user
paper_authors.user: wsdwJohn1231
paper_authors._id: 6809bebd0f6dfd7bd5159b77
paper_authors.name: Kaicheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-24T09:10:22.302Z
paper_authors._id: 63e202f352b7578dba448ab5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: Kaichengalex
paper_authors._id: 6809bebd0f6dfd7bd5159b78
paper_authors.name: Jun Wang
paper_authors._id: 6809bebd0f6dfd7bd5159b79
paper_authors.name: Haoran Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-24T11:26:36.470Z
paper_authors._id: 61384b860317b0a5c10877d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631080954171-61384b860317b0a5c10877d3.jpeg
paper_authors.fullname: Haoran Xu
paper_authors.type: user
paper_authors.user: haoranxu
paper_authors._id: 6809bebd0f6dfd7bd5159b7a
paper_authors.name: Ziyong Feng
paper_authors._id: 6809bebd0f6dfd7bd5159b7b
paper_authors.name: Yupei Wang
paper_title: Decoupled Global-Local Alignment for Improving Compositional
    Understanding
paper_summary: Contrastive Language-Image Pre-training (CLIP) has achieved success on
  multiple downstream tasks by aligning image and text modalities. However, the
  nature of global contrastive learning limits CLIP's ability to comprehend
  compositional concepts, such as relations and attributes. Although recent
  studies employ global hard negative samples to improve compositional
  understanding, these methods significantly compromise the model's inherent
  general capabilities by forcibly distancing textual negative samples from
  images in the embedding space. To overcome this limitation, we introduce a
  Decoupled Global-Local Alignment (DeGLA) framework that improves compositional
  understanding while substantially mitigating losses in general capabilities. To
  optimize the retention of the model's inherent capabilities, we incorporate a
  self-distillation mechanism within the global alignment process, aligning the
  learnable image-text encoder with a frozen teacher model derived from an
  exponential moving average. Under the constraint of self-distillation, it
  effectively mitigates the catastrophic forgetting of pretrained knowledge
  during fine-tuning. To improve compositional understanding, we first leverage
  the in-context learning capability of Large Language Models (LLMs) to construct
  about 2M high-quality negative captions across five types. Subsequently, we
  propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC)
  loss to enhance vision-language compositionally. Extensive experimental results
  demonstrate the effectiveness of the DeGLA framework. Compared to previous
  state-of-the-art methods, DeGLA achieves an average enhancement of 3.5% across
  the VALSE, SugarCrepe, and ARO benchmarks. Concurrently, it obtains an average
  performance improvement of 13.0% on zero-shot classification tasks across
  eleven datasets. Our code will be released at
  https://github.com/xiaoxing2001/DeGLA
paper_ai_summary: The DeGLA framework enhances compositional understanding in vision-language models through decoupled global-local alignment, self-distillation, and image-text grounded contrastive losses, improving performance on benchmarks and zero-shot classification tasks.
paper_ai_keywords: Contrastive Language-Image Pre-training (CLIP) global contrastive learning compositional concepts global hard negative samples Decoupled Global-Local Alignment (DeGLA) self-distillation frozen teacher model exponential moving average catastrophic forgetting Large Language Models (LLMs) negative captions Image-Grounded Contrast (IGC) loss Text-Grounded Contrast (TGC) loss VALSE SugarCrepe ARO benchmarks zero-shot classification tasks