title: SILC: Improving Vision Language Pretraining with Self-Distillation
summary: Image-Text pretraining on web-scale image caption dataset has become the
  default recipe for open vocabulary classification and retrieval models thanks
  to the success of CLIP and its variants. Several works have also used CLIP
  features for dense prediction tasks and have shown the emergence of open-set
  abilities. However, the contrastive objective only focuses on image-text
  alignment and does not incentivise image feature learning for dense prediction
  tasks. In this work, we propose the simple addition of local-to-global
  correspondence learning by self-distillation as an additional objective for
  contrastive pre-training to propose SILC. We show that distilling local image
  features from an exponential moving average (EMA) teacher model significantly
  improves model performance on several computer vision tasks including
  classification, retrieval, and especially segmentation. We further show that
  SILC scales better with the same training duration compared to the baselines.
  Our model SILC sets a new state of the art for zero-shot classification, few
  shot classification, image and text retrieval, zero-shot segmentation, and open
  vocabulary segmentation.
paper_authors._id: 6535f3cd33c5982a2977855c
paper_authors.name: Muhammad Ferjad Naeem
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T15:38:00.778Z
paper_authors._id: 67bf33a512368ec2fad4fe29
paper_authors.avatarUrl: /avatars/ea5c03744ec1c2bcc0e6c13efc8f7ddc.svg
paper_authors.fullname: Muhammad Ferjad Naeem
paper_authors.type: user
paper_authors.user: ferjad
paper_authors._id: 6535f3cd33c5982a2977855d
paper_authors.name: Yongqin Xian
paper_authors._id: 6535f3cd33c5982a2977855e
paper_authors.name: Xiaohua Zhai
paper_authors._id: 6535f3cd33c5982a2977855f
paper_authors.name: Lukas Hoyer
paper_authors._id: 6535f3cd33c5982a29778560
paper_authors.name: Luc Van Gool
paper_authors._id: 6535f3cd33c5982a29778561
paper_authors.name: Federico Tombari
paper_title: SILC: Improving Vision Language Pretraining with Self-Distillation
paper_summary: Image-Text pretraining on web-scale image caption dataset has become the
  default recipe for open vocabulary classification and retrieval models thanks
  to the success of CLIP and its variants. Several works have also used CLIP
  features for dense prediction tasks and have shown the emergence of open-set
  abilities. However, the contrastive objective only focuses on image-text
  alignment and does not incentivise image feature learning for dense prediction
  tasks. In this work, we propose the simple addition of local-to-global
  correspondence learning by self-distillation as an additional objective for
  contrastive pre-training to propose SILC. We show that distilling local image
  features from an exponential moving average (EMA) teacher model significantly
  improves model performance on several computer vision tasks including
  classification, retrieval, and especially segmentation. We further show that
  SILC scales better with the same training duration compared to the baselines.
  Our model SILC sets a new state of the art for zero-shot classification, few
  shot classification, image and text retrieval, zero-shot segmentation, and open
  vocabulary segmentation.
paper_ai_summary: Adding local-to-global correspondence learning via self-distillation to contrastive pre-training improves model performance across various computer vision tasks, particularly segmentation, and sets new benchmarks in zero-shot classification, few-shot classification, and retrieval.
paper_ai_keywords: contrastive objective local-to-global correspondence self-distillation exponential moving average (EMA) model performance computer vision tasks classification retrieval segmentation zero-shot classification few-shot classification zero-shot segmentation open vocabulary segmentation