SILC: Improving Vision Language Pretraining with Self-Distillation
Image-Text pretraining on web-scale image caption dataset has become the
default recipe for open vocabulary classification and retrieval models thanks
to the success of CLIP and its variants. Several works have also used CLIP
features for dense prediction tasks and have shown the emergence of open-set
abilities. However, the contrastive objective only focuses on image-text
alignment and does not incentivise image feature learning for dense prediction
tasks. In this work, we propose the simple addition of local-to-global
correspondence learning by self-distillation as an additional objective for
contrastive pre-training to propose SILC. We show that distilling local image
features from an exponential moving average (EMA) teacher model significantly
improves model performance on several computer vision tasks including
classification, retrieval, and especially segmentation. We further show that
SILC scales better with the same training duration compared to the baselines.
Our model SILC sets a new state of the art for zero-shot classification, few
shot classification, image and text retrieval, zero-shot segmentation, and open
vocabulary segmentation.
6535f3cd33c5982a2977855c
Muhammad Ferjad Naeem
claimed_verified
2025-02-26T15:38:00.778Z
67bf33a512368ec2fad4fe29
/avatars/ea5c03744ec1c2bcc0e6c13efc8f7ddc.svg
Muhammad Ferjad Naeem
user
ferjad
6535f3cd33c5982a2977855d
Yongqin Xian
6535f3cd33c5982a2977855e
Xiaohua Zhai
6535f3cd33c5982a2977855f
Lukas Hoyer
6535f3cd33c5982a29778560
Luc Van Gool
6535f3cd33c5982a29778561
Federico Tombari
SILC: Improving Vision Language Pretraining with Self-Distillation
Image-Text pretraining on web-scale image caption dataset has become the
default recipe for open vocabulary classification and retrieval models thanks
to the success of CLIP and its variants. Several works have also used CLIP
features for dense prediction tasks and have shown the emergence of open-set
abilities. However, the contrastive objective only focuses on image-text
alignment and does not incentivise image feature learning for dense prediction
tasks. In this work, we propose the simple addition of local-to-global
correspondence learning by self-distillation as an additional objective for
contrastive pre-training to propose SILC. We show that distilling local image
features from an exponential moving average (EMA) teacher model significantly
improves model performance on several computer vision tasks including
classification, retrieval, and especially segmentation. We further show that
SILC scales better with the same training duration compared to the baselines.
Our model SILC sets a new state of the art for zero-shot classification, few
shot classification, image and text retrieval, zero-shot segmentation, and open
vocabulary segmentation.
Adding local-to-global correspondence learning via self-distillation to contrastive pre-training improves model performance across various computer vision tasks, particularly segmentation, and sets new benchmarks in zero-shot classification, few-shot classification, and retrieval.
contrastive objective local-to-global correspondence self-distillation exponential moving average (EMA) model performance computer vision tasks classification retrieval segmentation zero-shot classification few-shot classification zero-shot segmentation open vocabulary segmentation