title: ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free
    Domain Adaptation
summary: Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
  outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
  accuracy on ImageNet without seeing any example, which leads to potential
  benefits to many tasks that have no labeled data. However, while applying CLIP
  to a downstream target domain, the presence of visual and text domain gaps and
  cross-modality misalignment can greatly impact the model performance. To
  address such challenges, we propose ReCLIP, the first source-free domain
  adaptation method for vision-language models, which does not require any source
  data or target labeled data. ReCLIP first learns a projection space to mitigate
  the misaligned visual-text embeddings and learns pseudo labels, and then
  deploys cross-modality self-training with the pseudo labels, to update visual
  and text encoders, refine labels and reduce domain gaps and misalignments
  iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
  average error rate of CLIP from 30.17% to 25.06% on 22 image classification
  benchmarks.
paper_authors._id: 64d30aacaca4293a673dff47
paper_authors.name: Hu. Xuefeng
paper_authors._id: 64d30aacaca4293a673dff48
paper_authors.name: Zhang. Ke
paper_authors._id: 64d30aacaca4293a673dff49
paper_authors.name: Xia. Lu
paper_authors._id: 64d30aacaca4293a673dff4a
paper_authors.name: Chen. Albert
paper_authors._id: 64d30aacaca4293a673dff4b
paper_authors.name: Luo. Jiajia
paper_authors._id: 64d30aacaca4293a673dff4c
paper_authors.name: Sun. Yuyin
paper_authors._id: 64d30aacaca4293a673dff4d
paper_authors.name: Wang. Ken
paper_authors._id: 64d30aacaca4293a673dff4e
paper_authors.name: Qiao. Nan
paper_authors._id: 64d30aacaca4293a673dff4f
paper_authors.name: Zeng. Xiao
paper_authors._id: 64d30aacaca4293a673dff50
paper_authors.name: Sun. Min
paper_authors._id: 64d30aacaca4293a673dff51
paper_authors.name: Kuo. Cheng-Hao
paper_authors._id: 64d30aacaca4293a673dff52
paper_authors.name: Nevatia. Ram
paper_title: ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free
    Domain Adaptation
paper_summary: Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
  outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
  accuracy on ImageNet without seeing any example, which leads to potential
  benefits to many tasks that have no labeled data. However, while applying CLIP
  to a downstream target domain, the presence of visual and text domain gaps and
  cross-modality misalignment can greatly impact the model performance. To
  address such challenges, we propose ReCLIP, the first source-free domain
  adaptation method for vision-language models, which does not require any source
  data or target labeled data. ReCLIP first learns a projection space to mitigate
  the misaligned visual-text embeddings and learns pseudo labels, and then
  deploys cross-modality self-training with the pseudo labels, to update visual
  and text encoders, refine labels and reduce domain gaps and misalignments
  iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
  average error rate of CLIP from 30.17% to 25.06% on 22 image classification
  benchmarks.
paper_ai_keywords: CLIP zero-shot classification ImageNet ReCLIP source-free domain adaptation projection space visual-text embeddings pseudo labels cross-modality self-training visual encoders text encoders image classification benchmarks