title: un^2CLIP: Improving CLIP's Visual Detail Capturing Ability via
    Inverting unCLIP
summary: Contrastive Language-Image Pre-training (CLIP) has become a foundation model
  and has been applied to various vision and multimodal tasks. However, recent
  works indicate that CLIP falls short in distinguishing detailed differences in
  images and shows suboptimal performance on dense-prediction and vision-centric
  multimodal tasks. Therefore, this work focuses on improving existing CLIP
  models, aiming to capture as many visual details in images as possible. We find
  that a specific type of generative models, unCLIP, provides a suitable
  framework for achieving our goal. Specifically, unCLIP trains an image
  generator conditioned on the CLIP image embedding. In other words, it inverts
  the CLIP image encoder. Compared to discriminative models like CLIP, generative
  models are better at capturing image details because they are trained to learn
  the data distribution of images. Additionally, the conditional input space of
  unCLIP aligns with CLIP's original image-text embedding space. Therefore, we
  propose to invert unCLIP (dubbed un^2CLIP) to improve the CLIP model. In this
  way, the improved image encoder can gain unCLIP's visual detail capturing
  ability while preserving its alignment with the original text encoder
  simultaneously. We evaluate our improved CLIP across various tasks to which
  CLIP has been applied, including the challenging MMVP-VLM benchmark, the
  dense-prediction open-vocabulary segmentation task, and multimodal large
  language model tasks. Experiments show that un^2CLIP significantly improves
  the original CLIP and previous CLIP improvement methods. Code and models will
  be available at https://github.com/LiYinqi/un2CLIP.
paper_authors._id: 683d3f3100c71614babecb8c
paper_authors.name: Yinqi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:40:19.037Z
paper_authors._id: 64395702bb7ded0a0fee8889
paper_authors.avatarUrl: /avatars/afe8f9b6de358497b0db8a03f8a3a704.svg
paper_authors.fullname: Yinqi Li
paper_authors.type: user
paper_authors.user: yinqi
paper_authors._id: 683d3f3100c71614babecb8d
paper_authors.name: Jiahe Zhao
paper_authors._id: 683d3f3100c71614babecb8e
paper_authors.name: Hong Chang
paper_authors._id: 683d3f3100c71614babecb8f
paper_authors.name: Ruibing Hou
paper_authors._id: 683d3f3100c71614babecb90
paper_authors.name: Shiguang Shan
paper_authors._id: 683d3f3100c71614babecb91
paper_authors.name: Xilin Chen
paper_title: un^2CLIP: Improving CLIP's Visual Detail Capturing Ability via
    Inverting unCLIP
paper_summary: Contrastive Language-Image Pre-training (CLIP) has become a foundation model
  and has been applied to various vision and multimodal tasks. However, recent
  works indicate that CLIP falls short in distinguishing detailed differences in
  images and shows suboptimal performance on dense-prediction and vision-centric
  multimodal tasks. Therefore, this work focuses on improving existing CLIP
  models, aiming to capture as many visual details in images as possible. We find
  that a specific type of generative models, unCLIP, provides a suitable
  framework for achieving our goal. Specifically, unCLIP trains an image
  generator conditioned on the CLIP image embedding. In other words, it inverts
  the CLIP image encoder. Compared to discriminative models like CLIP, generative
  models are better at capturing image details because they are trained to learn
  the data distribution of images. Additionally, the conditional input space of
  unCLIP aligns with CLIP's original image-text embedding space. Therefore, we
  propose to invert unCLIP (dubbed un^2CLIP) to improve the CLIP model. In this
  way, the improved image encoder can gain unCLIP's visual detail capturing
  ability while preserving its alignment with the original text encoder
  simultaneously. We evaluate our improved CLIP across various tasks to which
  CLIP has been applied, including the challenging MMVP-VLM benchmark, the
  dense-prediction open-vocabulary segmentation task, and multimodal large
  language model tasks. Experiments show that un^2CLIP significantly improves
  the original CLIP and previous CLIP improvement methods. Code and models will
  be available at https://github.com/LiYinqi/un2CLIP.
paper_ai_summary: A generative model framework, unCLIP, is inverted to improve CLIP's ability to capture detailed visual information while maintaining text alignment.
paper_ai_keywords: Contrastive Language-Image Pre-training CLIP unCLIP image generator image encoding data distribution dense-prediction vision-centric multimodal open-vocabulary segmentation multimodal large language model MMVP-VLM benchmark