un^2CLIP: Improving CLIP's Visual Detail Capturing Ability via
  Inverting unCLIP
Contrastive Language-Image Pre-training (CLIP) has become a foundation model
and has been applied to various vision and multimodal tasks. However, recent
works indicate that CLIP falls short in distinguishing detailed differences in
images and shows suboptimal performance on dense-prediction and vision-centric
multimodal tasks. Therefore, this work focuses on improving existing CLIP
models, aiming to capture as many visual details in images as possible. We find
that a specific type of generative models, unCLIP, provides a suitable
framework for achieving our goal. Specifically, unCLIP trains an image
generator conditioned on the CLIP image embedding. In other words, it inverts
the CLIP image encoder. Compared to discriminative models like CLIP, generative
models are better at capturing image details because they are trained to learn
the data distribution of images. Additionally, the conditional input space of
unCLIP aligns with CLIP's original image-text embedding space. Therefore, we
propose to invert unCLIP (dubbed un^2CLIP) to improve the CLIP model. In this
way, the improved image encoder can gain unCLIP's visual detail capturing
ability while preserving its alignment with the original text encoder
simultaneously. We evaluate our improved CLIP across various tasks to which
CLIP has been applied, including the challenging MMVP-VLM benchmark, the
dense-prediction open-vocabulary segmentation task, and multimodal large
language model tasks. Experiments show that un^2CLIP significantly improves
the original CLIP and previous CLIP improvement methods. Code and models will
be available at https://github.com/LiYinqi/un2CLIP.
683d3f3100c71614babecb8c
Yinqi Li
claimed_verified
2025-06-02T07:40:19.037Z
64395702bb7ded0a0fee8889
/avatars/afe8f9b6de358497b0db8a03f8a3a704.svg
Yinqi Li
user
yinqi
683d3f3100c71614babecb8d
Jiahe Zhao
683d3f3100c71614babecb8e
Hong Chang
683d3f3100c71614babecb8f
Ruibing Hou
683d3f3100c71614babecb90
Shiguang Shan
683d3f3100c71614babecb91
Xilin Chen
un^2CLIP: Improving CLIP's Visual Detail Capturing Ability via
  Inverting unCLIP
Contrastive Language-Image Pre-training (CLIP) has become a foundation model
and has been applied to various vision and multimodal tasks. However, recent
works indicate that CLIP falls short in distinguishing detailed differences in
images and shows suboptimal performance on dense-prediction and vision-centric
multimodal tasks. Therefore, this work focuses on improving existing CLIP
models, aiming to capture as many visual details in images as possible. We find
that a specific type of generative models, unCLIP, provides a suitable
framework for achieving our goal. Specifically, unCLIP trains an image
generator conditioned on the CLIP image embedding. In other words, it inverts
the CLIP image encoder. Compared to discriminative models like CLIP, generative
models are better at capturing image details because they are trained to learn
the data distribution of images. Additionally, the conditional input space of
unCLIP aligns with CLIP's original image-text embedding space. Therefore, we
propose to invert unCLIP (dubbed un^2CLIP) to improve the CLIP model. In this
way, the improved image encoder can gain unCLIP's visual detail capturing
ability while preserving its alignment with the original text encoder
simultaneously. We evaluate our improved CLIP across various tasks to which
CLIP has been applied, including the challenging MMVP-VLM benchmark, the
dense-prediction open-vocabulary segmentation task, and multimodal large
language model tasks. Experiments show that un^2CLIP significantly improves
the original CLIP and previous CLIP improvement methods. Code and models will
be available at https://github.com/LiYinqi/un2CLIP.
A generative model framework, unCLIP, is inverted to improve CLIP's ability to capture detailed visual information while maintaining text alignment.
Contrastive Language-Image Pre-training CLIP unCLIP image generator image encoding data distribution dense-prediction vision-centric multimodal open-vocabulary segmentation multimodal large language model MMVP-VLM benchmark