title: Diffusion Feedback Helps CLIP See Better
summary: Contrastive Language-Image Pre-training (CLIP), which excels at abstracting
  open-world representations across domains and modalities, has become a
  foundation for a variety of vision and multimodal tasks. However, recent
  studies reveal that CLIP has severe visual shortcomings, such as which can
  hardly distinguish orientation, quantity, color, structure, etc. These visual
  shortcomings also limit the perception capabilities of multimodal large
  language models (MLLMs) built on CLIP. The main reason could be that the
  image-text pairs used to train CLIP are inherently biased, due to the lack of
  the distinctiveness of the text and the diversity of images. In this work, we
  present a simple post-training approach for CLIP models, which largely
  overcomes its visual shortcomings via a self-supervised diffusion process. We
  introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP.
  Specifically, DIVA leverages generative feedback from text-to-image diffusion
  models to optimize CLIP representations, with only images (without
  corresponding text). We demonstrate that DIVA improves CLIP's performance on
  the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities
  to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and
  vision models on multimodal understanding and segmentation tasks. Extensive
  evaluation on 29 image classification and retrieval benchmarks confirms that
  our framework preserves CLIP's strong zero-shot capabilities. The code will be
  available at https://github.com/baaivision/DIVA.
paper_authors._id: 66a85e795f395fc0c6c7afac
paper_authors.name: Wenxuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T09:51:07.296Z
paper_authors._id: 656428b5462e5ebcbf537d4e
paper_authors.avatarUrl: /avatars/cbedecc9c6f2afee2ca6b72efb593561.svg
paper_authors.fullname: Wenxuan Wang
paper_authors.type: user
paper_authors.user: Rookielion
paper_authors._id: 66a85e795f395fc0c6c7afad
paper_authors.name: Quan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:42:05.741Z
paper_authors._id: 630d7a8f81ef9b1772b67f4c
paper_authors.avatarUrl: /avatars/00757abd6e548ccebb5bfb233be129a2.svg
paper_authors.fullname: Quan Sun
paper_authors.type: user
paper_authors.user: QuanSun
paper_authors._id: 66a85e795f395fc0c6c7afae
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:22:37.841Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 66a85e795f395fc0c6c7afaf
paper_authors.name: Yepeng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:42:34.745Z
paper_authors._id: 64dee07b3d3a7519f18ebbe2
paper_authors.avatarUrl: /avatars/8f113938715b094435e014509630aa68.svg
paper_authors.fullname: tang
paper_authors.type: user
paper_authors.user: yepeng98
paper_authors._id: 66a85e795f395fc0c6c7afb0
paper_authors.name: Jing Liu
paper_authors._id: 66a85e795f395fc0c6c7afb1
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:43:46.103Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: Diffusion Feedback Helps CLIP See Better
paper_summary: Contrastive Language-Image Pre-training (CLIP), which excels at abstracting
  open-world representations across domains and modalities, has become a
  foundation for a variety of vision and multimodal tasks. However, recent
  studies reveal that CLIP has severe visual shortcomings, such as which can
  hardly distinguish orientation, quantity, color, structure, etc. These visual
  shortcomings also limit the perception capabilities of multimodal large
  language models (MLLMs) built on CLIP. The main reason could be that the
  image-text pairs used to train CLIP are inherently biased, due to the lack of
  the distinctiveness of the text and the diversity of images. In this work, we
  present a simple post-training approach for CLIP models, which largely
  overcomes its visual shortcomings via a self-supervised diffusion process. We
  introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP.
  Specifically, DIVA leverages generative feedback from text-to-image diffusion
  models to optimize CLIP representations, with only images (without
  corresponding text). We demonstrate that DIVA improves CLIP's performance on
  the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities
  to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and
  vision models on multimodal understanding and segmentation tasks. Extensive
  evaluation on 29 image classification and retrieval benchmarks confirms that
  our framework preserves CLIP's strong zero-shot capabilities. The code will be
  available at https://github.com/baaivision/DIVA.
paper_ai_keywords: contrastive pre-training CLIP open-world representations vision tasks multimodal tasks visual shortcomings multimodal large language models MLLMs image-text pairs self-supervised diffusion process DIVA DIffusion model text-to-image diffusion models MMVP-VLM benchmark fine-grained visual abilities image classification retrieval benchmarks zero-shot capabilities