title: GenHancer: Imperfect Generative Models are Secretly Strong
    Vision-Centric Enhancers
summary: The synergy between generative and discriminative models receives growing
  attention. While discriminative Contrastive Language-Image Pre-Training (CLIP)
  excels in high-level semantics, it struggles with perceiving fine-grained
  visual details. Generally, to enhance representations, generative models take
  CLIP's visual features as conditions for reconstruction. However, the
  underlying principle remains underexplored. In this work, we empirically found
  that visually perfect generations are not always optimal for representation
  enhancement. The essence lies in effectively extracting fine-grained knowledge
  from generative models while mitigating irrelevant information. To explore
  critical factors, we delve into three aspects: (1) Conditioning mechanisms: We
  found that even a small number of local tokens can drastically reduce the
  difficulty of reconstruction, leading to collapsed training. We thus conclude
  that utilizing only global visual tokens as conditions is the most effective
  strategy. (2) Denoising configurations: We observed that end-to-end training
  introduces extraneous information. To address this, we propose a two-stage
  training strategy to prioritize learning useful visual knowledge. Additionally,
  we demonstrate that lightweight denoisers can yield remarkable improvements.
  (3) Generation paradigms: We explore both continuous and discrete denoisers
  with desirable outcomes, validating the versatility of our method. Through our
  in-depth explorations, we have finally arrived at an effective method, namely
  GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark,
  e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into
  multimodal large language models for better vision-centric performance. All the
  models and codes are made publicly available.
paper_authors._id: 67e3693eebafaa1efbed08d2
paper_authors.name: Shijie Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:44:37.274Z
paper_authors._id: 67d30d9ae45dc43004b31425
paper_authors.avatarUrl: /avatars/714f4c88710ab4cc75dda49c76dd3b09.svg
paper_authors.fullname: Shijie Ma
paper_authors.type: user
paper_authors.user: msj9817
paper_authors._id: 67e3693eebafaa1efbed08d3
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:55:42.975Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 67e3693eebafaa1efbed08d4
paper_authors.name: Teng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T15:10:26.982Z
paper_authors._id: 63a9414e32ed73936ec0a0c8
paper_authors.avatarUrl: /avatars/f181e1bb480502c2680be5296a036bdd.svg
paper_authors.fullname: wybertwang
paper_authors.type: user
paper_authors.user: wybertwang
paper_authors._id: 67e3693eebafaa1efbed08d5
paper_authors.name: Yuxin Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:56:02.897Z
paper_authors._id: 67bc21106a3e748d80d11dc7
paper_authors.avatarUrl: /avatars/fbc7e76a3266a3c06d03e85db96a51cf.svg
paper_authors.fullname: yuxin guo
paper_authors.type: user
paper_authors.user: aether25
paper_authors._id: 67e3693eebafaa1efbed08d6
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:56:08.653Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 67e3693eebafaa1efbed08d7
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:56:14.275Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: GenHancer: Imperfect Generative Models are Secretly Strong
    Vision-Centric Enhancers
paper_summary: The synergy between generative and discriminative models receives growing
  attention. While discriminative Contrastive Language-Image Pre-Training (CLIP)
  excels in high-level semantics, it struggles with perceiving fine-grained
  visual details. Generally, to enhance representations, generative models take
  CLIP's visual features as conditions for reconstruction. However, the
  underlying principle remains underexplored. In this work, we empirically found
  that visually perfect generations are not always optimal for representation
  enhancement. The essence lies in effectively extracting fine-grained knowledge
  from generative models while mitigating irrelevant information. To explore
  critical factors, we delve into three aspects: (1) Conditioning mechanisms: We
  found that even a small number of local tokens can drastically reduce the
  difficulty of reconstruction, leading to collapsed training. We thus conclude
  that utilizing only global visual tokens as conditions is the most effective
  strategy. (2) Denoising configurations: We observed that end-to-end training
  introduces extraneous information. To address this, we propose a two-stage
  training strategy to prioritize learning useful visual knowledge. Additionally,
  we demonstrate that lightweight denoisers can yield remarkable improvements.
  (3) Generation paradigms: We explore both continuous and discrete denoisers
  with desirable outcomes, validating the versatility of our method. Through our
  in-depth explorations, we have finally arrived at an effective method, namely
  GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark,
  e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into
  multimodal large language models for better vision-centric performance. All the
  models and codes are made publicly available.
paper_ai_summary: GenHancer enhances CLIP by improving its fine-grained visual knowledge extraction through careful conditioning, denoising strategies, and generation paradigms, leading to better performance on multimodal tasks.
paper_ai_keywords: Contrastive Language-Image Pre-Training CLIP fine-grained visual details reconstruction global visual tokens end-to-end training two-stage training lightweight denoisers continuous denoisers discrete denoisers GenHancer MMVP-VLM benchmark