GenHancer: Imperfect Generative Models are Secretly Strong
  Vision-Centric Enhancers
The synergy between generative and discriminative models receives growing
attention. While discriminative Contrastive Language-Image Pre-Training (CLIP)
excels in high-level semantics, it struggles with perceiving fine-grained
visual details. Generally, to enhance representations, generative models take
CLIP's visual features as conditions for reconstruction. However, the
underlying principle remains underexplored. In this work, we empirically found
that visually perfect generations are not always optimal for representation
enhancement. The essence lies in effectively extracting fine-grained knowledge
from generative models while mitigating irrelevant information. To explore
critical factors, we delve into three aspects: (1) Conditioning mechanisms: We
found that even a small number of local tokens can drastically reduce the
difficulty of reconstruction, leading to collapsed training. We thus conclude
that utilizing only global visual tokens as conditions is the most effective
strategy. (2) Denoising configurations: We observed that end-to-end training
introduces extraneous information. To address this, we propose a two-stage
training strategy to prioritize learning useful visual knowledge. Additionally,
we demonstrate that lightweight denoisers can yield remarkable improvements.
(3) Generation paradigms: We explore both continuous and discrete denoisers
with desirable outcomes, validating the versatility of our method. Through our
in-depth explorations, we have finally arrived at an effective method, namely
GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark,
e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into
multimodal large language models for better vision-centric performance. All the
models and codes are made publicly available.
67e3693eebafaa1efbed08d2
Shijie Ma
claimed_verified
2025-03-26T20:44:37.274Z
67d30d9ae45dc43004b31425
/avatars/714f4c88710ab4cc75dda49c76dd3b09.svg
Shijie Ma
user
msj9817
67e3693eebafaa1efbed08d3
Yuying Ge
admin_assigned
2025-03-27T09:55:42.975Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
67e3693eebafaa1efbed08d4
Teng Wang
claimed_verified
2025-06-05T15:10:26.982Z
63a9414e32ed73936ec0a0c8
/avatars/f181e1bb480502c2680be5296a036bdd.svg
wybertwang
user
wybertwang
67e3693eebafaa1efbed08d5
Yuxin Guo
admin_assigned
2025-03-27T09:56:02.897Z
67bc21106a3e748d80d11dc7
/avatars/fbc7e76a3266a3c06d03e85db96a51cf.svg
yuxin guo
user
aether25
67e3693eebafaa1efbed08d6
Yixiao Ge
admin_assigned
2025-03-27T09:56:08.653Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
67e3693eebafaa1efbed08d7
Ying Shan
admin_assigned
2025-03-27T09:56:14.275Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
GenHancer: Imperfect Generative Models are Secretly Strong
  Vision-Centric Enhancers
The synergy between generative and discriminative models receives growing
attention. While discriminative Contrastive Language-Image Pre-Training (CLIP)
excels in high-level semantics, it struggles with perceiving fine-grained
visual details. Generally, to enhance representations, generative models take
CLIP's visual features as conditions for reconstruction. However, the
underlying principle remains underexplored. In this work, we empirically found
that visually perfect generations are not always optimal for representation
enhancement. The essence lies in effectively extracting fine-grained knowledge
from generative models while mitigating irrelevant information. To explore
critical factors, we delve into three aspects: (1) Conditioning mechanisms: We
found that even a small number of local tokens can drastically reduce the
difficulty of reconstruction, leading to collapsed training. We thus conclude
that utilizing only global visual tokens as conditions is the most effective
strategy. (2) Denoising configurations: We observed that end-to-end training
introduces extraneous information. To address this, we propose a two-stage
training strategy to prioritize learning useful visual knowledge. Additionally,
we demonstrate that lightweight denoisers can yield remarkable improvements.
(3) Generation paradigms: We explore both continuous and discrete denoisers
with desirable outcomes, validating the versatility of our method. Through our
in-depth explorations, we have finally arrived at an effective method, namely
GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark,
e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into
multimodal large language models for better vision-centric performance. All the
models and codes are made publicly available.
GenHancer enhances CLIP by improving its fine-grained visual knowledge extraction through careful conditioning, denoising strategies, and generation paradigms, leading to better performance on multimodal tasks.
Contrastive Language-Image Pre-Training CLIP fine-grained visual details reconstruction global visual tokens end-to-end training two-stage training lightweight denoisers continuous denoisers discrete denoisers GenHancer MMVP-VLM benchmark