MergeVQ: A Unified Framework for Visual Generation and Representation
  with Disentangled Token Merging and Quantization
Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great
success in both self-supervised pre-training and image generation. However,
most existing methods struggle to address the trade-off in shared latent space
for generation quality vs. representation learning and efficiency. To push the
limits of this paradigm, we propose MergeVQ, which incorporates token merging
techniques into VQ-based generative models to bridge the gap between image
generation and visual representation learning in a unified architecture. During
pre-training, MergeVQ decouples top-k semantics from latent space with the
token merge module after self-attention blocks in the encoder for subsequent
Look-up Free Quantization (LFQ) and global alignment and recovers their
fine-grained details through cross-attention in the decoder for reconstruction.
As for the second-stage generation, we introduce MergeAR, which performs KV
Cache compression for efficient raster-order prediction. Extensive experiments
on ImageNet verify that MergeVQ as an AR generative model achieves competitive
performance in both visual representation learning and image generation tasks
while maintaining favorable token efficiency and inference speed. The code and
model will be available at https://apexgen-x.github.io/MergeVQ.
67ecc3973d267d266649e075
Siyuan Li
claimed_verified
2025-04-03T08:28:38.819Z
640f7083208821a59b74c757
https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
Siyuan Li
user
Lupin1998
67ecc3973d267d266649e076
Luyuan Zhang
claimed_verified
2025-04-03T08:28:41.242Z
671b4781d2f774c5ec9ebd62
/avatars/b4f1cbaa6e092eda005f81f199a35e19.svg
Luyuan Zhang
user
LuyuanZhang01
67ecc3973d267d266649e077
Zedong Wang
claimed_verified
2025-04-03T07:52:39.150Z
6594d390674349122ce6f368
https://cdn-avatars.huggingface.co/v1/production/uploads/6594d390674349122ce6f368/KdWz6lZyGYQpjAgBDeiC1.jpeg
Zedong Wang (Jacky)
user
ZedongWangAI
67ecc3973d267d266649e078
Juanxi Tian
claimed_verified
2025-04-03T08:28:43.436Z
670880950e79a8b46f7ff9dd
https://cdn-avatars.huggingface.co/v1/production/uploads/670880950e79a8b46f7ff9dd/hA1TLhwlQblkFsq8wLrkB.jpeg
Juanxi Tian
user
Juanxi
67ecc3973d267d266649e079
Cheng Tan
claimed_verified
2025-04-03T19:21:03.903Z
64be296a46cc3cdfbb057f7e
/avatars/f9f27567b3d59085bf32dd1c708dc90c.svg
Cheng Tan
user
chengtan9907
67ecc3973d267d266649e07a
Zicheng Liu
67ecc3973d267d266649e07b
Chang Yu
claimed_verified
2025-04-03T13:33:08.711Z
63578f79a1f8ad1c31bd2148
/avatars/e91cf0a7c71a9533556267e67bf0610f.svg
Y
user
CY-7
67ecc3973d267d266649e07c
Qingsong Xie
admin_assigned
2025-04-03T13:34:47.666Z
64184911db24526c7c9ca035
/avatars/2be15485bdd9e00857d4bff107d1577e.svg
qingsong xie
user
kongming11
67ecc3973d267d266649e07d
Haonan Lu
admin_assigned
2025-04-03T13:34:57.853Z
603864181b642aef9350af81
/avatars/27ec0b885cbedef007876c8c98c69d15.svg
Haonan Lu
user
luhaonan20
67ecc3973d267d266649e07e
Haoqian Wang
67ecc3973d267d266649e07f
Zhen Lei
MergeVQ: A Unified Framework for Visual Generation and Representation
  with Disentangled Token Merging and Quantization
Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great
success in both self-supervised pre-training and image generation. However,
most existing methods struggle to address the trade-off in shared latent space
for generation quality vs. representation learning and efficiency. To push the
limits of this paradigm, we propose MergeVQ, which incorporates token merging
techniques into VQ-based generative models to bridge the gap between image
generation and visual representation learning in a unified architecture. During
pre-training, MergeVQ decouples top-k semantics from latent space with the
token merge module after self-attention blocks in the encoder for subsequent
Look-up Free Quantization (LFQ) and global alignment and recovers their
fine-grained details through cross-attention in the decoder for reconstruction.
As for the second-stage generation, we introduce MergeAR, which performs KV
Cache compression for efficient raster-order prediction. Extensive experiments
on ImageNet verify that MergeVQ as an AR generative model achieves competitive
performance in both visual representation learning and image generation tasks
while maintaining favorable token efficiency and inference speed. The code and
model will be available at https://apexgen-x.github.io/MergeVQ.
MergeVQ and MergeAR enhance MIM with VQ by integrating token merging and KV cache compression, improving both image generation and visual representation learning.
MIM Vector Quantization MergeVQ token merging Look-up Free Quantization self-attention cross-attention decoder MergeAR KV Cache compression AR generative model visual representation learning image generation