title: MergeVQ: A Unified Framework for Visual Generation and Representation
    with Disentangled Token Merging and Quantization
summary: Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great
  success in both self-supervised pre-training and image generation. However,
  most existing methods struggle to address the trade-off in shared latent space
  for generation quality vs. representation learning and efficiency. To push the
  limits of this paradigm, we propose MergeVQ, which incorporates token merging
  techniques into VQ-based generative models to bridge the gap between image
  generation and visual representation learning in a unified architecture. During
  pre-training, MergeVQ decouples top-k semantics from latent space with the
  token merge module after self-attention blocks in the encoder for subsequent
  Look-up Free Quantization (LFQ) and global alignment and recovers their
  fine-grained details through cross-attention in the decoder for reconstruction.
  As for the second-stage generation, we introduce MergeAR, which performs KV
  Cache compression for efficient raster-order prediction. Extensive experiments
  on ImageNet verify that MergeVQ as an AR generative model achieves competitive
  performance in both visual representation learning and image generation tasks
  while maintaining favorable token efficiency and inference speed. The code and
  model will be available at https://apexgen-x.github.io/MergeVQ.
paper_authors._id: 67ecc3973d267d266649e075
paper_authors.name: Siyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:28:38.819Z
paper_authors._id: 640f7083208821a59b74c757
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
paper_authors.fullname: Siyuan Li
paper_authors.type: user
paper_authors.user: Lupin1998
paper_authors._id: 67ecc3973d267d266649e076
paper_authors.name: Luyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:28:41.242Z
paper_authors._id: 671b4781d2f774c5ec9ebd62
paper_authors.avatarUrl: /avatars/b4f1cbaa6e092eda005f81f199a35e19.svg
paper_authors.fullname: Luyuan Zhang
paper_authors.type: user
paper_authors.user: LuyuanZhang01
paper_authors._id: 67ecc3973d267d266649e077
paper_authors.name: Zedong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T07:52:39.150Z
paper_authors._id: 6594d390674349122ce6f368
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6594d390674349122ce6f368/KdWz6lZyGYQpjAgBDeiC1.jpeg
paper_authors.fullname: Zedong Wang (Jacky)
paper_authors.type: user
paper_authors.user: ZedongWangAI
paper_authors._id: 67ecc3973d267d266649e078
paper_authors.name: Juanxi Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:28:43.436Z
paper_authors._id: 670880950e79a8b46f7ff9dd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/670880950e79a8b46f7ff9dd/hA1TLhwlQblkFsq8wLrkB.jpeg
paper_authors.fullname: Juanxi Tian
paper_authors.type: user
paper_authors.user: Juanxi
paper_authors._id: 67ecc3973d267d266649e079
paper_authors.name: Cheng Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T19:21:03.903Z
paper_authors._id: 64be296a46cc3cdfbb057f7e
paper_authors.avatarUrl: /avatars/f9f27567b3d59085bf32dd1c708dc90c.svg
paper_authors.fullname: Cheng Tan
paper_authors.type: user
paper_authors.user: chengtan9907
paper_authors._id: 67ecc3973d267d266649e07a
paper_authors.name: Zicheng Liu
paper_authors._id: 67ecc3973d267d266649e07b
paper_authors.name: Chang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T13:33:08.711Z
paper_authors._id: 63578f79a1f8ad1c31bd2148
paper_authors.avatarUrl: /avatars/e91cf0a7c71a9533556267e67bf0610f.svg
paper_authors.fullname: Y
paper_authors.type: user
paper_authors.user: CY-7
paper_authors._id: 67ecc3973d267d266649e07c
paper_authors.name: Qingsong Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:34:47.666Z
paper_authors._id: 64184911db24526c7c9ca035
paper_authors.avatarUrl: /avatars/2be15485bdd9e00857d4bff107d1577e.svg
paper_authors.fullname: qingsong xie
paper_authors.type: user
paper_authors.user: kongming11
paper_authors._id: 67ecc3973d267d266649e07d
paper_authors.name: Haonan Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:34:57.853Z
paper_authors._id: 603864181b642aef9350af81
paper_authors.avatarUrl: /avatars/27ec0b885cbedef007876c8c98c69d15.svg
paper_authors.fullname: Haonan Lu
paper_authors.type: user
paper_authors.user: luhaonan20
paper_authors._id: 67ecc3973d267d266649e07e
paper_authors.name: Haoqian Wang
paper_authors._id: 67ecc3973d267d266649e07f
paper_authors.name: Zhen Lei
paper_title: MergeVQ: A Unified Framework for Visual Generation and Representation
    with Disentangled Token Merging and Quantization
paper_summary: Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great
  success in both self-supervised pre-training and image generation. However,
  most existing methods struggle to address the trade-off in shared latent space
  for generation quality vs. representation learning and efficiency. To push the
  limits of this paradigm, we propose MergeVQ, which incorporates token merging
  techniques into VQ-based generative models to bridge the gap between image
  generation and visual representation learning in a unified architecture. During
  pre-training, MergeVQ decouples top-k semantics from latent space with the
  token merge module after self-attention blocks in the encoder for subsequent
  Look-up Free Quantization (LFQ) and global alignment and recovers their
  fine-grained details through cross-attention in the decoder for reconstruction.
  As for the second-stage generation, we introduce MergeAR, which performs KV
  Cache compression for efficient raster-order prediction. Extensive experiments
  on ImageNet verify that MergeVQ as an AR generative model achieves competitive
  performance in both visual representation learning and image generation tasks
  while maintaining favorable token efficiency and inference speed. The code and
  model will be available at https://apexgen-x.github.io/MergeVQ.
paper_ai_summary: MergeVQ and MergeAR enhance MIM with VQ by integrating token merging and KV cache compression, improving both image generation and visual representation learning.
paper_ai_keywords: MIM Vector Quantization MergeVQ token merging Look-up Free Quantization self-attention cross-attention decoder MergeAR KV Cache compression AR generative model visual representation learning image generation