title: Stabilize the Latent Space for Image Autoregressive Modeling: A Unified
    Perspective
summary: Latent-based image generative models, such as Latent Diffusion Models (LDMs)
  and Mask Image Models (MIMs), have achieved notable success in image generation
  tasks. These models typically leverage reconstructive autoencoders like VQGAN
  or VAE to encode pixels into a more compact latent space and learn the data
  distribution in the latent space instead of directly from pixels. However, this
  practice raises a pertinent question: Is it truly the optimal choice? In
  response, we begin with an intriguing observation: despite sharing the same
  latent space, autoregressive models significantly lag behind LDMs and MIMs in
  image generation. This finding contrasts sharply with the field of NLP, where
  the autoregressive model GPT has established a commanding presence. To address
  this discrepancy, we introduce a unified perspective on the relationship
  between latent space and generative models, emphasizing the stability of latent
  space in image generative modeling. Furthermore, we propose a simple but
  effective discrete image tokenizer to stabilize the latent space for image
  generative modeling. Experimental results show that image autoregressive
  modeling with our tokenizer (DiGIT) benefits both image understanding and image
  generation with the next token prediction principle, which is inherently
  straightforward for GPT models but challenging for other generative models.
  Remarkably, for the first time, a GPT-style autoregressive model for images
  outperforms LDMs, which also exhibits substantial improvement akin to GPT when
  scaling up model size. Our findings underscore the potential of an optimized
  latent space and the integration of discrete tokenization in advancing the
  capabilities of image generative models. The code is available at
  https://github.com/DAMO-NLP-SG/DiGIT.
paper_authors._id: 6710bfb58bbf77ae7996d0e9
paper_authors.name: Yongxin Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-16T19:47:15.032Z
paper_authors._id: 637c7d8a88699fba70e1e1ff
paper_authors.avatarUrl: /avatars/e7f60fab6ca9ad5ecd320bd7cd51ce2e.svg
paper_authors.fullname: yongxinzhu
paper_authors.type: user
paper_authors.user: youngsheen
paper_authors._id: 6710bfb58bbf77ae7996d0ea
paper_authors.name: Bocheng Li
paper_authors._id: 6710bfb58bbf77ae7996d0eb
paper_authors.name: Hang Zhang
paper_authors._id: 6710bfb58bbf77ae7996d0ec
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:29:39.670Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 6710bfb58bbf77ae7996d0ed
paper_authors.name: Linli Xu
paper_authors._id: 6710bfb58bbf77ae7996d0ee
paper_authors.name: Lidong Bing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:11.681Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: Stabilize the Latent Space for Image Autoregressive Modeling: A Unified
    Perspective
paper_summary: Latent-based image generative models, such as Latent Diffusion Models (LDMs)
  and Mask Image Models (MIMs), have achieved notable success in image generation
  tasks. These models typically leverage reconstructive autoencoders like VQGAN
  or VAE to encode pixels into a more compact latent space and learn the data
  distribution in the latent space instead of directly from pixels. However, this
  practice raises a pertinent question: Is it truly the optimal choice? In
  response, we begin with an intriguing observation: despite sharing the same
  latent space, autoregressive models significantly lag behind LDMs and MIMs in
  image generation. This finding contrasts sharply with the field of NLP, where
  the autoregressive model GPT has established a commanding presence. To address
  this discrepancy, we introduce a unified perspective on the relationship
  between latent space and generative models, emphasizing the stability of latent
  space in image generative modeling. Furthermore, we propose a simple but
  effective discrete image tokenizer to stabilize the latent space for image
  generative modeling. Experimental results show that image autoregressive
  modeling with our tokenizer (DiGIT) benefits both image understanding and image
  generation with the next token prediction principle, which is inherently
  straightforward for GPT models but challenging for other generative models.
  Remarkably, for the first time, a GPT-style autoregressive model for images
  outperforms LDMs, which also exhibits substantial improvement akin to GPT when
  scaling up model size. Our findings underscore the potential of an optimized
  latent space and the integration of discrete tokenization in advancing the
  capabilities of image generative models. The code is available at
  https://github.com/DAMO-NLP-SG/DiGIT.
paper_ai_summary: A discrete image tokenizer stabilizes latent space for image generative modeling, enabling GPT-style autoregressive models to outperform Latent Diffusion Models.
paper_ai_keywords: Latent Diffusion Models Mask Image Models VQGAN VAE latent space image generative modeling autoregressive models image understanding image generation DiGIT