title: Reconstruction vs. Generation: Taming Optimization Dilemma in Latent
    Diffusion Models
summary: Latent diffusion models with Transformer architectures excel at generating
  high-fidelity images. However, recent studies reveal an optimization dilemma in
  this two-stage design: while increasing the per-token feature dimension in
  visual tokenizers improves reconstruction quality, it requires substantially
  larger diffusion models and more training iterations to achieve comparable
  generation performance. Consequently, existing systems often settle for
  sub-optimal solutions, either producing visual artifacts due to information
  loss within tokenizers or failing to converge fully due to expensive
  computation costs. We argue that this dilemma stems from the inherent
  difficulty in learning unconstrained high-dimensional latent spaces. To address
  this, we propose aligning the latent space with pre-trained vision foundation
  models when training the visual tokenizers. Our proposed VA-VAE (Vision
  foundation model Aligned Variational AutoEncoder) significantly expands the
  reconstruction-generation frontier of latent diffusion models, enabling faster
  convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces.
  To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with
  improved training strategies and architecture designs, termed LightningDiT. The
  integrated system achieves state-of-the-art (SOTA) performance on ImageNet
  256x256 generation with an FID score of 1.35 while demonstrating remarkable
  training efficiency by reaching an FID score of 2.11 in just 64
  epochs--representing an over 21 times convergence speedup compared to the
  original DiT. Models and codes are available at:
  https://github.com/hustvl/LightningDiT.
paper_authors._id: 677753a28376dfe003a3fbd3
paper_authors.name: Jingfeng Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:17:09.787Z
paper_authors._id: 67756c9c846a267749304255
paper_authors.avatarUrl: /avatars/01f09805b561887c55d1b9ad4e96b461.svg
paper_authors.fullname: Jingfeng Yao
paper_authors.type: user
paper_authors.user: MapleF9
paper_authors._id: 677753a28376dfe003a3fbd4
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:17:16.030Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_title: Reconstruction vs. Generation: Taming Optimization Dilemma in Latent
    Diffusion Models
paper_summary: Latent diffusion models with Transformer architectures excel at generating
  high-fidelity images. However, recent studies reveal an optimization dilemma in
  this two-stage design: while increasing the per-token feature dimension in
  visual tokenizers improves reconstruction quality, it requires substantially
  larger diffusion models and more training iterations to achieve comparable
  generation performance. Consequently, existing systems often settle for
  sub-optimal solutions, either producing visual artifacts due to information
  loss within tokenizers or failing to converge fully due to expensive
  computation costs. We argue that this dilemma stems from the inherent
  difficulty in learning unconstrained high-dimensional latent spaces. To address
  this, we propose aligning the latent space with pre-trained vision foundation
  models when training the visual tokenizers. Our proposed VA-VAE (Vision
  foundation model Aligned Variational AutoEncoder) significantly expands the
  reconstruction-generation frontier of latent diffusion models, enabling faster
  convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces.
  To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with
  improved training strategies and architecture designs, termed LightningDiT. The
  integrated system achieves state-of-the-art (SOTA) performance on ImageNet
  256x256 generation with an FID score of 1.35 while demonstrating remarkable
  training efficiency by reaching an FID score of 2.11 in just 64
  epochs--representing an over 21 times convergence speedup compared to the
  original DiT. Models and codes are available at:
  https://github.com/hustvl/LightningDiT.
paper_ai_summary: VA-VAE aligns latent spaces with pre-trained vision foundation models, enhancing DiT's training efficiency and performance on high-fidelity image generation.
paper_ai_keywords: latent diffusion models Transformer architectures visual tokenizers diffusion models pre-trained vision foundation models Variational AutoEncoder DiT ImageNet FID score LightningDiT