title: Learnings from Scaling Visual Tokenizers for Reconstruction and
    Generation
summary: Visual tokenization via auto-encoding empowers state-of-the-art image and
  video generative models by compressing pixels into a latent space. Although
  scaling Transformer-based generators has been central to recent advances, the
  tokenizer component itself is rarely scaled, leaving open questions about how
  auto-encoder design choices influence both its objective of reconstruction and
  downstream generative performance. Our work aims to conduct an exploration of
  scaling in auto-encoders to fill in this blank. To facilitate this exploration,
  we replace the typical convolutional backbone with an enhanced Vision
  Transformer architecture for Tokenization (ViTok). We train ViTok on
  large-scale image and video datasets far exceeding ImageNet-1K, removing data
  constraints on tokenizer scaling. We first study how scaling the auto-encoder
  bottleneck affects both reconstruction and generation -- and find that while it
  is highly correlated with reconstruction, its relationship with generation is
  more complex. We next explored the effect of separately scaling the
  auto-encoders' encoder and decoder on reconstruction and generation
  performance. Crucially, we find that scaling the encoder yields minimal gains
  for either reconstruction or generation, while scaling the decoder boosts
  reconstruction but the benefits for generation are mixed. Building on our
  exploration, we design ViTok as a lightweight auto-encoder that achieves
  competitive performance with state-of-the-art auto-encoders on ImageNet-1K and
  COCO reconstruction tasks (256p and 512p) while outperforming existing
  auto-encoders on 16-frame 128p video reconstruction for UCF-101, all with 2-5x
  fewer FLOPs. When integrated with Diffusion Transformers, ViTok demonstrates
  competitive performance on image generation for ImageNet-1K and sets new
  state-of-the-art benchmarks for class-conditional video generation on UCF-101.
paper_authors._id: 6789e275810f471d6aa3d2fa
paper_authors.name: Philippe Hansen-Estruch
paper_authors._id: 6789e275810f471d6aa3d2fb
paper_authors.name: David Yan
paper_authors._id: 6789e275810f471d6aa3d2fc
paper_authors.name: Ching-Yao Chung
paper_authors._id: 6789e275810f471d6aa3d2fd
paper_authors.name: Orr Zohar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-17T10:38:14.289Z
paper_authors._id: 648c9605565e3a44f3c9bb7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
paper_authors.fullname: Orr Zohar
paper_authors.type: user
paper_authors.user: orrzohar
paper_authors._id: 6789e275810f471d6aa3d2fe
paper_authors.name: Jialiang Wang
paper_authors._id: 6789e275810f471d6aa3d2ff
paper_authors.name: Tingbo Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-17T10:38:38.454Z
paper_authors._id: 655846d7ed8df83128f5826a
paper_authors.avatarUrl: /avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
paper_authors.fullname: Hou
paper_authors.type: user
paper_authors.user: Tingbo
paper_authors._id: 6789e275810f471d6aa3d300
paper_authors.name: Tao Xu
paper_authors._id: 6789e275810f471d6aa3d301
paper_authors.name: Sriram Vishwanath
paper_authors._id: 6789e275810f471d6aa3d302
paper_authors.name: Peter Vajda
paper_authors._id: 6789e275810f471d6aa3d303
paper_authors.name: Xinlei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-17T10:38:56.604Z
paper_authors._id: 63e58e3a006a775275e59e41
paper_authors.avatarUrl: /avatars/75262a35b27a2ae1939df9118120d99e.svg
paper_authors.fullname: Xinlei Chen
paper_authors.type: user
paper_authors.user: endernewton
paper_title: Learnings from Scaling Visual Tokenizers for Reconstruction and
    Generation
paper_summary: Visual tokenization via auto-encoding empowers state-of-the-art image and
  video generative models by compressing pixels into a latent space. Although
  scaling Transformer-based generators has been central to recent advances, the
  tokenizer component itself is rarely scaled, leaving open questions about how
  auto-encoder design choices influence both its objective of reconstruction and
  downstream generative performance. Our work aims to conduct an exploration of
  scaling in auto-encoders to fill in this blank. To facilitate this exploration,
  we replace the typical convolutional backbone with an enhanced Vision
  Transformer architecture for Tokenization (ViTok). We train ViTok on
  large-scale image and video datasets far exceeding ImageNet-1K, removing data
  constraints on tokenizer scaling. We first study how scaling the auto-encoder
  bottleneck affects both reconstruction and generation -- and find that while it
  is highly correlated with reconstruction, its relationship with generation is
  more complex. We next explored the effect of separately scaling the
  auto-encoders' encoder and decoder on reconstruction and generation
  performance. Crucially, we find that scaling the encoder yields minimal gains
  for either reconstruction or generation, while scaling the decoder boosts
  reconstruction but the benefits for generation are mixed. Building on our
  exploration, we design ViTok as a lightweight auto-encoder that achieves
  competitive performance with state-of-the-art auto-encoders on ImageNet-1K and
  COCO reconstruction tasks (256p and 512p) while outperforming existing
  auto-encoders on 16-frame 128p video reconstruction for UCF-101, all with 2-5x
  fewer FLOPs. When integrated with Diffusion Transformers, ViTok demonstrates
  competitive performance on image generation for ImageNet-1K and sets new
  state-of-the-art benchmarks for class-conditional video generation on UCF-101.
paper_ai_summary: Replacing the convolutional backbone with a Vision Transformer for tokenization yields competitive auto-encoder performance and sets benchmarks for class-conditional video generation when integrated with Diffusion Transformers.
paper_ai_keywords: Visual tokenization auto-encoding latent space Transformer-based generators Vision Transformer Tokenization ViTok large-scale datasets image reconstruction video reconstruction Diffusion Transformers FLOPs