Learnings from Scaling Visual Tokenizers for Reconstruction and
  Generation
Visual tokenization via auto-encoding empowers state-of-the-art image and
video generative models by compressing pixels into a latent space. Although
scaling Transformer-based generators has been central to recent advances, the
tokenizer component itself is rarely scaled, leaving open questions about how
auto-encoder design choices influence both its objective of reconstruction and
downstream generative performance. Our work aims to conduct an exploration of
scaling in auto-encoders to fill in this blank. To facilitate this exploration,
we replace the typical convolutional backbone with an enhanced Vision
Transformer architecture for Tokenization (ViTok). We train ViTok on
large-scale image and video datasets far exceeding ImageNet-1K, removing data
constraints on tokenizer scaling. We first study how scaling the auto-encoder
bottleneck affects both reconstruction and generation -- and find that while it
is highly correlated with reconstruction, its relationship with generation is
more complex. We next explored the effect of separately scaling the
auto-encoders' encoder and decoder on reconstruction and generation
performance. Crucially, we find that scaling the encoder yields minimal gains
for either reconstruction or generation, while scaling the decoder boosts
reconstruction but the benefits for generation are mixed. Building on our
exploration, we design ViTok as a lightweight auto-encoder that achieves
competitive performance with state-of-the-art auto-encoders on ImageNet-1K and
COCO reconstruction tasks (256p and 512p) while outperforming existing
auto-encoders on 16-frame 128p video reconstruction for UCF-101, all with 2-5x
fewer FLOPs. When integrated with Diffusion Transformers, ViTok demonstrates
competitive performance on image generation for ImageNet-1K and sets new
state-of-the-art benchmarks for class-conditional video generation on UCF-101.
6789e275810f471d6aa3d2fa
Philippe Hansen-Estruch
6789e275810f471d6aa3d2fb
David Yan
6789e275810f471d6aa3d2fc
Ching-Yao Chung
6789e275810f471d6aa3d2fd
Orr Zohar
admin_assigned
2025-01-17T10:38:14.289Z
648c9605565e3a44f3c9bb7b
https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
Orr Zohar
user
orrzohar
6789e275810f471d6aa3d2fe
Jialiang Wang
6789e275810f471d6aa3d2ff
Tingbo Hou
admin_assigned
2025-01-17T10:38:38.454Z
655846d7ed8df83128f5826a
/avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
Hou
user
Tingbo
6789e275810f471d6aa3d300
Tao Xu
6789e275810f471d6aa3d301
Sriram Vishwanath
6789e275810f471d6aa3d302
Peter Vajda
6789e275810f471d6aa3d303
Xinlei Chen
admin_assigned
2025-01-17T10:38:56.604Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
Learnings from Scaling Visual Tokenizers for Reconstruction and
  Generation
Visual tokenization via auto-encoding empowers state-of-the-art image and
video generative models by compressing pixels into a latent space. Although
scaling Transformer-based generators has been central to recent advances, the
tokenizer component itself is rarely scaled, leaving open questions about how
auto-encoder design choices influence both its objective of reconstruction and
downstream generative performance. Our work aims to conduct an exploration of
scaling in auto-encoders to fill in this blank. To facilitate this exploration,
we replace the typical convolutional backbone with an enhanced Vision
Transformer architecture for Tokenization (ViTok). We train ViTok on
large-scale image and video datasets far exceeding ImageNet-1K, removing data
constraints on tokenizer scaling. We first study how scaling the auto-encoder
bottleneck affects both reconstruction and generation -- and find that while it
is highly correlated with reconstruction, its relationship with generation is
more complex. We next explored the effect of separately scaling the
auto-encoders' encoder and decoder on reconstruction and generation
performance. Crucially, we find that scaling the encoder yields minimal gains
for either reconstruction or generation, while scaling the decoder boosts
reconstruction but the benefits for generation are mixed. Building on our
exploration, we design ViTok as a lightweight auto-encoder that achieves
competitive performance with state-of-the-art auto-encoders on ImageNet-1K and
COCO reconstruction tasks (256p and 512p) while outperforming existing
auto-encoders on 16-frame 128p video reconstruction for UCF-101, all with 2-5x
fewer FLOPs. When integrated with Diffusion Transformers, ViTok demonstrates
competitive performance on image generation for ImageNet-1K and sets new
state-of-the-art benchmarks for class-conditional video generation on UCF-101.
Replacing the convolutional backbone with a Vision Transformer for tokenization yields competitive auto-encoder performance and sets benchmarks for class-conditional video generation when integrated with Diffusion Transformers.
Visual tokenization auto-encoding latent space Transformer-based generators Vision Transformer Tokenization ViTok large-scale datasets image reconstruction video reconstruction Diffusion Transformers FLOPs