Vision Foundation Models as Effective Visual Tokenizers for
  Autoregressive Image Generation
Leveraging the powerful representations of pre-trained vision foundation
models -- traditionally used for visual comprehension -- we explore a novel
direction: building an image tokenizer directly atop such models, a largely
underexplored area. Specifically, we employ a frozen vision foundation model as
the encoder of our tokenizer. To enhance its effectiveness, we introduce two
key components: (1) a region-adaptive quantization framework that reduces
redundancy in the pre-trained features on regular 2D grids, and (2) a semantic
reconstruction objective that aligns the tokenizer's outputs with the
foundation model's representations to preserve semantic fidelity. Based on
these designs, our proposed image tokenizer, VFMTok, achieves substantial
improvements in image reconstruction and generation quality, while also
enhancing token efficiency. It further boosts autoregressive (AR) generation --
achieving a gFID of 2.07 on ImageNet benchmarks, while accelerating model
convergence by three times, and enabling high-fidelity class-conditional
synthesis without the need for classifier-free guidance (CFG). The code will be
released publicly to benefit the community.
6874b80f257d4f04353703a8
Anlin Zheng
6874b80f257d4f04353703a9
Xin Wen
claimed_verified
2025-07-15T19:10:26.461Z
63483629ac5172169929da0e
https://cdn-avatars.huggingface.co/v1/production/uploads/1665676793089-noauth.jpeg
Xin Wen
user
xwen99
6874b80f257d4f04353703aa
Xuanyang Zhang
claimed_verified
2025-07-15T19:10:22.334Z
6675854966c4fa6d0cee4d50
/avatars/aa6041a97985078e82cc89bfbade9828.svg
xuanyang zhang
user
xuanyangz
6874b80f257d4f04353703ab
Chuofan Ma
6874b80f257d4f04353703ac
Tiancai Wang
6874b80f257d4f04353703ad
Gang Yu
6874b80f257d4f04353703ae
Xiangyu Zhang
6874b80f257d4f04353703af
Xiaojuan Qi
claimed_verified
2025-07-15T19:10:24.424Z
6875266f9cd3191dfddc7071
/avatars/64c581910833b111e9a7bae5b8740229.svg
xiaojuan qi
user
xjqi
Vision Foundation Models as Effective Visual Tokenizers for
  Autoregressive Image Generation
Leveraging the powerful representations of pre-trained vision foundation
models -- traditionally used for visual comprehension -- we explore a novel
direction: building an image tokenizer directly atop such models, a largely
underexplored area. Specifically, we employ a frozen vision foundation model as
the encoder of our tokenizer. To enhance its effectiveness, we introduce two
key components: (1) a region-adaptive quantization framework that reduces
redundancy in the pre-trained features on regular 2D grids, and (2) a semantic
reconstruction objective that aligns the tokenizer's outputs with the
foundation model's representations to preserve semantic fidelity. Based on
these designs, our proposed image tokenizer, VFMTok, achieves substantial
improvements in image reconstruction and generation quality, while also
enhancing token efficiency. It further boosts autoregressive (AR) generation --
achieving a gFID of 2.07 on ImageNet benchmarks, while accelerating model
convergence by three times, and enabling high-fidelity class-conditional
synthesis without the need for classifier-free guidance (CFG). The code will be
released publicly to benefit the community.
A novel image tokenizer built on pre-trained vision foundation models improves image reconstruction, generation quality, and token efficiency, enhancing autoregressive generation and class-conditional synthesis.
vision foundation models image tokenizer region-adaptive quantization semantic reconstruction objective VFMTok gFID autoregressive generation class-conditional synthesis classifier-free guidance