title: TokBench: Evaluating Your Visual Tokenizer before Visual Generation
summary: In this work, we reveal the limitations of visual tokenizers and VAEs in
  preserving fine-grained features, and propose a benchmark to evaluate
  reconstruction performance for two challenging visual contents: text and face.
  Visual tokenizers and VAEs have significantly advanced visual generation and
  multimodal modeling by providing more efficient compressed or quantized image
  representations. However, while helping production models reduce computational
  burdens, the information loss from image compression fundamentally limits the
  upper bound of visual generation quality. To evaluate this upper bound, we
  focus on assessing reconstructed text and facial features since they typically:
  1) exist at smaller scales, 2) contain dense and rich textures, 3) are prone to
  collapse, and 4) are highly sensitive to human vision. We first collect and
  curate a diverse set of clear text and face images from existing datasets.
  Unlike approaches using VLM models, we employ established OCR and face
  recognition models for evaluation, ensuring accuracy while maintaining an
  exceptionally lightweight assessment process <span style="font-weight: bold;
  color: rgb(214, 21, 21);">requiring just 2GB memory and 4 minutes</span> to
  complete. Using our benchmark, we analyze text and face reconstruction quality
  across various scales for different image tokenizers and VAEs. Our results show
  modern visual tokenizers still struggle to preserve fine-grained features,
  especially at smaller scales. We further extend this evaluation framework to
  video, conducting comprehensive analysis of video tokenizers. Additionally, we
  demonstrate that traditional metrics fail to accurately reflect reconstruction
  performance for faces and text, while our proposed metrics serve as an
  effective complement.
paper_authors._id: 6839d7b3ede0c6c6e33e9363
paper_authors.name: Junfeng Wu
paper_authors._id: 6839d7b3ede0c6c6e33e9364
paper_authors.name: Dongliang Luo
paper_authors._id: 6839d7b3ede0c6c6e33e9365
paper_authors.name: Weizhi Zhao
paper_authors._id: 6839d7b3ede0c6c6e33e9366
paper_authors.name: Zhihao Xie
paper_authors._id: 6839d7b3ede0c6c6e33e9367
paper_authors.name: Yuanhao Wang
paper_authors._id: 6839d7b3ede0c6c6e33e9368
paper_authors.name: Junyi Li
paper_authors._id: 6839d7b3ede0c6c6e33e9369
paper_authors.name: Xudong Xie
paper_authors._id: 6839d7b3ede0c6c6e33e936a
paper_authors.name: Yuliang Liu
paper_authors._id: 6839d7b3ede0c6c6e33e936b
paper_authors.name: Xiang Bai
paper_title: TokBench: Evaluating Your Visual Tokenizer before Visual Generation
paper_summary: In this work, we reveal the limitations of visual tokenizers and VAEs in
  preserving fine-grained features, and propose a benchmark to evaluate
  reconstruction performance for two challenging visual contents: text and face.
  Visual tokenizers and VAEs have significantly advanced visual generation and
  multimodal modeling by providing more efficient compressed or quantized image
  representations. However, while helping production models reduce computational
  burdens, the information loss from image compression fundamentally limits the
  upper bound of visual generation quality. To evaluate this upper bound, we
  focus on assessing reconstructed text and facial features since they typically:
  1) exist at smaller scales, 2) contain dense and rich textures, 3) are prone to
  collapse, and 4) are highly sensitive to human vision. We first collect and
  curate a diverse set of clear text and face images from existing datasets.
  Unlike approaches using VLM models, we employ established OCR and face
  recognition models for evaluation, ensuring accuracy while maintaining an
  exceptionally lightweight assessment process <span style="font-weight: bold;
  color: rgb(214, 21, 21);">requiring just 2GB memory and 4 minutes</span> to
  complete. Using our benchmark, we analyze text and face reconstruction quality
  across various scales for different image tokenizers and VAEs. Our results show
  modern visual tokenizers still struggle to preserve fine-grained features,
  especially at smaller scales. We further extend this evaluation framework to
  video, conducting comprehensive analysis of video tokenizers. Additionally, we
  demonstrate that traditional metrics fail to accurately reflect reconstruction
  performance for faces and text, while our proposed metrics serve as an
  effective complement.
paper_ai_summary: Evaluation of visual tokenizers and VAEs on fine-grained feature reconstruction, focusing on text and face, reveals limitations in preserving detailed visual content and highlights the need for specialized metrics.
paper_ai_keywords: visual tokenizers VAEs fine-grained features benchmarking text reconstruction facial feature reconstruction image compression information loss visual generation quality multimodal modeling OCR face recognition video tokenizers reconstruction metrics