title: End-to-End Vision Tokenizer Tuning
summary: Existing vision tokenization isolates the optimization of vision tokenizers
  from downstream training, implicitly assuming the visual tokens can generalize
  well across various tasks, e.g., image generation and visual question
  answering. The vision tokenizer optimized for low-level reconstruction is
  agnostic to downstream tasks requiring varied representations and semantics.
  This decoupled paradigm introduces a critical misalignment: The loss of the
  vision tokenization can be the representation bottleneck for target tasks. For
  example, errors in tokenizing text in a given image lead to poor results when
  recognizing or generating them. To address this, we propose ETT, an end-to-end
  vision tokenizer tuning approach that enables joint optimization between vision
  tokenization and target autoregressive tasks. Unlike prior autoregressive
  models that use only discrete indices from a frozen vision tokenizer, ETT
  leverages the visual embeddings of the tokenizer codebook, and optimizes the
  vision tokenizers end-to-end with both reconstruction and caption objectives.
  ETT can be seamlessly integrated into existing training pipelines with minimal
  architecture modifications. Our ETT is simple to implement and integrate,
  without the need to adjust the original codebooks or architectures of the
  employed large language models. Extensive experiments demonstrate that our
  proposed end-to-end vision tokenizer tuning unlocks significant performance
  gains, i.e., 2-6% for multimodal understanding and visual generation tasks
  compared to frozen tokenizer baselines, while preserving the original
  reconstruction capability. We hope this very simple and strong method can
  empower multimodal foundation models besides image generation and
  understanding.
paper_authors._id: 6826a5d8154611642ada50da
paper_authors.name: Wenxuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T13:24:09.061Z
paper_authors._id: 656428b5462e5ebcbf537d4e
paper_authors.avatarUrl: /avatars/cbedecc9c6f2afee2ca6b72efb593561.svg
paper_authors.fullname: Wenxuan Wang
paper_authors.type: user
paper_authors.user: Rookielion
paper_authors._id: 6826a5d8154611642ada50db
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T07:12:04.101Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 6826a5d8154611642ada50dc
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:16:58.537Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 6826a5d8154611642ada50dd
paper_authors.name: Haiwen Diao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:17:04.879Z
paper_authors._id: 64b4a717aa03b6520839e9b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
paper_authors.fullname: Haiwen Diao
paper_authors.type: user
paper_authors.user: Paranioar
paper_authors._id: 6826a5d8154611642ada50de
paper_authors.name: Zhuoyan Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:17:11.336Z
paper_authors._id: 657187de7644d1128571495e
paper_authors.avatarUrl: /avatars/89412c94fd6136f6680055551de3ddc4.svg
paper_authors.fullname: Zhuoyan Luo
paper_authors.type: user
paper_authors.user: RobertLuo1
paper_authors._id: 6826a5d8154611642ada50df
paper_authors.name: Huchuan Lu
paper_authors._id: 6826a5d8154611642ada50e0
paper_authors.name: Jing Liu
paper_authors._id: 6826a5d8154611642ada50e1
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:17:32.558Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: End-to-End Vision Tokenizer Tuning
paper_summary: Existing vision tokenization isolates the optimization of vision tokenizers
  from downstream training, implicitly assuming the visual tokens can generalize
  well across various tasks, e.g., image generation and visual question
  answering. The vision tokenizer optimized for low-level reconstruction is
  agnostic to downstream tasks requiring varied representations and semantics.
  This decoupled paradigm introduces a critical misalignment: The loss of the
  vision tokenization can be the representation bottleneck for target tasks. For
  example, errors in tokenizing text in a given image lead to poor results when
  recognizing or generating them. To address this, we propose ETT, an end-to-end
  vision tokenizer tuning approach that enables joint optimization between vision
  tokenization and target autoregressive tasks. Unlike prior autoregressive
  models that use only discrete indices from a frozen vision tokenizer, ETT
  leverages the visual embeddings of the tokenizer codebook, and optimizes the
  vision tokenizers end-to-end with both reconstruction and caption objectives.
  ETT can be seamlessly integrated into existing training pipelines with minimal
  architecture modifications. Our ETT is simple to implement and integrate,
  without the need to adjust the original codebooks or architectures of the
  employed large language models. Extensive experiments demonstrate that our
  proposed end-to-end vision tokenizer tuning unlocks significant performance
  gains, i.e., 2-6% for multimodal understanding and visual generation tasks
  compared to frozen tokenizer baselines, while preserving the original
  reconstruction capability. We hope this very simple and strong method can
  empower multimodal foundation models besides image generation and
  understanding.
paper_ai_summary: ETT is an end-to-end vision tokenizer tuning method that integrates visual tokenizer training with autoregressive tasks, significantly improving performance in multimodal understanding and visual generation.
paper_ai_keywords: vision tokenization end-to-end vision tokenizer tuning autoregressive models visual embeddings codebook multimodal understanding visual generation