title: A Token-level Text Image Foundation Model for Document Understanding
summary: In recent years, general visual foundation models (VFMs) have witnessed
  increasing adoption, particularly as image encoders for popular multi-modal
  large language models (MLLMs). However, without semantically fine-grained
  supervision, these models still encounter fundamental prediction errors in the
  context of downstream text-image-related tasks, i.e., perception, understanding
  and reasoning with images containing small and dense texts. To bridge this gap,
  we develop TokenOCR, the first token-level visual foundation model specifically
  tailored for text-image-related tasks, designed to support a variety of
  traditional downstream applications. To facilitate the pretraining of TokenOCR,
  we also devise a high-quality data production pipeline that constructs the
  first token-level image text dataset, TokenIT, comprising 20 million images and
  1.8 billion token-mask pairs. Furthermore, leveraging this foundation with
  exceptional image-as-text capability, we seamlessly replace previous VFMs with
  TokenOCR to construct a document-level MLLM, TokenVL, for VQA-based document
  understanding tasks. Finally, extensive experiments demonstrate the
  effectiveness of TokenOCR and TokenVL. Code, datasets, and weights will be
  available at https://token-family.github.io/TokenOCR_project.
paper_authors._id: 67c7c17380815cf71fb9139d
paper_authors.name: Tongkun Guan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T15:47:05.706Z
paper_authors._id: 650d4a36cbd0c7d550d3b41b
paper_authors.avatarUrl: /avatars/0f628ccbbad894098b201a188eb3bd0b.svg
paper_authors.fullname: TongkunGuan
paper_authors.type: user
paper_authors.user: TongkunGuan
paper_authors._id: 67c7c17380815cf71fb9139e
paper_authors.name: Zining Wang
paper_authors._id: 67c7c17380815cf71fb9139f
paper_authors.name: Pei Fu
paper_authors._id: 67c7c17380815cf71fb913a0
paper_authors.name: Zhengtao Guo
paper_authors._id: 67c7c17380815cf71fb913a1
paper_authors.name: Wei Shen
paper_authors._id: 67c7c17380815cf71fb913a2
paper_authors.name: Kai Zhou
paper_authors._id: 67c7c17380815cf71fb913a3
paper_authors.name: Tiezhu Yue
paper_authors._id: 67c7c17380815cf71fb913a4
paper_authors.name: Chen Duan
paper_authors._id: 67c7c17380815cf71fb913a5
paper_authors.name: Hao Sun
paper_authors._id: 67c7c17380815cf71fb913a6
paper_authors.name: Qianyi Jiang
paper_authors._id: 67c7c17380815cf71fb913a7
paper_authors.name: Junfeng Luo
paper_authors._id: 67c7c17380815cf71fb913a8
paper_authors.name: Xiaokang Yang
paper_title: A Token-level Text Image Foundation Model for Document Understanding
paper_summary: In recent years, general visual foundation models (VFMs) have witnessed
  increasing adoption, particularly as image encoders for popular multi-modal
  large language models (MLLMs). However, without semantically fine-grained
  supervision, these models still encounter fundamental prediction errors in the
  context of downstream text-image-related tasks, i.e., perception, understanding
  and reasoning with images containing small and dense texts. To bridge this gap,
  we develop TokenOCR, the first token-level visual foundation model specifically
  tailored for text-image-related tasks, designed to support a variety of
  traditional downstream applications. To facilitate the pretraining of TokenOCR,
  we also devise a high-quality data production pipeline that constructs the
  first token-level image text dataset, TokenIT, comprising 20 million images and
  1.8 billion token-mask pairs. Furthermore, leveraging this foundation with
  exceptional image-as-text capability, we seamlessly replace previous VFMs with
  TokenOCR to construct a document-level MLLM, TokenVL, for VQA-based document
  understanding tasks. Finally, extensive experiments demonstrate the
  effectiveness of TokenOCR and TokenVL. Code, datasets, and weights will be
  available at https://token-family.github.io/TokenOCR_project.
paper_ai_summary: TokenOCR is a token-level visual foundation model designed to improve text-image understanding, constructed using a novel dataset for pretraining and integrated into a document-level MLLM named TokenVL, demonstrating superior performance in VQA tasks.
paper_ai_keywords: visual foundation models multi-modal large language models TokenOCR token-level image text dataset TokenIT image-as-text capability document-level MLLM TokenVL VQA-based document understanding