title: TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal
    Large Language Models
summary: Multimodal Large Language Models (MLLMs) have shown impressive results on
  various multimodal tasks. However, most existing MLLMs are not well suited for
  document-oriented tasks, which require fine-grained image perception and
  information compression. In this paper, we present TextHawk, a MLLM that is
  specifically designed for document-oriented tasks, while preserving the general
  capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained
  perception by designing four dedicated components. Firstly, a ReSampling and
  ReArrangement (ReSA) module is proposed to reduce the redundancy in the
  document texts and lower the computational cost of the MLLM. We explore
  encoding the positions of each local feature by presenting Scalable Positional
  Embeddings (SPEs), which can preserve the scalability of various image sizes. A
  Query Proposal Network (QPN) is then adopted to initialize the queries
  dynamically among different sub-images. To further enhance the fine-grained
  visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention
  (MLCA) mechanism that captures the hierarchical structure and semantic
  relations of document images. Furthermore, we create a new instruction-tuning
  dataset for document-oriented tasks by enriching the multimodal document data
  with Gemini Pro. We conduct extensive experiments on both general and
  document-oriented MLLM benchmarks, and show that TextHawk outperforms the
  state-of-the-art methods, demonstrating its effectiveness and superiority in
  fine-grained document perception and general abilities.
paper_authors._id: 661e1387f40801e72305d509
paper_authors.name: Ya-Qi Yu
paper_authors._id: 661e1387f40801e72305d50a
paper_authors.name: Minghui Liao
paper_authors._id: 661e1387f40801e72305d50b
paper_authors.name: Jihao Wu
paper_authors._id: 661e1387f40801e72305d50c
paper_authors.name: Yongxin Liao
paper_authors._id: 661e1387f40801e72305d50d
paper_authors.name: Xiaoyu Zheng
paper_authors._id: 661e1387f40801e72305d50e
paper_authors.name: Wei Zeng
paper_title: TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal
    Large Language Models
paper_summary: Multimodal Large Language Models (MLLMs) have shown impressive results on
  various multimodal tasks. However, most existing MLLMs are not well suited for
  document-oriented tasks, which require fine-grained image perception and
  information compression. In this paper, we present TextHawk, a MLLM that is
  specifically designed for document-oriented tasks, while preserving the general
  capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained
  perception by designing four dedicated components. Firstly, a ReSampling and
  ReArrangement (ReSA) module is proposed to reduce the redundancy in the
  document texts and lower the computational cost of the MLLM. We explore
  encoding the positions of each local feature by presenting Scalable Positional
  Embeddings (SPEs), which can preserve the scalability of various image sizes. A
  Query Proposal Network (QPN) is then adopted to initialize the queries
  dynamically among different sub-images. To further enhance the fine-grained
  visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention
  (MLCA) mechanism that captures the hierarchical structure and semantic
  relations of document images. Furthermore, we create a new instruction-tuning
  dataset for document-oriented tasks by enriching the multimodal document data
  with Gemini Pro. We conduct extensive experiments on both general and
  document-oriented MLLM benchmarks, and show that TextHawk outperforms the
  state-of-the-art methods, demonstrating its effectiveness and superiority in
  fine-grained document perception and general abilities.
paper_ai_summary: TextHawk, a multimodal large language model designed for document-oriented tasks, outperforms existing models by using specialized components for efficient text and image perception.
paper_ai_keywords: ReSampling and ReArrangement (ReSA) module Scalable Positional Embeddings (SPEs) Query Proposal Network (QPN) Multi-Level Cross-Attention (MLCA) mechanism