title: mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page
    Document Understanding
summary: Multimodel Large Language Models(MLLMs) have achieved promising OCR-free
  Document Understanding performance by increasing the supported resolution of
  document images. However, this comes at the cost of generating thousands of
  visual tokens for a single document image, leading to excessive GPU memory and
  slower inference times, particularly in multi-page document comprehension. In
  this work, to address these challenges, we propose a High-resolution
  DocCompressor module to compress each high-resolution document image into 324
  tokens, guided by low-resolution global visual features. With this compression
  module, to strengthen multi-page document comprehension ability and balance
  both token efficiency and question-answering performance, we develop the
  DocOwl2 under a three-stage training framework: Single-image Pretraining,
  Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new
  state-of-the-art across multi-page document understanding benchmarks and
  reduces first token latency by more than 50%, demonstrating advanced
  capabilities in multi-page questioning answering, explanation with evidence
  pages, and cross-page structure understanding. Additionally, compared to
  single-image MLLMs trained on similar data, our DocOwl2 achieves comparable
  single-page understanding performance with less than 20% of the visual tokens.
  Our codes, models, and data are publicly available at
  https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
paper_authors._id: 66da7321bb5f2d96e46b1392
paper_authors.name: Anwen Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T07:06:20.990Z
paper_authors._id: 6584f2f941dbedb146fbb902
paper_authors.avatarUrl: /avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
paper_authors.fullname: AnwenHu
paper_authors.type: user
paper_authors.user: AnwenHu
paper_authors._id: 66da7321bb5f2d96e46b1393
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T07:06:18.759Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 66da7321bb5f2d96e46b1394
paper_authors.name: Liang Zhang
paper_authors._id: 66da7321bb5f2d96e46b1395
paper_authors.name: Jiabo Ye
paper_authors._id: 66da7321bb5f2d96e46b1396
paper_authors.name: Ming Yan
paper_authors._id: 66da7321bb5f2d96e46b1397
paper_authors.name: Ji Zhang
paper_authors._id: 66da7321bb5f2d96e46b1398
paper_authors.name: Qin Jin
paper_authors._id: 66da7321bb5f2d96e46b1399
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:31:11.178Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 66da7321bb5f2d96e46b139a
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:30:53.626Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page
    Document Understanding
paper_summary: Multimodel Large Language Models(MLLMs) have achieved promising OCR-free
  Document Understanding performance by increasing the supported resolution of
  document images. However, this comes at the cost of generating thousands of
  visual tokens for a single document image, leading to excessive GPU memory and
  slower inference times, particularly in multi-page document comprehension. In
  this work, to address these challenges, we propose a High-resolution
  DocCompressor module to compress each high-resolution document image into 324
  tokens, guided by low-resolution global visual features. With this compression
  module, to strengthen multi-page document comprehension ability and balance
  both token efficiency and question-answering performance, we develop the
  DocOwl2 under a three-stage training framework: Single-image Pretraining,
  Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new
  state-of-the-art across multi-page document understanding benchmarks and
  reduces first token latency by more than 50%, demonstrating advanced
  capabilities in multi-page questioning answering, explanation with evidence
  pages, and cross-page structure understanding. Additionally, compared to
  single-image MLLMs trained on similar data, our DocOwl2 achieves comparable
  single-page understanding performance with less than 20% of the visual tokens.
  Our codes, models, and data are publicly available at
  https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
paper_ai_summary: A high-resolution compression module and three-stage training framework enhance multi-page document understanding and reduce latency while maintaining performance.
paper_ai_keywords: multimodel large language models OCR-free document understanding visual tokens high-resolution DocCompressor low-resolution global visual features single-image pretraining multi-image continue-pretraining multi-task finetuning multi-page document comprehension single-page understanding cross-page structure understanding