mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page
  Document Understanding
Multimodel Large Language Models(MLLMs) have achieved promising OCR-free
Document Understanding performance by increasing the supported resolution of
document images. However, this comes at the cost of generating thousands of
visual tokens for a single document image, leading to excessive GPU memory and
slower inference times, particularly in multi-page document comprehension. In
this work, to address these challenges, we propose a High-resolution
DocCompressor module to compress each high-resolution document image into 324
tokens, guided by low-resolution global visual features. With this compression
module, to strengthen multi-page document comprehension ability and balance
both token efficiency and question-answering performance, we develop the
DocOwl2 under a three-stage training framework: Single-image Pretraining,
Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new
state-of-the-art across multi-page document understanding benchmarks and
reduces first token latency by more than 50%, demonstrating advanced
capabilities in multi-page questioning answering, explanation with evidence
pages, and cross-page structure understanding. Additionally, compared to
single-image MLLMs trained on similar data, our DocOwl2 achieves comparable
single-page understanding performance with less than 20% of the visual tokens.
Our codes, models, and data are publicly available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
66da7321bb5f2d96e46b1392
Anwen Hu
claimed_verified
2024-09-06T07:06:20.990Z
6584f2f941dbedb146fbb902
/avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
AnwenHu
user
AnwenHu
66da7321bb5f2d96e46b1393
Haiyang Xu
claimed_verified
2024-09-06T07:06:18.759Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
66da7321bb5f2d96e46b1394
Liang Zhang
66da7321bb5f2d96e46b1395
Jiabo Ye
66da7321bb5f2d96e46b1396
Ming Yan
66da7321bb5f2d96e46b1397
Ji Zhang
66da7321bb5f2d96e46b1398
Qin Jin
66da7321bb5f2d96e46b1399
Fei Huang
admin_assigned
2024-09-06T09:31:11.178Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
66da7321bb5f2d96e46b139a
Jingren Zhou
admin_assigned
2024-09-06T09:30:53.626Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page
  Document Understanding
Multimodel Large Language Models(MLLMs) have achieved promising OCR-free
Document Understanding performance by increasing the supported resolution of
document images. However, this comes at the cost of generating thousands of
visual tokens for a single document image, leading to excessive GPU memory and
slower inference times, particularly in multi-page document comprehension. In
this work, to address these challenges, we propose a High-resolution
DocCompressor module to compress each high-resolution document image into 324
tokens, guided by low-resolution global visual features. With this compression
module, to strengthen multi-page document comprehension ability and balance
both token efficiency and question-answering performance, we develop the
DocOwl2 under a three-stage training framework: Single-image Pretraining,
Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new
state-of-the-art across multi-page document understanding benchmarks and
reduces first token latency by more than 50%, demonstrating advanced
capabilities in multi-page questioning answering, explanation with evidence
pages, and cross-page structure understanding. Additionally, compared to
single-image MLLMs trained on similar data, our DocOwl2 achieves comparable
single-page understanding performance with less than 20% of the visual tokens.
Our codes, models, and data are publicly available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
A high-resolution compression module and three-stage training framework enhance multi-page document understanding and reduce latency while maintaining performance.
multimodel large language models OCR-free document understanding visual tokens high-resolution DocCompressor low-resolution global visual features single-image pretraining multi-image continue-pretraining multi-task finetuning multi-page document comprehension single-page understanding cross-page structure understanding