mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document
  Understanding
Structure information is critical for understanding the semantics of
text-rich images, such as documents, tables, and charts. Existing Multimodal
Large Language Models (MLLMs) for Visual Document Understanding are equipped
with text recognition ability but lack general structure understanding
abilities for text-rich document images. In this work, we emphasize the
importance of structure information in Visual Document Understanding and
propose the Unified Structure Learning to boost the performance of MLLMs. Our
Unified Structure Learning comprises structure-aware parsing tasks and
multi-grained text localization tasks across 5 domains: document, webpage,
table, chart, and natural image. To better encode structure information, we
design a simple and effective vision-to-text module H-Reducer, which can not
only maintain the layout information but also reduce the length of visual
features by merging horizontal adjacent patches through convolution, enabling
the LLM to understand high-resolution images more efficiently. Furthermore, by
constructing structure-aware text sequences and multi-grained pairs of texts
and bounding boxes for publicly available text-rich images, we build a
comprehensive training set DocStruct4M to support structure learning. Finally,
we construct a small but high-quality reasoning tuning dataset DocReason25K to
trigger the detailed explanation ability in the document domain. Our model
DocOwl 1.5 achieves state-of-the-art performance on 10 visual document
understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM
by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are
publicly available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.
65fa62c3e12a435e55d9eeb6
Anwen Hu
admin_assigned
2024-03-21T11:57:42.669Z
6584f2f941dbedb146fbb902
/avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
AnwenHu
user
AnwenHu
65fa62c3e12a435e55d9eeb7
Haiyang Xu
claimed_verified
2024-03-21T08:27:12.448Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
65fa62c3e12a435e55d9eeb8
Jiabo Ye
65fa62c3e12a435e55d9eeb9
Ming Yan
claimed_verified
2024-08-14T08:12:05.576Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
65fa62c3e12a435e55d9eeba
Liang Zhang
65fa62c3e12a435e55d9eebb
Bo Zhang
65fa62c3e12a435e55d9eebc
Chen Li
65fa62c3e12a435e55d9eebd
Ji Zhang
65fa62c3e12a435e55d9eebe
Qin Jin
65fa62c3e12a435e55d9eebf
Fei Huang
admin_assigned
2024-06-06T07:15:41.073Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
65fa62c3e12a435e55d9eec0
Jingren Zhou
admin_assigned
2024-03-21T11:59:49.811Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document
  Understanding
Structure information is critical for understanding the semantics of
text-rich images, such as documents, tables, and charts. Existing Multimodal
Large Language Models (MLLMs) for Visual Document Understanding are equipped
with text recognition ability but lack general structure understanding
abilities for text-rich document images. In this work, we emphasize the
importance of structure information in Visual Document Understanding and
propose the Unified Structure Learning to boost the performance of MLLMs. Our
Unified Structure Learning comprises structure-aware parsing tasks and
multi-grained text localization tasks across 5 domains: document, webpage,
table, chart, and natural image. To better encode structure information, we
design a simple and effective vision-to-text module H-Reducer, which can not
only maintain the layout information but also reduce the length of visual
features by merging horizontal adjacent patches through convolution, enabling
the LLM to understand high-resolution images more efficiently. Furthermore, by
constructing structure-aware text sequences and multi-grained pairs of texts
and bounding boxes for publicly available text-rich images, we build a
comprehensive training set DocStruct4M to support structure learning. Finally,
we construct a small but high-quality reasoning tuning dataset DocReason25K to
trigger the detailed explanation ability in the document domain. Our model
DocOwl 1.5 achieves state-of-the-art performance on 10 visual document
understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM
by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are
publicly available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.
A Unified Structure Learning framework is introduced to enhance Visual Document Understanding by integrating structure-aware parsing and multi-grained text localization tasks using a vision-to-text module H-Reducer, achieving state-of-the-art results on multiple benchmarks.
Multimodal Large Language Models Visual Document Understanding structure-aware parsing multi-grained text localization H-Reducer vision-to-text layout information visual features convolution high-resolution images structure-aware text sequences bounding boxes DocStruct4M DocReason25K DocOwl 1.5