title: mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document
    Understanding
summary: Structure information is critical for understanding the semantics of
  text-rich images, such as documents, tables, and charts. Existing Multimodal
  Large Language Models (MLLMs) for Visual Document Understanding are equipped
  with text recognition ability but lack general structure understanding
  abilities for text-rich document images. In this work, we emphasize the
  importance of structure information in Visual Document Understanding and
  propose the Unified Structure Learning to boost the performance of MLLMs. Our
  Unified Structure Learning comprises structure-aware parsing tasks and
  multi-grained text localization tasks across 5 domains: document, webpage,
  table, chart, and natural image. To better encode structure information, we
  design a simple and effective vision-to-text module H-Reducer, which can not
  only maintain the layout information but also reduce the length of visual
  features by merging horizontal adjacent patches through convolution, enabling
  the LLM to understand high-resolution images more efficiently. Furthermore, by
  constructing structure-aware text sequences and multi-grained pairs of texts
  and bounding boxes for publicly available text-rich images, we build a
  comprehensive training set DocStruct4M to support structure learning. Finally,
  we construct a small but high-quality reasoning tuning dataset DocReason25K to
  trigger the detailed explanation ability in the document domain. Our model
  DocOwl 1.5 achieves state-of-the-art performance on 10 visual document
  understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM
  by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are
  publicly available at
  https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.
paper_authors._id: 65fa62c3e12a435e55d9eeb6
paper_authors.name: Anwen Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:57:42.669Z
paper_authors._id: 6584f2f941dbedb146fbb902
paper_authors.avatarUrl: /avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
paper_authors.fullname: AnwenHu
paper_authors.type: user
paper_authors.user: AnwenHu
paper_authors._id: 65fa62c3e12a435e55d9eeb7
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:27:12.448Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 65fa62c3e12a435e55d9eeb8
paper_authors.name: Jiabo Ye
paper_authors._id: 65fa62c3e12a435e55d9eeb9
paper_authors.name: Ming Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T08:12:05.576Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 65fa62c3e12a435e55d9eeba
paper_authors.name: Liang Zhang
paper_authors._id: 65fa62c3e12a435e55d9eebb
paper_authors.name: Bo Zhang
paper_authors._id: 65fa62c3e12a435e55d9eebc
paper_authors.name: Chen Li
paper_authors._id: 65fa62c3e12a435e55d9eebd
paper_authors.name: Ji Zhang
paper_authors._id: 65fa62c3e12a435e55d9eebe
paper_authors.name: Qin Jin
paper_authors._id: 65fa62c3e12a435e55d9eebf
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T07:15:41.073Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 65fa62c3e12a435e55d9eec0
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:59:49.811Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document
    Understanding
paper_summary: Structure information is critical for understanding the semantics of
  text-rich images, such as documents, tables, and charts. Existing Multimodal
  Large Language Models (MLLMs) for Visual Document Understanding are equipped
  with text recognition ability but lack general structure understanding
  abilities for text-rich document images. In this work, we emphasize the
  importance of structure information in Visual Document Understanding and
  propose the Unified Structure Learning to boost the performance of MLLMs. Our
  Unified Structure Learning comprises structure-aware parsing tasks and
  multi-grained text localization tasks across 5 domains: document, webpage,
  table, chart, and natural image. To better encode structure information, we
  design a simple and effective vision-to-text module H-Reducer, which can not
  only maintain the layout information but also reduce the length of visual
  features by merging horizontal adjacent patches through convolution, enabling
  the LLM to understand high-resolution images more efficiently. Furthermore, by
  constructing structure-aware text sequences and multi-grained pairs of texts
  and bounding boxes for publicly available text-rich images, we build a
  comprehensive training set DocStruct4M to support structure learning. Finally,
  we construct a small but high-quality reasoning tuning dataset DocReason25K to
  trigger the detailed explanation ability in the document domain. Our model
  DocOwl 1.5 achieves state-of-the-art performance on 10 visual document
  understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM
  by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are
  publicly available at
  https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.
paper_ai_summary: A Unified Structure Learning framework is introduced to enhance Visual Document Understanding by integrating structure-aware parsing and multi-grained text localization tasks using a vision-to-text module H-Reducer, achieving state-of-the-art results on multiple benchmarks.
paper_ai_keywords: Multimodal Large Language Models Visual Document Understanding structure-aware parsing multi-grained text localization H-Reducer vision-to-text layout information visual features convolution high-resolution images structure-aware text sequences bounding boxes DocStruct4M DocReason25K DocOwl 1.5