title: mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document
    Understanding
summary: Document understanding refers to automatically extract, analyze and
  comprehend information from various types of digital documents, such as a web
  page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl,
  have demonstrated promising zero-shot capabilities in shallow OCR-free text
  recognition, indicating their potential for OCR-free document understanding.
  Nevertheless, without in-domain training, these models tend to ignore
  fine-grained OCR features, such as sophisticated tables or large blocks of
  text, which are essential for OCR-free document understanding. In this paper,
  we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding.
  Specifically, we first construct a instruction tuning dataset featuring a wide
  range of visual-text understanding tasks. Then, we strengthen the OCR-free
  document understanding ability by jointly train the model on language-only,
  general vision-and-language, and document instruction tuning dataset with our
  unified instruction tuning strategy. We also build an OCR-free document
  instruction understanding evaluation set LLMDoc to better compare models'
  capabilities on instruct compliance and document understanding. Experimental
  results show that our model outperforms existing multi-modal models,
  demonstrating its strong ability of document understanding. Besides, without
  specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream
  tasks. Our code, models, training data and evaluation set are available at
  https://github.com/X-PLUG/mPLUG-DocOwl.
paper_authors._id: 64a769bbfac891738ef7831f
paper_authors.name: Jiabo Ye
paper_authors._id: 64a769bbfac891738ef78320
paper_authors.name: Anwen Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T07:06:50.111Z
paper_authors._id: 6584f2f941dbedb146fbb902
paper_authors.avatarUrl: /avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
paper_authors.fullname: AnwenHu
paper_authors.type: user
paper_authors.user: AnwenHu
paper_authors._id: 64a769bbfac891738ef78321
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-11T07:55:56.152Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 64a769bbfac891738ef78322
paper_authors.name: Qinghao Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T08:47:49.347Z
paper_authors._id: 64530fc01a57e1179c1fe4c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
paper_authors.fullname: QinghaoYe
paper_authors.type: user
paper_authors.user: MAGAer13
paper_authors._id: 64a769bbfac891738ef78323
paper_authors.name: Ming Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T08:12:26.430Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 64a769bbfac891738ef78324
paper_authors.name: Yuhao Dan
paper_authors._id: 64a769bbfac891738ef78325
paper_authors.name: Chenlin Zhao
paper_authors._id: 64a769bbfac891738ef78326
paper_authors.name: Guohai Xu
paper_authors._id: 64a769bbfac891738ef78327
paper_authors.name: Chenliang Li
paper_authors._id: 64a769bbfac891738ef78328
paper_authors.name: Junfeng Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:54:40.235Z
paper_authors._id: 63086c94c038bf42d56795ca
paper_authors.avatarUrl: /avatars/91b9d959022d7dc22bbdb524a9de173a.svg
paper_authors.fullname: Junfeng Tian
paper_authors.type: user
paper_authors.user: rgtjf
paper_authors._id: 64a769bbfac891738ef78329
paper_authors.name: Qian Qi
paper_authors._id: 64a769bbfac891738ef7832a
paper_authors.name: Ji Zhang
paper_authors._id: 64a769bbfac891738ef7832b
paper_authors.name: Fei Huang
paper_title: mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document
    Understanding
paper_summary: Document understanding refers to automatically extract, analyze and
  comprehend information from various types of digital documents, such as a web
  page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl,
  have demonstrated promising zero-shot capabilities in shallow OCR-free text
  recognition, indicating their potential for OCR-free document understanding.
  Nevertheless, without in-domain training, these models tend to ignore
  fine-grained OCR features, such as sophisticated tables or large blocks of
  text, which are essential for OCR-free document understanding. In this paper,
  we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding.
  Specifically, we first construct a instruction tuning dataset featuring a wide
  range of visual-text understanding tasks. Then, we strengthen the OCR-free
  document understanding ability by jointly train the model on language-only,
  general vision-and-language, and document instruction tuning dataset with our
  unified instruction tuning strategy. We also build an OCR-free document
  instruction understanding evaluation set LLMDoc to better compare models'
  capabilities on instruct compliance and document understanding. Experimental
  results show that our model outperforms existing multi-modal models,
  demonstrating its strong ability of document understanding. Besides, without
  specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream
  tasks. Our code, models, training data and evaluation set are available at
  https://github.com/X-PLUG/mPLUG-DocOwl.
paper_ai_summary: mPLUG-DocOwl, an extension of mPLUG-Owl, enhances OCR-free document understanding through instruction tuning and achieves superior performance across various visual-text tasks.
paper_ai_keywords: Multi-model Large Language Models MLLMs mPLUG-Owl OCR-free text recognition visual-text understanding instruction tuning document instruction tuning dataset unified instruction tuning strategy LLMDoc document understanding instruct compliance downstream tasks