mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document
  Understanding
Document understanding refers to automatically extract, analyze and
comprehend information from various types of digital documents, such as a web
page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl,
have demonstrated promising zero-shot capabilities in shallow OCR-free text
recognition, indicating their potential for OCR-free document understanding.
Nevertheless, without in-domain training, these models tend to ignore
fine-grained OCR features, such as sophisticated tables or large blocks of
text, which are essential for OCR-free document understanding. In this paper,
we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding.
Specifically, we first construct a instruction tuning dataset featuring a wide
range of visual-text understanding tasks. Then, we strengthen the OCR-free
document understanding ability by jointly train the model on language-only,
general vision-and-language, and document instruction tuning dataset with our
unified instruction tuning strategy. We also build an OCR-free document
instruction understanding evaluation set LLMDoc to better compare models'
capabilities on instruct compliance and document understanding. Experimental
results show that our model outperforms existing multi-modal models,
demonstrating its strong ability of document understanding. Besides, without
specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream
tasks. Our code, models, training data and evaluation set are available at
https://github.com/X-PLUG/mPLUG-DocOwl.
64a769bbfac891738ef7831f
Jiabo Ye
64a769bbfac891738ef78320
Anwen Hu
claimed_verified
2024-09-06T07:06:50.111Z
6584f2f941dbedb146fbb902
/avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
AnwenHu
user
AnwenHu
64a769bbfac891738ef78321
Haiyang Xu
claimed_verified
2023-07-11T07:55:56.152Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
64a769bbfac891738ef78322
Qinghao Ye
claimed_verified
2025-05-22T08:47:49.347Z
64530fc01a57e1179c1fe4c0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
QinghaoYe
user
MAGAer13
64a769bbfac891738ef78323
Ming Yan
claimed_verified
2024-08-14T08:12:26.430Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
64a769bbfac891738ef78324
Yuhao Dan
64a769bbfac891738ef78325
Chenlin Zhao
64a769bbfac891738ef78326
Guohai Xu
64a769bbfac891738ef78327
Chenliang Li
64a769bbfac891738ef78328
Junfeng Tian
claimed_verified
2024-09-24T07:54:40.235Z
63086c94c038bf42d56795ca
/avatars/91b9d959022d7dc22bbdb524a9de173a.svg
Junfeng Tian
user
rgtjf
64a769bbfac891738ef78329
Qian Qi
64a769bbfac891738ef7832a
Ji Zhang
64a769bbfac891738ef7832b
Fei Huang
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document
  Understanding
Document understanding refers to automatically extract, analyze and
comprehend information from various types of digital documents, such as a web
page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl,
have demonstrated promising zero-shot capabilities in shallow OCR-free text
recognition, indicating their potential for OCR-free document understanding.
Nevertheless, without in-domain training, these models tend to ignore
fine-grained OCR features, such as sophisticated tables or large blocks of
text, which are essential for OCR-free document understanding. In this paper,
we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding.
Specifically, we first construct a instruction tuning dataset featuring a wide
range of visual-text understanding tasks. Then, we strengthen the OCR-free
document understanding ability by jointly train the model on language-only,
general vision-and-language, and document instruction tuning dataset with our
unified instruction tuning strategy. We also build an OCR-free document
instruction understanding evaluation set LLMDoc to better compare models'
capabilities on instruct compliance and document understanding. Experimental
results show that our model outperforms existing multi-modal models,
demonstrating its strong ability of document understanding. Besides, without
specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream
tasks. Our code, models, training data and evaluation set are available at
https://github.com/X-PLUG/mPLUG-DocOwl.
mPLUG-DocOwl, an extension of mPLUG-Owl, enhances OCR-free document understanding through instruction tuning and achieves superior performance across various visual-text tasks.
Multi-model Large Language Models MLLMs mPLUG-Owl OCR-free text recognition visual-text understanding instruction tuning document instruction tuning dataset unified instruction tuning strategy LLMDoc document understanding instruct compliance downstream tasks