DocLLM: A layout-aware generative language model for multimodal document
  understanding
Enterprise documents such as forms, invoices, receipts, reports, contracts,
and other similar records, often carry rich semantics at the intersection of
textual and spatial modalities. The visual cues offered by their complex
layouts play a crucial role in comprehending these documents effectively. In
this paper, we present DocLLM, a lightweight extension to traditional large
language models (LLMs) for reasoning over visual documents, taking into account
both textual semantics and spatial layout. Our model differs from existing
multimodal LLMs by avoiding expensive image encoders and focuses exclusively on
bounding box information to incorporate the spatial layout structure.
Specifically, the cross-alignment between text and spatial modalities is
captured by decomposing the attention mechanism in classical transformers to a
set of disentangled matrices. Furthermore, we devise a pre-training objective
that learns to infill text segments. This approach allows us to address
irregular layouts and heterogeneous content frequently encountered in visual
documents. The pre-trained model is fine-tuned using a large-scale instruction
dataset, covering four core document intelligence tasks. We demonstrate that
our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks,
and generalizes well to 4 out of 5 previously unseen datasets.
6594c32616227c7a2de9619c
Dongsheng Wang
claimed_verified
2024-01-04T09:34:47.892Z
64863e4e60b30c3615b92812
/avatars/e4a271b53e567f9423bcd9e62cf77889.svg
Dongsheng Wang
user
dswang2011
6594c32616227c7a2de9619d
Natraj Raman
6594c32616227c7a2de9619e
Mathieu Sibue
claimed_verified
2024-01-03T14:43:20.248Z
652e446ad2937a53ed7f90e9
/avatars/b33e0af2c34a6430ab719e9e6d7d833b.svg
Mathieu Sibue
user
mathieu-sibue
6594c32616227c7a2de9619f
Zhiqiang Ma
claimed_verified
2024-01-04T09:34:50.761Z
64b702e8fa7eabaae5f0cbfb
/avatars/512690f60997ae81b79b999a634a1861.svg
Zhiqiang Ma
user
zqmaster
6594c32616227c7a2de961a0
Petr Babkin
6594c32616227c7a2de961a1
Simerjot Kaur
6594c32616227c7a2de961a2
Yulong Pei
6594c32616227c7a2de961a3
Armineh Nourbakhsh
claimed_verified
2024-01-04T09:34:53.242Z
5e79436f30dc073f817a2b91
/avatars/5ffac317818689106be1a00ee69cded0.svg
Armi Nouri
user
arminouri
6594c32616227c7a2de961a4
Xiaomo Liu
claimed_verified
2024-01-04T09:35:30.740Z
61b895807b296cc68f98658d
/avatars/ad8af5621af7edb6ba21d0d24455246c.svg
Shawn Liu
user
langqinren
DocLLM: A layout-aware generative language model for multimodal document
  understanding
Enterprise documents such as forms, invoices, receipts, reports, contracts,
and other similar records, often carry rich semantics at the intersection of
textual and spatial modalities. The visual cues offered by their complex
layouts play a crucial role in comprehending these documents effectively. In
this paper, we present DocLLM, a lightweight extension to traditional large
language models (LLMs) for reasoning over visual documents, taking into account
both textual semantics and spatial layout. Our model differs from existing
multimodal LLMs by avoiding expensive image encoders and focuses exclusively on
bounding box information to incorporate the spatial layout structure.
Specifically, the cross-alignment between text and spatial modalities is
captured by decomposing the attention mechanism in classical transformers to a
set of disentangled matrices. Furthermore, we devise a pre-training objective
that learns to infill text segments. This approach allows us to address
irregular layouts and heterogeneous content frequently encountered in visual
documents. The pre-trained model is fine-tuned using a large-scale instruction
dataset, covering four core document intelligence tasks. We demonstrate that
our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks,
and generalizes well to 4 out of 5 previously unseen datasets.
DocLLM extends large language models for visual document understanding by focusing on bounding box information and disentangled matrices for multimodal attention, achieving state-of-the-art performance on document intelligence tasks.
multimodal LLMs image encoders bounding box information attention mechanism disentangled matrices pre-training objective infill text segments document intelligence tasks