title: DocLLM: A layout-aware generative language model for multimodal document
    understanding
summary: Enterprise documents such as forms, invoices, receipts, reports, contracts,
  and other similar records, often carry rich semantics at the intersection of
  textual and spatial modalities. The visual cues offered by their complex
  layouts play a crucial role in comprehending these documents effectively. In
  this paper, we present DocLLM, a lightweight extension to traditional large
  language models (LLMs) for reasoning over visual documents, taking into account
  both textual semantics and spatial layout. Our model differs from existing
  multimodal LLMs by avoiding expensive image encoders and focuses exclusively on
  bounding box information to incorporate the spatial layout structure.
  Specifically, the cross-alignment between text and spatial modalities is
  captured by decomposing the attention mechanism in classical transformers to a
  set of disentangled matrices. Furthermore, we devise a pre-training objective
  that learns to infill text segments. This approach allows us to address
  irregular layouts and heterogeneous content frequently encountered in visual
  documents. The pre-trained model is fine-tuned using a large-scale instruction
  dataset, covering four core document intelligence tasks. We demonstrate that
  our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks,
  and generalizes well to 4 out of 5 previously unseen datasets.
paper_authors._id: 6594c32616227c7a2de9619c
paper_authors.name: Dongsheng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:34:47.892Z
paper_authors._id: 64863e4e60b30c3615b92812
paper_authors.avatarUrl: /avatars/e4a271b53e567f9423bcd9e62cf77889.svg
paper_authors.fullname: Dongsheng Wang
paper_authors.type: user
paper_authors.user: dswang2011
paper_authors._id: 6594c32616227c7a2de9619d
paper_authors.name: Natraj Raman
paper_authors._id: 6594c32616227c7a2de9619e
paper_authors.name: Mathieu Sibue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-03T14:43:20.248Z
paper_authors._id: 652e446ad2937a53ed7f90e9
paper_authors.avatarUrl: /avatars/b33e0af2c34a6430ab719e9e6d7d833b.svg
paper_authors.fullname: Mathieu Sibue
paper_authors.type: user
paper_authors.user: mathieu-sibue
paper_authors._id: 6594c32616227c7a2de9619f
paper_authors.name: Zhiqiang Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:34:50.761Z
paper_authors._id: 64b702e8fa7eabaae5f0cbfb
paper_authors.avatarUrl: /avatars/512690f60997ae81b79b999a634a1861.svg
paper_authors.fullname: Zhiqiang Ma
paper_authors.type: user
paper_authors.user: zqmaster
paper_authors._id: 6594c32616227c7a2de961a0
paper_authors.name: Petr Babkin
paper_authors._id: 6594c32616227c7a2de961a1
paper_authors.name: Simerjot Kaur
paper_authors._id: 6594c32616227c7a2de961a2
paper_authors.name: Yulong Pei
paper_authors._id: 6594c32616227c7a2de961a3
paper_authors.name: Armineh Nourbakhsh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:34:53.242Z
paper_authors._id: 5e79436f30dc073f817a2b91
paper_authors.avatarUrl: /avatars/5ffac317818689106be1a00ee69cded0.svg
paper_authors.fullname: Armi Nouri
paper_authors.type: user
paper_authors.user: arminouri
paper_authors._id: 6594c32616227c7a2de961a4
paper_authors.name: Xiaomo Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:35:30.740Z
paper_authors._id: 61b895807b296cc68f98658d
paper_authors.avatarUrl: /avatars/ad8af5621af7edb6ba21d0d24455246c.svg
paper_authors.fullname: Shawn Liu
paper_authors.type: user
paper_authors.user: langqinren
paper_title: DocLLM: A layout-aware generative language model for multimodal document
    understanding
paper_summary: Enterprise documents such as forms, invoices, receipts, reports, contracts,
  and other similar records, often carry rich semantics at the intersection of
  textual and spatial modalities. The visual cues offered by their complex
  layouts play a crucial role in comprehending these documents effectively. In
  this paper, we present DocLLM, a lightweight extension to traditional large
  language models (LLMs) for reasoning over visual documents, taking into account
  both textual semantics and spatial layout. Our model differs from existing
  multimodal LLMs by avoiding expensive image encoders and focuses exclusively on
  bounding box information to incorporate the spatial layout structure.
  Specifically, the cross-alignment between text and spatial modalities is
  captured by decomposing the attention mechanism in classical transformers to a
  set of disentangled matrices. Furthermore, we devise a pre-training objective
  that learns to infill text segments. This approach allows us to address
  irregular layouts and heterogeneous content frequently encountered in visual
  documents. The pre-trained model is fine-tuned using a large-scale instruction
  dataset, covering four core document intelligence tasks. We demonstrate that
  our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks,
  and generalizes well to 4 out of 5 previously unseen datasets.
paper_ai_summary: DocLLM extends large language models for visual document understanding by focusing on bounding box information and disentangled matrices for multimodal attention, achieving state-of-the-art performance on document intelligence tasks.
paper_ai_keywords: multimodal LLMs image encoders bounding box information attention mechanism disentangled matrices pre-training objective infill text segments document intelligence tasks