title: LMDX: Language Model-based Document Information Extraction and
    Localization
summary: Large Language Models (LLM) have revolutionized Natural Language Processing
  (NLP), improving state-of-the-art on many existing tasks and exhibiting
  emergent capabilities. However, LLMs have not yet been successfully applied on
  semi-structured document information extraction, which is at the core of many
  document processing workflows and consists of extracting key entities from a
  visually rich document (VRD) given a predefined target schema. The main
  obstacles to LLM adoption in that task have been the absence of layout encoding
  within LLMs, critical for a high quality extraction, and the lack of a
  grounding mechanism ensuring the answer is not hallucinated. In this paper, we
  introduce Language Model-based Document Information Extraction and Localization
  (LMDX), a methodology to adapt arbitrary LLMs for document information
  extraction. LMDX can do extraction of singular, repeated, and hierarchical
  entities, both with and without training data, while providing grounding
  guarantees and localizing the entities within the document. In particular, we
  apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks,
  setting a new state-of-the-art and showing how LMDX enables the creation of
  high quality, data-efficient parsers.
paper_authors._id: 650ba3e612c086ce18bcd345
paper_authors.name: Vincent Perot
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-21T17:56:45.682Z
paper_authors._id: 64e3f501bbee0c24b95e20c2
paper_authors.avatarUrl: /avatars/bae7b5297e779d91da21ff50389dd90a.svg
paper_authors.fullname: Vincent
paper_authors.type: user
paper_authors.user: vperot
paper_authors._id: 650ba3e612c086ce18bcd346
paper_authors.name: Kai Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-21T17:56:41.904Z
paper_authors._id: 650c6f0a5d57feb847fc9067
paper_authors.avatarUrl: /avatars/3c11e332671a23c95968b9379f82be9e.svg
paper_authors.fullname: Kai Kang
paper_authors.type: user
paper_authors.user: ghoulken
paper_authors._id: 650ba3e612c086ce18bcd347
paper_authors.name: Florian Luisier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-25T10:51:44.944Z
paper_authors._id: 650f1e6adba31194cb440d44
paper_authors.avatarUrl: /avatars/aa7d3280c80a53ff2f9c524db0f03b5d.svg
paper_authors.fullname: Florian Luisier
paper_authors.type: user
paper_authors.user: fluisier
paper_authors._id: 650ba3e612c086ce18bcd348
paper_authors.name: Guolong Su
paper_authors._id: 650ba3e612c086ce18bcd349
paper_authors.name: Xiaoyu Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-13T07:53:05.598Z
paper_authors._id: 65287653f34dd2dc6dc0c22d
paper_authors.avatarUrl: /avatars/4e1d832f4871ff61af7a65b23e13724e.svg
paper_authors.fullname: Xiaoyu(Sean) Sun
paper_authors.type: user
paper_authors.user: sunse
paper_authors._id: 650ba3e612c086ce18bcd34a
paper_authors.name: Ramya Sree Boppana
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-22T10:17:39.576Z
paper_authors._id: 650c8550cbd0c7d550b8cfac
paper_authors.avatarUrl: /avatars/bce6d09fa6595308b71828d8da7545b5.svg
paper_authors.fullname: Ramya Sree Boppana
paper_authors.type: user
paper_authors.user: RamyaSreeBoppana
paper_authors._id: 650ba3e612c086ce18bcd34b
paper_authors.name: Zilong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:58:03.731Z
paper_authors._id: 64b722dd2f5a966b97282236
paper_authors.avatarUrl: /avatars/aa4ef471069cea0dffc8c7e4ee61a2fd.svg
paper_authors.fullname: Zilong Wang
paper_authors.type: user
paper_authors.user: zilongwang
paper_authors._id: 650ba3e612c086ce18bcd34c
paper_authors.name: Jiaqi Mu
paper_authors._id: 650ba3e612c086ce18bcd34d
paper_authors.name: Hao Zhang
paper_authors._id: 650ba3e612c086ce18bcd34e
paper_authors.name: Nan Hua
paper_title: LMDX: Language Model-based Document Information Extraction and
    Localization
paper_summary: Large Language Models (LLM) have revolutionized Natural Language Processing
  (NLP), improving state-of-the-art on many existing tasks and exhibiting
  emergent capabilities. However, LLMs have not yet been successfully applied on
  semi-structured document information extraction, which is at the core of many
  document processing workflows and consists of extracting key entities from a
  visually rich document (VRD) given a predefined target schema. The main
  obstacles to LLM adoption in that task have been the absence of layout encoding
  within LLMs, critical for a high quality extraction, and the lack of a
  grounding mechanism ensuring the answer is not hallucinated. In this paper, we
  introduce Language Model-based Document Information Extraction and Localization
  (LMDX), a methodology to adapt arbitrary LLMs for document information
  extraction. LMDX can do extraction of singular, repeated, and hierarchical
  entities, both with and without training data, while providing grounding
  guarantees and localizing the entities within the document. In particular, we
  apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks,
  setting a new state-of-the-art and showing how LMDX enables the creation of
  high quality, data-efficient parsers.
paper_ai_keywords: Large Language Models (LLM) Natural Language Processing (NLP) semi-structured document information extraction visually rich document (VRD) predefined target schema layout encoding grounding mechanism Language Model-based Document Information Extraction and Localization (LMDX) extraction of singular, repeated, and hierarchical entities training data grounding guarantees localization PaLM 2-S VRDU CORD benchmarks high quality parsers data-efficient parsers