LMDX: Language Model-based Document Information Extraction and
  Localization
Large Language Models (LLM) have revolutionized Natural Language Processing
(NLP), improving state-of-the-art on many existing tasks and exhibiting
emergent capabilities. However, LLMs have not yet been successfully applied on
semi-structured document information extraction, which is at the core of many
document processing workflows and consists of extracting key entities from a
visually rich document (VRD) given a predefined target schema. The main
obstacles to LLM adoption in that task have been the absence of layout encoding
within LLMs, critical for a high quality extraction, and the lack of a
grounding mechanism ensuring the answer is not hallucinated. In this paper, we
introduce Language Model-based Document Information Extraction and Localization
(LMDX), a methodology to adapt arbitrary LLMs for document information
extraction. LMDX can do extraction of singular, repeated, and hierarchical
entities, both with and without training data, while providing grounding
guarantees and localizing the entities within the document. In particular, we
apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks,
setting a new state-of-the-art and showing how LMDX enables the creation of
high quality, data-efficient parsers.
650ba3e612c086ce18bcd345
Vincent Perot
claimed_verified
2023-09-21T17:56:45.682Z
64e3f501bbee0c24b95e20c2
/avatars/bae7b5297e779d91da21ff50389dd90a.svg
Vincent
user
vperot
650ba3e612c086ce18bcd346
Kai Kang
claimed_verified
2023-09-21T17:56:41.904Z
650c6f0a5d57feb847fc9067
/avatars/3c11e332671a23c95968b9379f82be9e.svg
Kai Kang
user
ghoulken
650ba3e612c086ce18bcd347
Florian Luisier
claimed_verified
2023-09-25T10:51:44.944Z
650f1e6adba31194cb440d44
/avatars/aa7d3280c80a53ff2f9c524db0f03b5d.svg
Florian Luisier
user
fluisier
650ba3e612c086ce18bcd348
Guolong Su
650ba3e612c086ce18bcd349
Xiaoyu Sun
claimed_verified
2023-10-13T07:53:05.598Z
65287653f34dd2dc6dc0c22d
/avatars/4e1d832f4871ff61af7a65b23e13724e.svg
Xiaoyu(Sean) Sun
user
sunse
650ba3e612c086ce18bcd34a
Ramya Sree Boppana
claimed_verified
2023-09-22T10:17:39.576Z
650c8550cbd0c7d550b8cfac
/avatars/bce6d09fa6595308b71828d8da7545b5.svg
Ramya Sree Boppana
user
RamyaSreeBoppana
650ba3e612c086ce18bcd34b
Zilong Wang
admin_assigned
2023-09-21T09:58:03.731Z
64b722dd2f5a966b97282236
/avatars/aa4ef471069cea0dffc8c7e4ee61a2fd.svg
Zilong Wang
user
zilongwang
650ba3e612c086ce18bcd34c
Jiaqi Mu
650ba3e612c086ce18bcd34d
Hao Zhang
650ba3e612c086ce18bcd34e
Nan Hua
LMDX: Language Model-based Document Information Extraction and
  Localization
Large Language Models (LLM) have revolutionized Natural Language Processing
(NLP), improving state-of-the-art on many existing tasks and exhibiting
emergent capabilities. However, LLMs have not yet been successfully applied on
semi-structured document information extraction, which is at the core of many
document processing workflows and consists of extracting key entities from a
visually rich document (VRD) given a predefined target schema. The main
obstacles to LLM adoption in that task have been the absence of layout encoding
within LLMs, critical for a high quality extraction, and the lack of a
grounding mechanism ensuring the answer is not hallucinated. In this paper, we
introduce Language Model-based Document Information Extraction and Localization
(LMDX), a methodology to adapt arbitrary LLMs for document information
extraction. LMDX can do extraction of singular, repeated, and hierarchical
entities, both with and without training data, while providing grounding
guarantees and localizing the entities within the document. In particular, we
apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks,
setting a new state-of-the-art and showing how LMDX enables the creation of
high quality, data-efficient parsers.
LMDX adapts large language models for document information extraction with grounding guarantees, setting a new state-of-the-art on VRDU and CORD benchmarks.
Large Language Models Natural Language Processing LLM document information extraction visually rich document layout encoding grounding mechanism LMDX PaLM 2-S VRDU CORD benchmarks data-efficient parsers