title: PDFTriage: Question Answering over Long, Structured Documents
summary: Large Language Models (LLMs) have issues with document question answering
  (QA) in situations where the document is unable to fit in the small context
  length of an LLM. To overcome this issue, most existing works focus on
  retrieving the relevant context from the document, representing them as plain
  text. However, documents such as PDFs, web pages, and presentations are
  naturally structured with different pages, tables, sections, and so on.
  Representing such structured documents as plain text is incongruous with the
  user's mental model of these documents with rich structure. When a system has
  to query the document for context, this incongruity is brought to the fore, and
  seemingly trivial questions can trip up the QA system. To bridge this
  fundamental gap in handling structured documents, we propose an approach called
  PDFTriage that enables models to retrieve the context based on either structure
  or content. Our experiments demonstrate the effectiveness of the proposed
  PDFTriage-augmented models across several classes of questions where existing
  retrieval-augmented LLMs fail. To facilitate further research on this
  fundamental problem, we release our benchmark dataset consisting of 900+
  human-generated questions over 80 structured documents from 10 different
  categories of question types for document QA.
paper_authors._id: 65090c23c2d4c8fc37854a7e
paper_authors.name: Jon Saad-Falcon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:12:31.495Z
paper_authors._id: 6035790c56d1aa1de2953e47
paper_authors.avatarUrl: /avatars/c5fba6f1aed48d3b61298ee085018ee2.svg
paper_authors.fullname: Jon Saad-Falcon
paper_authors.type: user
paper_authors.user: jonsaadfalcon
paper_authors._id: 65090c23c2d4c8fc37854a7f
paper_authors.name: Joe Barrow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:13:13.217Z
paper_authors._id: 64b71b089ebb7e6c7dce4d38
paper_authors.avatarUrl: /avatars/b5601682b6500f30369ad7660e88bb55.svg
paper_authors.fullname: Joe Barrow
paper_authors.type: user
paper_authors.user: jbarrow
paper_authors._id: 65090c23c2d4c8fc37854a80
paper_authors.name: Alexa Siu
paper_authors._id: 65090c23c2d4c8fc37854a81
paper_authors.name: Ani Nenkova
paper_authors._id: 65090c23c2d4c8fc37854a82
paper_authors.name: Ryan A. Rossi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:13:35.102Z
paper_authors._id: 62a3ab83e4dd6252344d27cd
paper_authors.avatarUrl: /avatars/7ca8510f70a58dc207b104240e30c35c.svg
paper_authors.fullname: Ryan A. Rossi
paper_authors.type: user
paper_authors.user: ryanrossi
paper_authors._id: 65090c23c2d4c8fc37854a83
paper_authors.name: Franck Dernoncourt
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-19T18:56:10.355Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_title: PDFTriage: Question Answering over Long, Structured Documents
paper_summary: Large Language Models (LLMs) have issues with document question answering
  (QA) in situations where the document is unable to fit in the small context
  length of an LLM. To overcome this issue, most existing works focus on
  retrieving the relevant context from the document, representing them as plain
  text. However, documents such as PDFs, web pages, and presentations are
  naturally structured with different pages, tables, sections, and so on.
  Representing such structured documents as plain text is incongruous with the
  user's mental model of these documents with rich structure. When a system has
  to query the document for context, this incongruity is brought to the fore, and
  seemingly trivial questions can trip up the QA system. To bridge this
  fundamental gap in handling structured documents, we propose an approach called
  PDFTriage that enables models to retrieve the context based on either structure
  or content. Our experiments demonstrate the effectiveness of the proposed
  PDFTriage-augmented models across several classes of questions where existing
  retrieval-augmented LLMs fail. To facilitate further research on this
  fundamental problem, we release our benchmark dataset consisting of 900+
  human-generated questions over 80 structured documents from 10 different
  categories of question types for document QA.
paper_ai_summary: PDFTriage enables LLMs to retrieve context effectively from structured documents like PDFs and web pages for better document QA performance.
paper_ai_keywords: LLMs document question answering QA context length PDFTriage