title: PEARL: Prompting Large Language Models to Plan and Execute Actions Over
    Long Documents
summary: Strategies such as chain-of-thought prompting improve the performance of
  large language models (LLMs) on complex reasoning tasks by decomposing input
  examples into intermediate steps. However, it remains unclear how to apply such
  methods to reason over long input documents, in which both the decomposition
  and the output of each intermediate step are non-trivial to obtain. In this
  work, we propose PEARL, a prompting framework to improve reasoning over long
  documents, which consists of three stages: action mining, plan formulation, and
  plan execution. More specifically, given a question about a long document,
  PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE,
  FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain
  the answer. Each stage of PEARL is implemented via zero-shot or few-shot
  prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate
  PEARL on a challenging subset of the QuALITY dataset, which contains questions
  that require complex reasoning over long narrative texts. PEARL outperforms
  zero-shot and chain-of-thought prompting on this dataset, and ablation
  experiments show that each stage of PEARL is critical to its performance.
  Overall, PEARL is a first step towards leveraging LLMs to reason over long
  documents.
paper_authors._id: 646ede5acbb7bb99651444f4
paper_authors.name: Simeng Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-25T15:20:54.810Z
paper_authors._id: 62cd08c1ff01ad1c9e509bc9
paper_authors.avatarUrl: /avatars/2dc8a0c381962e8c35c6571c406b46ab.svg
paper_authors.fullname: Simeng Sun
paper_authors.type: user
paper_authors.user: simsun131
paper_authors._id: 646ede5acbb7bb99651444f5
paper_authors.name: Yang Liu
paper_authors._id: 646ede5acbb7bb99651444f6
paper_authors.name: Shuohang Wang
paper_authors._id: 646ede5acbb7bb99651444f7
paper_authors.name: Chenguang Zhu
paper_authors._id: 646ede5acbb7bb99651444f8
paper_authors.name: Mohit Iyyer
paper_title: PEARL: Prompting Large Language Models to Plan and Execute Actions Over
    Long Documents
paper_summary: Strategies such as chain-of-thought prompting improve the performance of
  large language models (LLMs) on complex reasoning tasks by decomposing input
  examples into intermediate steps. However, it remains unclear how to apply such
  methods to reason over long input documents, in which both the decomposition
  and the output of each intermediate step are non-trivial to obtain. In this
  work, we propose PEARL, a prompting framework to improve reasoning over long
  documents, which consists of three stages: action mining, plan formulation, and
  plan execution. More specifically, given a question about a long document,
  PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE,
  FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain
  the answer. Each stage of PEARL is implemented via zero-shot or few-shot
  prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate
  PEARL on a challenging subset of the QuALITY dataset, which contains questions
  that require complex reasoning over long narrative texts. PEARL outperforms
  zero-shot and chain-of-thought prompting on this dataset, and ablation
  experiments show that each stage of PEARL is critical to its performance.
  Overall, PEARL is a first step towards leveraging LLMs to reason over long
  documents.
paper_ai_summary: PEARL is a prompting framework that decomposes complex questions into a sequence of actions and executes them using LLMs to reason over long documents, outperforming existing methods.
paper_ai_keywords: chain-of-thought prompting large language models zero-shot prompting few-shot prompting action mining plan formulation plan execution QuALITY dataset complex reasoning long narrative texts