PEARL: Prompting Large Language Models to Plan and Execute Actions Over
  Long Documents
Strategies such as chain-of-thought prompting improve the performance of
large language models (LLMs) on complex reasoning tasks by decomposing input
examples into intermediate steps. However, it remains unclear how to apply such
methods to reason over long input documents, in which both the decomposition
and the output of each intermediate step are non-trivial to obtain. In this
work, we propose PEARL, a prompting framework to improve reasoning over long
documents, which consists of three stages: action mining, plan formulation, and
plan execution. More specifically, given a question about a long document,
PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE,
FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain
the answer. Each stage of PEARL is implemented via zero-shot or few-shot
prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate
PEARL on a challenging subset of the QuALITY dataset, which contains questions
that require complex reasoning over long narrative texts. PEARL outperforms
zero-shot and chain-of-thought prompting on this dataset, and ablation
experiments show that each stage of PEARL is critical to its performance.
Overall, PEARL is a first step towards leveraging LLMs to reason over long
documents.
646ede5acbb7bb99651444f4
Simeng Sun
claimed_verified
2023-05-25T15:20:54.810Z
62cd08c1ff01ad1c9e509bc9
/avatars/2dc8a0c381962e8c35c6571c406b46ab.svg
Simeng Sun
user
simsun131
646ede5acbb7bb99651444f5
Yang Liu
646ede5acbb7bb99651444f6
Shuohang Wang
646ede5acbb7bb99651444f7
Chenguang Zhu
646ede5acbb7bb99651444f8
Mohit Iyyer
PEARL: Prompting Large Language Models to Plan and Execute Actions Over
  Long Documents
Strategies such as chain-of-thought prompting improve the performance of
large language models (LLMs) on complex reasoning tasks by decomposing input
examples into intermediate steps. However, it remains unclear how to apply such
methods to reason over long input documents, in which both the decomposition
and the output of each intermediate step are non-trivial to obtain. In this
work, we propose PEARL, a prompting framework to improve reasoning over long
documents, which consists of three stages: action mining, plan formulation, and
plan execution. More specifically, given a question about a long document,
PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE,
FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain
the answer. Each stage of PEARL is implemented via zero-shot or few-shot
prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate
PEARL on a challenging subset of the QuALITY dataset, which contains questions
that require complex reasoning over long narrative texts. PEARL outperforms
zero-shot and chain-of-thought prompting on this dataset, and ablation
experiments show that each stage of PEARL is critical to its performance.
Overall, PEARL is a first step towards leveraging LLMs to reason over long
documents.
PEARL is a prompting framework that decomposes complex questions into a sequence of actions and executes them using LLMs to reason over long documents, outperforming existing methods.
chain-of-thought prompting large language models zero-shot prompting few-shot prompting action mining plan formulation plan execution QuALITY dataset complex reasoning long narrative texts