title: FLARE: Faithful Logic-Aided Reasoning and Exploration
summary: Modern Question Answering (QA) and Reasoning approaches based on Large
  Language Models (LLMs) commonly use prompting techniques, such as
  Chain-of-Thought (CoT), assuming the resulting generation will have a more
  granular exploration and reasoning over the question space and scope. However,
  such methods struggle with generating outputs that are faithful to the
  intermediate chain of reasoning produced by the model. On the other end of the
  spectrum, neuro-symbolic methods such as Faithful CoT (F-CoT) propose to
  combine LLMs with external symbolic solvers. While such approaches boast a high
  degree of faithfulness, they usually require a model trained for code
  generation and struggle with tasks that are ambiguous or hard to formalise
  strictly. We introduce Faithful Logic-Aided
  Reasoning and Exploration (\ours), a novel
  interpretable approach for traversing the problem space using task
  decompositions. We use the LLM to plan a solution, soft-formalise the query
  into facts and predicates using a logic programming code and simulate that code
  execution using an exhaustive multi-hop search over the defined space. Our
  method allows us to compute the faithfulness of the reasoning process w.r.t.
  the generated code and analyse the steps of the multi-hop search without
  relying on external solvers. Our methods achieve SOTA results on 7
  out of 9 diverse reasoning benchmarks. We also show that model
  faithfulness positively correlates with overall performance and further
  demonstrate that {\ours} allows pinpointing the decisive factors
  sufficient for and leading to the correct answer with optimal reasoning during
  the multi-hop search.
paper_authors._id: 67119ddcfe58fd40dddc6adb
paper_authors.name: Erik Arakelyan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T12:52:42.860Z
paper_authors._id: 624cdbc55ee9cb3d4557ca21
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649204144832-noauth.jpeg
paper_authors.fullname: Erik Arakelyan
paper_authors.type: user
paper_authors.user: osoblanco
paper_authors._id: 67119ddcfe58fd40dddc6adc
paper_authors.name: Pasquale Minervini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T15:51:38.242Z
paper_authors._id: 61001311e043e15c13412d30
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
paper_authors.fullname: Pasquale Minervini
paper_authors.type: user
paper_authors.user: pminervini
paper_authors._id: 67119ddcfe58fd40dddc6add
paper_authors.name: Pat Verga
paper_authors._id: 67119ddcfe58fd40dddc6ade
paper_authors.name: Patrick Lewis
paper_authors._id: 67119ddcfe58fd40dddc6adf
paper_authors.name: Isabelle Augenstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:38:20.318Z
paper_authors._id: 608918b7df398c3b285ce960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1621507769190-608918b7df398c3b285ce960.jpeg
paper_authors.fullname: Isabelle Augenstein
paper_authors.type: user
paper_authors.user: IAugenstein
paper_title: FLARE: Faithful Logic-Aided Reasoning and Exploration
paper_summary: Modern Question Answering (QA) and Reasoning approaches based on Large
  Language Models (LLMs) commonly use prompting techniques, such as
  Chain-of-Thought (CoT), assuming the resulting generation will have a more
  granular exploration and reasoning over the question space and scope. However,
  such methods struggle with generating outputs that are faithful to the
  intermediate chain of reasoning produced by the model. On the other end of the
  spectrum, neuro-symbolic methods such as Faithful CoT (F-CoT) propose to
  combine LLMs with external symbolic solvers. While such approaches boast a high
  degree of faithfulness, they usually require a model trained for code
  generation and struggle with tasks that are ambiguous or hard to formalise
  strictly. We introduce Faithful Logic-Aided
  Reasoning and Exploration (\ours), a novel
  interpretable approach for traversing the problem space using task
  decompositions. We use the LLM to plan a solution, soft-formalise the query
  into facts and predicates using a logic programming code and simulate that code
  execution using an exhaustive multi-hop search over the defined space. Our
  method allows us to compute the faithfulness of the reasoning process w.r.t.
  the generated code and analyse the steps of the multi-hop search without
  relying on external solvers. Our methods achieve SOTA results on 7
  out of 9 diverse reasoning benchmarks. We also show that model
  faithfulness positively correlates with overall performance and further
  demonstrate that {\ours} allows pinpointing the decisive factors
  sufficient for and leading to the correct answer with optimal reasoning during
  the multi-hop search.
paper_ai_summary: A novel logic-aided reasoning method using LLMs achieves state-of-the-art results on diverse reasoning benchmarks by soft-formalizing queries and simulating code execution without external solvers.
paper_ai_keywords: Large Language Models Chain-of-Thought Faithful CoT logic programming multi-hop search reasoning benchmarks model faithfulness