FLARE: Faithful Logic-Aided Reasoning and Exploration
Modern Question Answering (QA) and Reasoning approaches based on Large
Language Models (LLMs) commonly use prompting techniques, such as
Chain-of-Thought (CoT), assuming the resulting generation will have a more
granular exploration and reasoning over the question space and scope. However,
such methods struggle with generating outputs that are faithful to the
intermediate chain of reasoning produced by the model. On the other end of the
spectrum, neuro-symbolic methods such as Faithful CoT (F-CoT) propose to
combine LLMs with external symbolic solvers. While such approaches boast a high
degree of faithfulness, they usually require a model trained for code
generation and struggle with tasks that are ambiguous or hard to formalise
strictly. We introduce Faithful Logic-Aided
Reasoning and Exploration (\ours), a novel
interpretable approach for traversing the problem space using task
decompositions. We use the LLM to plan a solution, soft-formalise the query
into facts and predicates using a logic programming code and simulate that code
execution using an exhaustive multi-hop search over the defined space. Our
method allows us to compute the faithfulness of the reasoning process w.r.t.
the generated code and analyse the steps of the multi-hop search without
relying on external solvers. Our methods achieve SOTA results on 7
out of 9 diverse reasoning benchmarks. We also show that model
faithfulness positively correlates with overall performance and further
demonstrate that {\ours} allows pinpointing the decisive factors
sufficient for and leading to the correct answer with optimal reasoning during
the multi-hop search.
67119ddcfe58fd40dddc6adb
Erik Arakelyan
claimed_verified
2024-10-18T12:52:42.860Z
624cdbc55ee9cb3d4557ca21
https://cdn-avatars.huggingface.co/v1/production/uploads/1649204144832-noauth.jpeg
Erik Arakelyan
user
osoblanco
67119ddcfe58fd40dddc6adc
Pasquale Minervini
claimed_verified
2025-02-10T15:51:38.242Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
67119ddcfe58fd40dddc6add
Pat Verga
67119ddcfe58fd40dddc6ade
Patrick Lewis
67119ddcfe58fd40dddc6adf
Isabelle Augenstein
claimed_verified
2025-02-20T09:38:20.318Z
608918b7df398c3b285ce960
https://cdn-avatars.huggingface.co/v1/production/uploads/1621507769190-608918b7df398c3b285ce960.jpeg
Isabelle Augenstein
user
IAugenstein
FLARE: Faithful Logic-Aided Reasoning and Exploration
Modern Question Answering (QA) and Reasoning approaches based on Large
Language Models (LLMs) commonly use prompting techniques, such as
Chain-of-Thought (CoT), assuming the resulting generation will have a more
granular exploration and reasoning over the question space and scope. However,
such methods struggle with generating outputs that are faithful to the
intermediate chain of reasoning produced by the model. On the other end of the
spectrum, neuro-symbolic methods such as Faithful CoT (F-CoT) propose to
combine LLMs with external symbolic solvers. While such approaches boast a high
degree of faithfulness, they usually require a model trained for code
generation and struggle with tasks that are ambiguous or hard to formalise
strictly. We introduce Faithful Logic-Aided
Reasoning and Exploration (\ours), a novel
interpretable approach for traversing the problem space using task
decompositions. We use the LLM to plan a solution, soft-formalise the query
into facts and predicates using a logic programming code and simulate that code
execution using an exhaustive multi-hop search over the defined space. Our
method allows us to compute the faithfulness of the reasoning process w.r.t.
the generated code and analyse the steps of the multi-hop search without
relying on external solvers. Our methods achieve SOTA results on 7
out of 9 diverse reasoning benchmarks. We also show that model
faithfulness positively correlates with overall performance and further
demonstrate that {\ours} allows pinpointing the decisive factors
sufficient for and leading to the correct answer with optimal reasoning during
the multi-hop search.
A novel logic-aided reasoning method using LLMs achieves state-of-the-art results on diverse reasoning benchmarks by soft-formalizing queries and simulating code execution without external solvers.
Large Language Models Chain-of-Thought Faithful CoT logic programming multi-hop search reasoning benchmarks model faithfulness