RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Recent Large Language Models (LLMs) have reported high accuracy on reasoning
benchmarks. However, it is still unclear whether the observed results arise
from true reasoning or from statistical recall of the training set. Inspired by
the ladder of causation (Pearl, 2009) and its three levels (associations,
interventions and counterfactuals), this paper introduces RE-IMAGINE, a
framework to characterize a hierarchy of reasoning ability in LLMs, alongside
an automated pipeline to generate problem variations at different levels of the
hierarchy. By altering problems in an intermediate symbolic representation,
RE-IMAGINE generates arbitrarily many problems that are not solvable using
memorization alone. Moreover, the framework is general and can work across
reasoning domains, including math, code, and logic. We demonstrate our
framework on four widely-used benchmarks to evaluate several families of LLMs,
and observe reductions in performance when the models are queried with problem
variations. These assessments indicate a degree of reliance on statistical
recall for past performance, and open the door to further research targeting
skills across the reasoning hierarchy.
68546a187bc8d012d4ca991f
Xinnuo Xu
68546a187bc8d012d4ca9920
Rachel Lawrence
68546a187bc8d012d4ca9921
Kshitij Dubey
claimed_verified
2025-06-20T12:19:19.484Z
669e070afbc9ba8a128a2807
/avatars/babae90dc758a865dfb76a710ac1da5f.svg
KD
user
Ksh000
68546a187bc8d012d4ca9922
Atharva Pandey
68546a187bc8d012d4ca9923
Risa Ueno
68546a187bc8d012d4ca9924
Fabian Falck
68546a187bc8d012d4ca9925
Aditya V. Nori
68546a187bc8d012d4ca9926
Rahul Sharma
68546a187bc8d012d4ca9927
Amit Sharma
68546a187bc8d012d4ca9928
Javier Gonzalez
RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Recent Large Language Models (LLMs) have reported high accuracy on reasoning
benchmarks. However, it is still unclear whether the observed results arise
from true reasoning or from statistical recall of the training set. Inspired by
the ladder of causation (Pearl, 2009) and its three levels (associations,
interventions and counterfactuals), this paper introduces RE-IMAGINE, a
framework to characterize a hierarchy of reasoning ability in LLMs, alongside
an automated pipeline to generate problem variations at different levels of the
hierarchy. By altering problems in an intermediate symbolic representation,
RE-IMAGINE generates arbitrarily many problems that are not solvable using
memorization alone. Moreover, the framework is general and can work across
reasoning domains, including math, code, and logic. We demonstrate our
framework on four widely-used benchmarks to evaluate several families of LLMs,
and observe reductions in performance when the models are queried with problem
variations. These assessments indicate a degree of reliance on statistical
recall for past performance, and open the door to further research targeting
skills across the reasoning hierarchy.
RE-IMAGINE evaluates the reasoning abilities of Large Language Models by generating variations of problems that cannot be solved by memorization, indicating reliance on statistical recall.
Large Language Models reasoning benchmarks ladder of causation RE-IMAGINE reasoning hierarchy symbolic representation problem variations reasoning domains memorization performance evaluation