ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
We investigate the logical reasoning capabilities of large language models
(LLMs) and their scalability in complex non-monotonic reasoning. To this end,
we introduce ZebraLogic, a comprehensive evaluation framework for assessing LLM
reasoning performance on logic grid puzzles derived from constraint
satisfaction problems (CSPs). ZebraLogic enables the generation of puzzles with
controllable and quantifiable complexity, facilitating a systematic study of
the scaling limits of models such as Llama, o1 models, and DeepSeek-R1. By
encompassing a broad range of search space complexities and diverse logical
constraints, ZebraLogic provides a structured environment to evaluate reasoning
under increasing difficulty.
  Our results reveal a significant decline in accuracy as problem complexity
grows -- a phenomenon we term the curse of complexity. This limitation persists
even with larger models and increased inference-time computation, suggesting
inherent constraints in current LLM reasoning capabilities. Additionally, we
explore strategies to enhance logical reasoning, including Best-of-N sampling,
backtracking mechanisms, and self-verification prompts. Our findings offer
critical insights into the scalability of LLM reasoning, highlight fundamental
limitations, and outline potential directions for improvement.
67a1a649f4aecd0dfc96ebf4
Bill Yuchen Lin
claimed_verified
2025-02-04T09:39:17.972Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
67a1a649f4aecd0dfc96ebf5
Ronan Le Bras
extracted_pending
2025-02-04T05:31:56.722Z
635049104e753c9940fefd71
https://cdn-avatars.huggingface.co/v1/production/uploads/635049104e753c9940fefd71/HgR43XIFw3dneY5ufrAE8.jpeg
Ronan Le Bras
user
ronanlb
67a1a649f4aecd0dfc96ebf6
Kyle Richardson
claimed_verified
2025-02-27T09:18:11.166Z
659c6f5e1d398a238152227d
/avatars/dba97fe8cb825102f1eae97104a71f64.svg
Kyle Richardson
user
yakazimir
67a1a649f4aecd0dfc96ebf7
Ashish Sabharwal
claimed_verified
2025-07-22T07:55:21.795Z
687e87f9b193d0f4c8d605c0
/avatars/1e2fa294cbb2017e380fae6695e35db1.svg
Ashish Sabharwal
user
ashish333
67a1a649f4aecd0dfc96ebf8
Radha Poovendran
67a1a649f4aecd0dfc96ebf9
Peter Clark
admin_assigned
2025-02-04T17:38:40.973Z
64d265cfbe712cda5ab7cc3f
/avatars/caab6fa5764a0271552ae589d352b592.svg
Peter Clarke
user
PeterClarke
67a1a649f4aecd0dfc96ebfa
Yejin Choi
admin_assigned
2025-02-04T17:38:53.950Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
We investigate the logical reasoning capabilities of large language models
(LLMs) and their scalability in complex non-monotonic reasoning. To this end,
we introduce ZebraLogic, a comprehensive evaluation framework for assessing LLM
reasoning performance on logic grid puzzles derived from constraint
satisfaction problems (CSPs). ZebraLogic enables the generation of puzzles with
controllable and quantifiable complexity, facilitating a systematic study of
the scaling limits of models such as Llama, o1 models, and DeepSeek-R1. By
encompassing a broad range of search space complexities and diverse logical
constraints, ZebraLogic provides a structured environment to evaluate reasoning
under increasing difficulty.
  Our results reveal a significant decline in accuracy as problem complexity
grows -- a phenomenon we term the curse of complexity. This limitation persists
even with larger models and increased inference-time computation, suggesting
inherent constraints in current LLM reasoning capabilities. Additionally, we
explore strategies to enhance logical reasoning, including Best-of-N sampling,
backtracking mechanisms, and self-verification prompts. Our findings offer
critical insights into the scalability of LLM reasoning, highlight fundamental
limitations, and outline potential directions for improvement.
LLMs exhibit decreased accuracy in complex non-monotonic reasoning, as demonstrated through the ZebraLogic framework assessing logic grid puzzles.
large language models (LLMs) non-monotonic reasoning ZebraLogic evaluation framework logic grid puzzles constraint satisfaction problems (CSPs) logical reasoning Best-of-N sampling backtracking mechanisms self-verification prompts