Enigmata: Scaling Logical Reasoning in Large Language Models with
  Synthetic Verifiable Puzzles
Large Language Models (LLMs), such as OpenAI's o1 and DeepSeek's R1, excel at
advanced reasoning tasks like math and coding via Reinforcement Learning with
Verifiable Rewards (RLVR), but still struggle with puzzles solvable by humans
without domain knowledge. We introduce Enigmata, the first comprehensive suite
tailored for improving LLMs with puzzle reasoning skills. It includes 36 tasks
across seven categories, each with 1) a generator that produces unlimited
examples with controllable difficulty and 2) a rule-based verifier for
automatic evaluation. This generator-verifier design supports scalable,
multi-task RL training, fine-grained analysis, and seamless RLVR integration.
We further propose Enigmata-Eval, a rigorous benchmark, and develop optimized
multi-task RLVR strategies. Our trained model, Qwen2.5-32B-Enigmata,
consistently surpasses o3-mini-high and o1 on the puzzle reasoning benchmarks
like Enigmata-Eval, ARC-AGI (32.8%), and ARC-AGI 2 (0.6%). It also generalizes
well to out-of-domain puzzle benchmarks and mathematical reasoning, with little
multi-tasking trade-off. When trained on larger models like Seed1.5-Thinking
(20B activated parameters and 200B total parameters), puzzle data from Enigmata
further boosts SoTA performance on advanced math and STEM reasoning tasks such
as AIME (2024-2025), BeyondAIME and GPQA (Diamond), showing nice generalization
benefits of Enigmata. This work offers a unified, controllable framework for
advancing logical reasoning in LLMs. Resources of this work can be found at
https://seed-enigmata.github.io.
68353e41f995630ab88c198b
Jiangjie Chen
claimed_verified
2025-05-27T07:49:21.006Z
606ed1884ffe81d1e03e81e5
https://cdn-avatars.huggingface.co/v1/production/uploads/1639375346654-606ed1884ffe81d1e03e81e5.png
Jiangjie Chen
user
jiangjiechen
68353e41f995630ab88c198c
Qianyu He
claimed_verified
2025-05-27T07:49:23.290Z
636b36351340f879a2ec2bb1
/avatars/260a1c15f9c14c967125469072020946.svg
QianyuHe
user
Abbey4799
68353e41f995630ab88c198d
Siyu Yuan
admin_assigned
2025-05-27T10:14:02.514Z
62d62b333bf5e059f7d2b286
https://cdn-avatars.huggingface.co/v1/production/uploads/1668513815771-62d62b333bf5e059f7d2b286.jpeg
Siyu Yuan
user
siyuyuan
68353e41f995630ab88c198e
Aili Chen
claimed_verified
2025-05-29T20:18:46.580Z
63f86b099f87cc3e645b51d9
/avatars/27ca5ba425640bf67474cee871e8e53a.svg
Ellie Chen
user
sheep33333
68353e41f995630ab88c198f
Zhicheng Cai
68353e41f995630ab88c1990
Weinan Dai
68353e41f995630ab88c1991
Hongli Yu
claimed_verified
2025-07-08T08:07:19.687Z
66c58a7e960ea4e66c632075
/avatars/527ad90f605e5aaf76298204dfa67d55.svg
hlyu
user
huiyeruzhou
68353e41f995630ab88c1992
Qiying Yu
68353e41f995630ab88c1993
Xuefeng Li
68353e41f995630ab88c1994
Jiaze Chen
68353e41f995630ab88c1995
Hao Zhou
68353e41f995630ab88c1996
Mingxuan Wang
Enigmata: Scaling Logical Reasoning in Large Language Models with
  Synthetic Verifiable Puzzles
Large Language Models (LLMs), such as OpenAI's o1 and DeepSeek's R1, excel at
advanced reasoning tasks like math and coding via Reinforcement Learning with
Verifiable Rewards (RLVR), but still struggle with puzzles solvable by humans
without domain knowledge. We introduce Enigmata, the first comprehensive suite
tailored for improving LLMs with puzzle reasoning skills. It includes 36 tasks
across seven categories, each with 1) a generator that produces unlimited
examples with controllable difficulty and 2) a rule-based verifier for
automatic evaluation. This generator-verifier design supports scalable,
multi-task RL training, fine-grained analysis, and seamless RLVR integration.
We further propose Enigmata-Eval, a rigorous benchmark, and develop optimized
multi-task RLVR strategies. Our trained model, Qwen2.5-32B-Enigmata,
consistently surpasses o3-mini-high and o1 on the puzzle reasoning benchmarks
like Enigmata-Eval, ARC-AGI (32.8%), and ARC-AGI 2 (0.6%). It also generalizes
well to out-of-domain puzzle benchmarks and mathematical reasoning, with little
multi-tasking trade-off. When trained on larger models like Seed1.5-Thinking
(20B activated parameters and 200B total parameters), puzzle data from Enigmata
further boosts SoTA performance on advanced math and STEM reasoning tasks such
as AIME (2024-2025), BeyondAIME and GPQA (Diamond), showing nice generalization
benefits of Enigmata. This work offers a unified, controllable framework for
advancing logical reasoning in LLMs. Resources of this work can be found at
https://seed-enigmata.github.io.
Enigmata is a comprehensive suite for improving LLMs in puzzle reasoning through scalable multi-task RL training, leading to better performance on benchmarks and advanced math tasks.
Large Language Models OpenAI's o1 DeepSeek's R1 Reinforcement Learning with Verifiable Rewards Enigmata Enigmata-Eval multi-task RL training puzzle reasoning rule-based verifier ARC-AGI Qwen2.5-32B-Enigmata Seed1.5-Thinking AIME BeyondAIME GPQA