title: Generative Evaluation of Complex Reasoning in Large Language Models
summary: With powerful large language models (LLMs) demonstrating superhuman reasoning
  capabilities, a critical question arises: Do LLMs genuinely reason, or do they
  merely recall answers from their extensive, web-scraped training datasets?
  Publicly released benchmarks inevitably become contaminated once incorporated
  into subsequent LLM training sets, undermining their reliability as faithful
  assessments. To address this, we introduce KUMO, a generative evaluation
  framework designed specifically for assessing reasoning in LLMs. KUMO
  synergistically combines LLMs with symbolic engines to dynamically produce
  diverse, multi-turn reasoning tasks that are partially observable and
  adjustable in difficulty. Through an automated pipeline, KUMO continuously
  generates novel tasks across open-ended domains, compelling models to
  demonstrate genuine generalization rather than memorization. We evaluated 23
  state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO,
  benchmarking their reasoning abilities against university students. Our
  findings reveal that many LLMs have outperformed university-level performance
  on easy reasoning tasks, and reasoning-scaled LLMs reach university-level
  performance on complex reasoning challenges. Moreover, LLM performance on KUMO
  tasks correlates strongly with results on newly released real-world reasoning
  benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for
  genuine LLM reasoning capabilities.
paper_authors._id: 67f099de103cb604facd26cd
paper_authors.name: Haowei Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-06T08:11:10.909Z
paper_authors._id: 63453f02a05b51f7ded3c579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
paper_authors.fullname: Andy Lin
paper_authors.type: user
paper_authors.user: pkuHaowei
paper_authors._id: 67f099de103cb604facd26ce
paper_authors.name: Xiangyu Wang
paper_authors._id: 67f099de103cb604facd26cf
paper_authors.name: Ruilin Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:41:34.723Z
paper_authors._id: 66d08a22526dae983ba5da22
paper_authors.avatarUrl: /avatars/9744f8801e5b951ce49e5844c180efd7.svg
paper_authors.fullname: Ruilin Yan
paper_authors.type: user
paper_authors.user: 0o0yrl
paper_authors._id: 67f099de103cb604facd26d0
paper_authors.name: Baizhou Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:41:40.792Z
paper_authors._id: 67345e16b96e9dfdbec150af
paper_authors.avatarUrl: /avatars/d502e901ccdc408a9f47ab9a664e7c81.svg
paper_authors.fullname: Baizhou Huang
paper_authors.type: user
paper_authors.user: BasilHCN
paper_authors._id: 67f099de103cb604facd26d1
paper_authors.name: Haotian Ye
paper_authors._id: 67f099de103cb604facd26d2
paper_authors.name: Jianhua Zhu
paper_authors._id: 67f099de103cb604facd26d3
paper_authors.name: Zihao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:42:33.691Z
paper_authors._id: 6757a1fde1a2dd778051b9e2
paper_authors.avatarUrl: /avatars/bd2b5bb509902e1ad88a4b63e6cd0187.svg
paper_authors.fullname: Zihao Wang
paper_authors.type: user
paper_authors.user: ZihaoWang
paper_authors._id: 67f099de103cb604facd26d4
paper_authors.name: James Zou
paper_authors._id: 67f099de103cb604facd26d5
paper_authors.name: Jianzhu Ma
paper_authors._id: 67f099de103cb604facd26d6
paper_authors.name: Yitao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T07:38:09.311Z
paper_authors._id: 64683a5776bb704aa14588b7
paper_authors.avatarUrl: /avatars/e532756f52c5b95981470ace41a10556.svg
paper_authors.fullname: Yitao Liang
paper_authors.type: user
paper_authors.user: YitaoLiang
paper_title: Generative Evaluation of Complex Reasoning in Large Language Models
paper_summary: With powerful large language models (LLMs) demonstrating superhuman reasoning
  capabilities, a critical question arises: Do LLMs genuinely reason, or do they
  merely recall answers from their extensive, web-scraped training datasets?
  Publicly released benchmarks inevitably become contaminated once incorporated
  into subsequent LLM training sets, undermining their reliability as faithful
  assessments. To address this, we introduce KUMO, a generative evaluation
  framework designed specifically for assessing reasoning in LLMs. KUMO
  synergistically combines LLMs with symbolic engines to dynamically produce
  diverse, multi-turn reasoning tasks that are partially observable and
  adjustable in difficulty. Through an automated pipeline, KUMO continuously
  generates novel tasks across open-ended domains, compelling models to
  demonstrate genuine generalization rather than memorization. We evaluated 23
  state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO,
  benchmarking their reasoning abilities against university students. Our
  findings reveal that many LLMs have outperformed university-level performance
  on easy reasoning tasks, and reasoning-scaled LLMs reach university-level
  performance on complex reasoning challenges. Moreover, LLM performance on KUMO
  tasks correlates strongly with results on newly released real-world reasoning
  benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for
  genuine LLM reasoning capabilities.
paper_ai_summary: KUMO, a generative evaluation framework combining LLMs with symbolic engines, assesses genuine reasoning in large language models through diverse, dynamically produced tasks.
paper_ai_keywords: large language models (LLMs) symbolic engines generative evaluation framework reasoning tasks university students reasoning-scaled LLMs real-world reasoning benchmarks