title: Diverse Inference and Verification for Advanced Reasoning
summary: Reasoning LLMs such as OpenAI o1, o3 and DeepSeek R1 have made significant
  progress in mathematics and coding, yet find challenging advanced tasks such as
  International Mathematical Olympiad (IMO) combinatorics problems, Abstraction
  and Reasoning Corpus (ARC) puzzles, and Humanity's Last Exam (HLE) questions.
  We use a diverse inference approach that combines multiple models and methods
  at test time. We find that verifying mathematics and code problems, and
  rejection sampling on other problems is simple and effective. We automatically
  verify correctness of solutions to IMO problems by Lean, and ARC puzzles by
  code, and find that best-of-N effectively answers HLE questions. Our approach
  increases answer accuracy on IMO combinatorics problems from 33.3% to 77.8%,
  accuracy on HLE questions from 8% to 37%, and solves 80% of ARC puzzles that
  948 humans could not and 26.5% of ARC puzzles that o3 high compute does not.
  Test-time simulations, reinforcement learning, and meta-learning with inference
  feedback improve generalization by adapting agent graph representations and
  varying prompts, code, and datasets. Our approach is reliable, robust, and
  scalable, and in the spirit of reproducible research, we will make it publicly
  available upon publication.
paper_authors._id: 67b2c1ac0303a07acd3f9443
paper_authors.name: Iddo Drori
paper_authors._id: 67b2c1ac0303a07acd3f9444
paper_authors.name: Gaston Longhitano
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:54:41.134Z
paper_authors._id: 665167c3b1181f7d10bd3b99
paper_authors.avatarUrl: /avatars/443a63003b15c2312bc59fea2e018362.svg
paper_authors.fullname: Gaston Longhitano
paper_authors.type: user
paper_authors.user: glongh
paper_authors._id: 67b2c1ac0303a07acd3f9445
paper_authors.name: Mao Mao
paper_authors._id: 67b2c1ac0303a07acd3f9446
paper_authors.name: Seunghwan Hyun
paper_authors._id: 67b2c1ac0303a07acd3f9447
paper_authors.name: Yuke Zhang
paper_authors._id: 67b2c1ac0303a07acd3f9448
paper_authors.name: Sungjun Park
paper_authors._id: 67b2c1ac0303a07acd3f9449
paper_authors.name: Zachary Meeks
paper_authors._id: 67b2c1ac0303a07acd3f944a
paper_authors.name: Xin-Yu Zhang
paper_authors._id: 67b2c1ac0303a07acd3f944b
paper_authors.name: Ben Segev
paper_authors._id: 67b2c1ac0303a07acd3f944c
paper_authors.name: Howard Yong
paper_authors._id: 67b2c1ac0303a07acd3f944d
paper_authors.name: Nakul Verma
paper_authors._id: 67b2c1ac0303a07acd3f944e
paper_authors.name: Avi Shporer
paper_authors._id: 67b2c1ac0303a07acd3f944f
paper_authors.name: Alon Amit
paper_authors._id: 67b2c1ac0303a07acd3f9450
paper_authors.name: Madeleine Udell
paper_title: Diverse Inference and Verification for Advanced Reasoning
paper_summary: Reasoning LLMs such as OpenAI o1, o3 and DeepSeek R1 have made significant
  progress in mathematics and coding, yet find challenging advanced tasks such as
  International Mathematical Olympiad (IMO) combinatorics problems, Abstraction
  and Reasoning Corpus (ARC) puzzles, and Humanity's Last Exam (HLE) questions.
  We use a diverse inference approach that combines multiple models and methods
  at test time. We find that verifying mathematics and code problems, and
  rejection sampling on other problems is simple and effective. We automatically
  verify correctness of solutions to IMO problems by Lean, and ARC puzzles by
  code, and find that best-of-N effectively answers HLE questions. Our approach
  increases answer accuracy on IMO combinatorics problems from 33.3% to 77.8%,
  accuracy on HLE questions from 8% to 37%, and solves 80% of ARC puzzles that
  948 humans could not and 26.5% of ARC puzzles that o3 high compute does not.
  Test-time simulations, reinforcement learning, and meta-learning with inference
  feedback improve generalization by adapting agent graph representations and
  varying prompts, code, and datasets. Our approach is reliable, robust, and
  scalable, and in the spirit of reproducible research, we will make it publicly
  available upon publication.
paper_ai_keywords: Reasoning LLMs International Mathematical Olympiad (IMO) Abstraction and Reasoning Corpus (ARC) Humanity's Last Exam (HLE) diverse inference approach Lean (verification tool) best-of-N verification correctness test-time simulations reinforcement learning meta-learning inference feedback agent graph representations varying prompts