title: CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive
    Program Synthesis
summary: Inductive program synthesis, or programming by example, requires synthesizing
  functions from input-output examples that generalize to unseen inputs. While
  large language model agents have shown promise in programming tasks guided by
  natural language, their ability to perform inductive program synthesis is
  underexplored. Existing evaluation protocols rely on static sets of examples
  and held-out tests, offering no feedback when synthesized functions are
  incorrect and failing to reflect real-world scenarios such as reverse
  engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge,
  a new evaluation framework where agents interact with a hidden target function
  by querying it with new inputs, synthesizing candidate functions, and
  iteratively refining their solutions using a differential testing oracle. This
  interactive setting encourages agents to perform function calls and
  self-correction based on feedback. We construct the first large-scale benchmark
  for general-purpose inductive program synthesis, featuring 1114 functions.
  Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%,
  highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on
  curated synthesis traces yields up to a 31% relative performance gain. CodeARC
  provides a more realistic and challenging testbed for evaluating LLM-based
  program synthesis and inductive reasoning.
paper_authors._id: 67eb710f1f8a09c48b2e3ba1
paper_authors.name: Anjiang Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:46:42.109Z
paper_authors._id: 674286496efe2b931f7ce354
paper_authors.avatarUrl: /avatars/8f920618b777dbff5f2a117ebd9e9caa.svg
paper_authors.fullname: Anjiang Wei
paper_authors.type: user
paper_authors.user: anjiangwei
paper_authors._id: 67eb710f1f8a09c48b2e3ba2
paper_authors.name: Tarun Suresh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:23:41.048Z
paper_authors._id: 65e7bb35e5e78134ab049942
paper_authors.avatarUrl: /avatars/3c0972f0d59e51ebb5c218ee736d4458.svg
paper_authors.fullname: Tarun Suresh
paper_authors.type: user
paper_authors.user: tarsur909
paper_authors._id: 67eb710f1f8a09c48b2e3ba3
paper_authors.name: Jiannan Cao
paper_authors._id: 67eb710f1f8a09c48b2e3ba4
paper_authors.name: Naveen Kannan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:26:07.373Z
paper_authors._id: 67edab8d8f7432ed6fdcfdcc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/TK7ombihhpoc8uklxyai4.png
paper_authors.fullname: Naveen Kannan
paper_authors.type: user
paper_authors.user: naveenkc
paper_authors._id: 67eb710f1f8a09c48b2e3ba5
paper_authors.name: Yuheng Wu
paper_authors._id: 67eb710f1f8a09c48b2e3ba6
paper_authors.name: Kai Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:43:08.150Z
paper_authors._id: 65de7628deee79773f0f46f6
paper_authors.avatarUrl: /avatars/6c509dbe96e47b47271eb74178c1c9ba.svg
paper_authors.fullname: Kai Yan
paper_authors.type: user
paper_authors.user: kaiyan289
paper_authors._id: 67eb710f1f8a09c48b2e3ba7
paper_authors.name: Thiago S. F. X. Teixeira
paper_authors._id: 67eb710f1f8a09c48b2e3ba8
paper_authors.name: Ke Wang
paper_authors._id: 67eb710f1f8a09c48b2e3ba9
paper_authors.name: Alex Aiken
paper_title: CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive
    Program Synthesis
paper_summary: Inductive program synthesis, or programming by example, requires synthesizing
  functions from input-output examples that generalize to unseen inputs. While
  large language model agents have shown promise in programming tasks guided by
  natural language, their ability to perform inductive program synthesis is
  underexplored. Existing evaluation protocols rely on static sets of examples
  and held-out tests, offering no feedback when synthesized functions are
  incorrect and failing to reflect real-world scenarios such as reverse
  engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge,
  a new evaluation framework where agents interact with a hidden target function
  by querying it with new inputs, synthesizing candidate functions, and
  iteratively refining their solutions using a differential testing oracle. This
  interactive setting encourages agents to perform function calls and
  self-correction based on feedback. We construct the first large-scale benchmark
  for general-purpose inductive program synthesis, featuring 1114 functions.
  Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%,
  highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on
  curated synthesis traces yields up to a 31% relative performance gain. CodeARC
  provides a more realistic and challenging testbed for evaluating LLM-based
  program synthesis and inductive reasoning.
paper_ai_summary: CodeARC introduces an interactive evaluation framework to assess the ability of large language model agents in inductive program synthesis using differential testing and iterative refinement.
paper_ai_keywords: inductive program synthesis programming by example large language model agents natural language evaluation framework CodeARC Code Abstraction and Reasoning Challenge hidden target function differential testing oracle function calls self-correction large-scale benchmark model evaluation fine-tuning LLaMA-3.1-8B-Instruct inductive reasoning