CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive
  Program Synthesis
Inductive program synthesis, or programming by example, requires synthesizing
functions from input-output examples that generalize to unseen inputs. While
large language model agents have shown promise in programming tasks guided by
natural language, their ability to perform inductive program synthesis is
underexplored. Existing evaluation protocols rely on static sets of examples
and held-out tests, offering no feedback when synthesized functions are
incorrect and failing to reflect real-world scenarios such as reverse
engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge,
a new evaluation framework where agents interact with a hidden target function
by querying it with new inputs, synthesizing candidate functions, and
iteratively refining their solutions using a differential testing oracle. This
interactive setting encourages agents to perform function calls and
self-correction based on feedback. We construct the first large-scale benchmark
for general-purpose inductive program synthesis, featuring 1114 functions.
Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%,
highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on
curated synthesis traces yields up to a 31% relative performance gain. CodeARC
provides a more realistic and challenging testbed for evaluating LLM-based
program synthesis and inductive reasoning.
67eb710f1f8a09c48b2e3ba1
Anjiang Wei
claimed_verified
2025-04-01T07:46:42.109Z
674286496efe2b931f7ce354
/avatars/8f920618b777dbff5f2a117ebd9e9caa.svg
Anjiang Wei
user
anjiangwei
67eb710f1f8a09c48b2e3ba2
Tarun Suresh
claimed_verified
2025-04-02T08:23:41.048Z
65e7bb35e5e78134ab049942
/avatars/3c0972f0d59e51ebb5c218ee736d4458.svg
Tarun Suresh
user
tarsur909
67eb710f1f8a09c48b2e3ba3
Jiannan Cao
67eb710f1f8a09c48b2e3ba4
Naveen Kannan
claimed_verified
2025-04-11T07:26:07.373Z
67edab8d8f7432ed6fdcfdcc
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/TK7ombihhpoc8uklxyai4.png
Naveen Kannan
user
naveenkc
67eb710f1f8a09c48b2e3ba5
Yuheng Wu
67eb710f1f8a09c48b2e3ba6
Kai Yan
admin_assigned
2025-04-02T09:43:08.150Z
65de7628deee79773f0f46f6
/avatars/6c509dbe96e47b47271eb74178c1c9ba.svg
Kai Yan
user
kaiyan289
67eb710f1f8a09c48b2e3ba7
Thiago S. F. X. Teixeira
67eb710f1f8a09c48b2e3ba8
Ke Wang
67eb710f1f8a09c48b2e3ba9
Alex Aiken
CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive
  Program Synthesis
Inductive program synthesis, or programming by example, requires synthesizing
functions from input-output examples that generalize to unseen inputs. While
large language model agents have shown promise in programming tasks guided by
natural language, their ability to perform inductive program synthesis is
underexplored. Existing evaluation protocols rely on static sets of examples
and held-out tests, offering no feedback when synthesized functions are
incorrect and failing to reflect real-world scenarios such as reverse
engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge,
a new evaluation framework where agents interact with a hidden target function
by querying it with new inputs, synthesizing candidate functions, and
iteratively refining their solutions using a differential testing oracle. This
interactive setting encourages agents to perform function calls and
self-correction based on feedback. We construct the first large-scale benchmark
for general-purpose inductive program synthesis, featuring 1114 functions.
Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%,
highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on
curated synthesis traces yields up to a 31% relative performance gain. CodeARC
provides a more realistic and challenging testbed for evaluating LLM-based
program synthesis and inductive reasoning.
CodeARC introduces an interactive evaluation framework to assess the ability of large language model agents in inductive program synthesis using differential testing and iterative refinement.
inductive program synthesis programming by example large language model agents natural language evaluation framework CodeARC Code Abstraction and Reasoning Challenge hidden target function differential testing oracle function calls self-correction large-scale benchmark model evaluation fine-tuning LLaMA-3.1-8B-Instruct inductive reasoning