Building A Proof-Oriented Programmer That Is 64% Better Than GPT-4o
  Under Data Scarsity
Existing LMs struggle with proof-oriented programming due to data scarcity,
which manifest in two key ways: (1) a lack of sufficient corpora for
proof-oriented programming languages such as F*, and (2) the absence of
large-scale, project-level proof-oriented implementations that can teach the
model the intricate reasoning process when performing proof-oriented
programming. We present the first on synthetic data augmentation for project
level proof oriented programming for both generation and repair. Our method
addresses data scarcity by synthesizing basic proof-oriented programming
problems for proficiency in that language; incorporating diverse coding data
for reasoning capability elicitation and creating new proofs and repair data
within existing repositories. This approach enables language models to both
synthesize and repair proofs for function- and repository-level code. We show
that our fine-tuned 14B parameter model, PoPilot, can exceed the performance of
the models that outperforms GPT-4o in project-level proof-oriented programming
by 64% relative margin, and can improve GPT-4o's performance by 54% by
repairing its outputs over GPT-4o's self-repair.
67b3f8cc1bfe04e82830b752
Dylan Zhang
67b3f8cc1bfe04e82830b753
Justin Wang
67b3f8cc1bfe04e82830b754
Tianran Sun
Building A Proof-Oriented Programmer That Is 64% Better Than GPT-4o
  Under Data Scarsity
Existing LMs struggle with proof-oriented programming due to data scarcity,
which manifest in two key ways: (1) a lack of sufficient corpora for
proof-oriented programming languages such as F*, and (2) the absence of
large-scale, project-level proof-oriented implementations that can teach the
model the intricate reasoning process when performing proof-oriented
programming. We present the first on synthetic data augmentation for project
level proof oriented programming for both generation and repair. Our method
addresses data scarcity by synthesizing basic proof-oriented programming
problems for proficiency in that language; incorporating diverse coding data
for reasoning capability elicitation and creating new proofs and repair data
within existing repositories. This approach enables language models to both
synthesize and repair proofs for function- and repository-level code. We show
that our fine-tuned 14B parameter model, PoPilot, can exceed the performance of
the models that outperforms GPT-4o in project-level proof-oriented programming
by 64% relative margin, and can improve GPT-4o's performance by 54% by
repairing its outputs over GPT-4o's self-repair.
A method for synthetic data augmentation in proof-oriented programming enhances model performance through basic problem synthesis and diverse code incorporation.
LMs proof-oriented programming F* synthetic data augmentation project level proof oriented programming proof synthesis repair data PoPilot model fine-tuning GPT-4o self-repair