title: Training Language Models on Synthetic Edit Sequences Improves Code
    Synthesis
summary: Software engineers mainly write code by editing existing programs. In
  contrast, large language models (LLMs) autoregressively synthesize programs in
  a single pass. One explanation for this is the scarcity of open-sourced edit
  data. While high-quality instruction data for code synthesis is already scarce,
  high-quality edit data is even scarcer. To fill this gap, we develop a
  synthetic data generation algorithm called LintSeq. This algorithm refactors
  existing code into a sequence of code edits by using a linter to procedurally
  sample across the error-free insertions that can be used to sequentially write
  programs. It outputs edit sequences as text strings consisting of consecutive
  program diffs. To test LintSeq, we use it to refactor a dataset of instruction
  + program pairs into instruction + program-diff-sequence tuples. Then, we
  instruction finetune a series of smaller LLMs ranging from 2.6B to 14B
  parameters on both the re-factored and original versions of this dataset,
  comparing zero-shot performance on code synthesis benchmarks. We show that
  during repeated sampling, edit sequence finetuned models produce more diverse
  programs than baselines. This results in better inference-time scaling for
  benchmark coverage as a function of samples, i.e. the fraction of problems
  "pass@k" solved by any attempt given "k" tries. For example, on HumanEval
  pass@50, small LLMs finetuned on synthetic edit sequences are competitive with
  GPT-4 and outperform models finetuned on the baseline dataset by +20% (+/-3%)
  in absolute score. Finally, we also pretrain our own tiny LMs for code
  understanding. We show that finetuning tiny models on synthetic code edits
  results in state-of-the-art code synthesis for the on-device model class. Our
  150M parameter edit sequence LM matches or outperforms code models with twice
  as many parameters, both with and without repeated sampling, including Codex
  and AlphaCode.
paper_authors._id: 66ff7b3d43510cc1bd347770
paper_authors.name: Ulyana Piterbarg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:16:39.075Z
paper_authors._id: 6337054e0267ebcf02637018
paper_authors.avatarUrl: /avatars/89ad12d6643f18706050a92efccd92e6.svg
paper_authors.fullname: Ulyana Piterbarg
paper_authors.type: user
paper_authors.user: upiter
paper_authors._id: 66ff7b3d43510cc1bd347771
paper_authors.name: Lerrel Pinto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:16:45.575Z
paper_authors._id: 6388cdb6685efaedd15dc0f3
paper_authors.avatarUrl: /avatars/b2b74454bf86bf61fd90ba088644f547.svg
paper_authors.fullname: Lerrel Pinto
paper_authors.type: user
paper_authors.user: lerrel
paper_authors._id: 66ff7b3d43510cc1bd347772
paper_authors.name: Rob Fergus
paper_title: Training Language Models on Synthetic Edit Sequences Improves Code
    Synthesis
paper_summary: Software engineers mainly write code by editing existing programs. In
  contrast, large language models (LLMs) autoregressively synthesize programs in
  a single pass. One explanation for this is the scarcity of open-sourced edit
  data. While high-quality instruction data for code synthesis is already scarce,
  high-quality edit data is even scarcer. To fill this gap, we develop a
  synthetic data generation algorithm called LintSeq. This algorithm refactors
  existing code into a sequence of code edits by using a linter to procedurally
  sample across the error-free insertions that can be used to sequentially write
  programs. It outputs edit sequences as text strings consisting of consecutive
  program diffs. To test LintSeq, we use it to refactor a dataset of instruction
  + program pairs into instruction + program-diff-sequence tuples. Then, we
  instruction finetune a series of smaller LLMs ranging from 2.6B to 14B
  parameters on both the re-factored and original versions of this dataset,
  comparing zero-shot performance on code synthesis benchmarks. We show that
  during repeated sampling, edit sequence finetuned models produce more diverse
  programs than baselines. This results in better inference-time scaling for
  benchmark coverage as a function of samples, i.e. the fraction of problems
  "pass@k" solved by any attempt given "k" tries. For example, on HumanEval
  pass@50, small LLMs finetuned on synthetic edit sequences are competitive with
  GPT-4 and outperform models finetuned on the baseline dataset by +20% (+/-3%)
  in absolute score. Finally, we also pretrain our own tiny LMs for code
  understanding. We show that finetuning tiny models on synthetic code edits
  results in state-of-the-art code synthesis for the on-device model class. Our
  150M parameter edit sequence LM matches or outperforms code models with twice
  as many parameters, both with and without repeated sampling, including Codex
  and AlphaCode.
paper_ai_keywords: large language models (LLMs) autoregressive synthesis edit data high-quality instruction data synthetic data generation algorithm LintSeq refactoring code edits linter program diffs inference-time scaling benchmark coverage HumanEval pass@k GPT-4 tiny LMs code understanding state-of-the-art code synthesis