title: SelfCodeAlign: Self-Alignment for Code Generation
summary: Instruction tuning is a supervised fine-tuning approach that significantly
  improves the ability of large language models (LLMs) to follow human
  instructions. We propose SelfCodeAlign, the first fully transparent and
  permissive pipeline for self-aligning code LLMs without extensive human
  annotations or distillation. SelfCodeAlign employs the same base model for
  inference throughout the data generation process. It first extracts diverse
  coding concepts from high-quality seed snippets to generate new tasks. It then
  samples multiple responses per task, pairs each with test cases, and validates
  them in a sandbox environment. Finally, passing examples are selected for
  instruction tuning. In our primary experiments, we use SelfCodeAlign with
  CodeQwen1.5-7B to generate a dataset of 74k instruction-response pairs.
  Finetuning on this dataset leads to a model that achieves a 67.1 pass@1 on
  HumanEval+, surpassing CodeLlama-70B-Instruct despite being ten times smaller.
  Across all benchmarks, this finetuned model consistently outperforms the
  original version trained with OctoPack, the previous state-of-the-art method
  for instruction tuning without human annotations or distillation. Additionally,
  we show that SelfCodeAlign is effective across LLMs of various sizes, from 3B
  to 33B, and that the base models can benefit more from alignment with their own
  data distribution. We further validate each component's effectiveness in our
  pipeline, showing that SelfCodeAlign outperforms both direct distillation from
  GPT-4o and leading GPT-3.5-based distillation methods, such as OSS-Instruct and
  Evol-Instruct. SelfCodeAlign has also led to the creation of
  StarCoder2-Instruct, the first fully transparent, permissively licensed, and
  self-aligned code LLM that achieves state-of-the-art coding performance.
paper_authors._id: 6724c3264a49116d8e0ce252
paper_authors.name: Yuxiang Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:21:06.810Z
paper_authors._id: 632a176259950c1d279d5ea7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
paper_authors.fullname: Yuxiang Wei
paper_authors.type: user
paper_authors.user: yuxiang630
paper_authors._id: 6724c3264a49116d8e0ce253
paper_authors.name: Federico Cassano
paper_authors._id: 6724c3264a49116d8e0ce254
paper_authors.name: Jiawei Liu
paper_authors._id: 6724c3264a49116d8e0ce255
paper_authors.name: Yifeng Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:21:04.685Z
paper_authors._id: 64ff904013f154652687bd80
paper_authors.avatarUrl: /avatars/a04deda1d5c9db0c15a9311806a4b2dc.svg
paper_authors.fullname: Yifeng Ding
paper_authors.type: user
paper_authors.user: YifengDing
paper_authors._id: 6724c3264a49116d8e0ce256
paper_authors.name: Naman Jain
paper_authors._id: 6724c3264a49116d8e0ce257
paper_authors.name: Zachary Mueller
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:21:08.691Z
paper_authors._id: 5f05297d5d08220171a0ad7d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594174616206-5f05297d5d08220171a0ad7d.png
paper_authors.fullname: Zachary Mueller
paper_authors.type: user
paper_authors.user: muellerzr
paper_authors._id: 6724c3264a49116d8e0ce258
paper_authors.name: Harm de Vries
paper_authors._id: 6724c3264a49116d8e0ce259
paper_authors.name: Leandro von Werra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:21:11.036Z
paper_authors._id: 5e48005437cb5b49818287a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
paper_authors.fullname: Leandro von Werra
paper_authors.type: user
paper_authors.user: lvwerra
paper_authors._id: 6724c3264a49116d8e0ce25a
paper_authors.name: Arjun Guha
paper_authors._id: 6724c3264a49116d8e0ce25b
paper_authors.name: Lingming Zhang
paper_title: SelfCodeAlign: Self-Alignment for Code Generation
paper_summary: Instruction tuning is a supervised fine-tuning approach that significantly
  improves the ability of large language models (LLMs) to follow human
  instructions. We propose SelfCodeAlign, the first fully transparent and
  permissive pipeline for self-aligning code LLMs without extensive human
  annotations or distillation. SelfCodeAlign employs the same base model for
  inference throughout the data generation process. It first extracts diverse
  coding concepts from high-quality seed snippets to generate new tasks. It then
  samples multiple responses per task, pairs each with test cases, and validates
  them in a sandbox environment. Finally, passing examples are selected for
  instruction tuning. In our primary experiments, we use SelfCodeAlign with
  CodeQwen1.5-7B to generate a dataset of 74k instruction-response pairs.
  Finetuning on this dataset leads to a model that achieves a 67.1 pass@1 on
  HumanEval+, surpassing CodeLlama-70B-Instruct despite being ten times smaller.
  Across all benchmarks, this finetuned model consistently outperforms the
  original version trained with OctoPack, the previous state-of-the-art method
  for instruction tuning without human annotations or distillation. Additionally,
  we show that SelfCodeAlign is effective across LLMs of various sizes, from 3B
  to 33B, and that the base models can benefit more from alignment with their own
  data distribution. We further validate each component's effectiveness in our
  pipeline, showing that SelfCodeAlign outperforms both direct distillation from
  GPT-4o and leading GPT-3.5-based distillation methods, such as OSS-Instruct and
  Evol-Instruct. SelfCodeAlign has also led to the creation of
  StarCoder2-Instruct, the first fully transparent, permissively licensed, and
  self-aligned code LLM that achieves state-of-the-art coding performance.
paper_ai_summary: SelfCodeAlign is a transparent and self-aligning pipeline for enhancing large language models' ability to follow human instructions without human annotations or distillation, significantly improving coding performance across various benchmarks.
paper_ai_keywords: instruction tuning SelfCodeAlign code LLMs seed snippets sandbox environment HumanEval+ CodeQwen1.5-7B CodeLlama-70B-Instruct OctoPack StarCoder2-Instruct