title: Magicoder: Source Code Is All You Need
summary: We introduce Magicoder, a series of fully open-source (code, weights, and
  data) Large Language Models (LLMs) for code that significantly closes the gap
  with top code models while having no more than 7B parameters. Magicoder models
  are trained on 75K synthetic instruction data using OSS-Instruct, a novel
  approach to enlightening LLMs with open-source code snippets to generate
  high-quality instruction data for code. Our main motivation is to mitigate the
  inherent bias of the synthetic data generated by LLMs by empowering them with a
  wealth of open-source references for the production of more diverse, realistic,
  and controllable data. The orthogonality of OSS-Instruct and other data
  generation methods like Evol-Instruct further enables us to build an enhanced
  MagicoderS. Both Magicoder and MagicoderS substantially outperform
  state-of-the-art code models with similar or even larger sizes on a wide range
  of coding benchmarks, including Python text-to-code generation, multilingual
  coding, and data-science program completion. Notably, MagicoderS-CL-7B based on
  CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in
  pass@1). Overall, OSS-Instruct opens a new direction for low-bias and
  high-quality instruction tuning using abundant open-source references.
paper_authors._id: 656e8cd31677b468ff6b8379
paper_authors.name: Yuxiang Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-05T08:40:19.196Z
paper_authors._id: 632a176259950c1d279d5ea7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
paper_authors.fullname: Yuxiang Wei
paper_authors.type: user
paper_authors.user: yuxiang630
paper_authors._id: 656e8cd31677b468ff6b837a
paper_authors.name: Zhe Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-05T14:13:33.245Z
paper_authors._id: 637bc237be63f60f4ffe2e67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637bc237be63f60f4ffe2e67/xitcPHGDiUaA-NDakitGt.png
paper_authors.fullname: Zhe Wang
paper_authors.type: user
paper_authors.user: zhewang
paper_authors._id: 656e8cd31677b468ff6b837b
paper_authors.name: Jiawei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:47:19.051Z
paper_authors._id: 644b584a9279988e0cbeb664
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644b584a9279988e0cbeb664/fz4pn7HeXXfpvz23v8E8t.png
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: ganler
paper_authors._id: 656e8cd31677b468ff6b837c
paper_authors.name: Yifeng Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:47:26.751Z
paper_authors._id: 64ff904013f154652687bd80
paper_authors.avatarUrl: /avatars/a04deda1d5c9db0c15a9311806a4b2dc.svg
paper_authors.fullname: Yifeng Ding
paper_authors.type: user
paper_authors.user: YifengDing
paper_authors._id: 656e8cd31677b468ff6b837d
paper_authors.name: Lingming Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-06T10:25:42.077Z
paper_authors._id: 656f473c14fa8cfccd14559e
paper_authors.avatarUrl: /avatars/8f4fef3d835a7a11c2ab66dbf04f3424.svg
paper_authors.fullname: Lingming Zhang
paper_authors.type: user
paper_authors.user: lingming
paper_title: Magicoder: Source Code Is All You Need
paper_summary: We introduce Magicoder, a series of fully open-source (code, weights, and
  data) Large Language Models (LLMs) for code that significantly closes the gap
  with top code models while having no more than 7B parameters. Magicoder models
  are trained on 75K synthetic instruction data using OSS-Instruct, a novel
  approach to enlightening LLMs with open-source code snippets to generate
  high-quality instruction data for code. Our main motivation is to mitigate the
  inherent bias of the synthetic data generated by LLMs by empowering them with a
  wealth of open-source references for the production of more diverse, realistic,
  and controllable data. The orthogonality of OSS-Instruct and other data
  generation methods like Evol-Instruct further enables us to build an enhanced
  MagicoderS. Both Magicoder and MagicoderS substantially outperform
  state-of-the-art code models with similar or even larger sizes on a wide range
  of coding benchmarks, including Python text-to-code generation, multilingual
  coding, and data-science program completion. Notably, MagicoderS-CL-7B based on
  CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in
  pass@1). Overall, OSS-Instruct opens a new direction for low-bias and
  high-quality instruction tuning using abundant open-source references.
paper_ai_keywords: Large Language Models (LLMs) Fully open-source Code Parameters Synthetic instruction data OSS-Instruct Pretrained autoencoders Open-source code snippets Diverse Realistic Controllable data Coding benchmarks Python text-to-code generation Multilingual coding Data-science program completion HumanEval+ CodeLlama ChatGPT