title: OctoPack: Instruction Tuning Code Large Language Models
summary: Finetuning large language models (LLMs) on instructions leads to vast
  performance improvements on natural language tasks. We apply instruction tuning
  using code, leveraging the natural structure of Git commits, which pair code
  changes with human instructions. We compile CommitPack: 4 terabytes of Git
  commits across 350 programming languages. We benchmark CommitPack against other
  natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B
  parameter StarCoder model, and achieve state-of-the-art performance among
  models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2%
  pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark
  to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis)
  across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models,
  OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among
  all permissive models, demonstrating CommitPack's benefits in generalizing to a
  wider set of languages and natural coding tasks. Code, models and data are
  freely available at https://github.com/bigcode-project/octopack.
paper_authors._id: 64dacfd68da011d65603631c
paper_authors.name: Niklas Muennighoff
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-15T01:19:14.006Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 64dacfd68da011d65603631d
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-17T13:53:46.154Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 64dacfd68da011d65603631e
paper_authors.name: Armel Zebaze
paper_authors._id: 64dacfd68da011d65603631f
paper_authors.name: Qinkai Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:57:37.981Z
paper_authors._id: 6231576e92e83fd1179ac3f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664543160657-6231576e92e83fd1179ac3f0.jpeg
paper_authors.fullname: Qinkai Zheng
paper_authors.type: user
paper_authors.user: Stanislas
paper_authors._id: 64dacfd68da011d656036320
paper_authors.name: Binyuan Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:57:56.438Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 64dacfd68da011d656036321
paper_authors.name: Terry Yue Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:58:05.766Z
paper_authors._id: 62b7fb545233925f253531c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
paper_authors.fullname: Terry Yue Zhuo
paper_authors.type: user
paper_authors.user: terryyz
paper_authors._id: 64dacfd68da011d656036322
paper_authors.name: Swayam Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:58:25.286Z
paper_authors._id: 62f47c093561a52aa5a67c90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f47c093561a52aa5a67c90/0JbnPzHI345sqnrmrxmCw.jpeg
paper_authors.fullname: Swayam Singh
paper_authors.type: user
paper_authors.user: rootacess
paper_authors._id: 64dacfd68da011d656036323
paper_authors.name: Xiangru Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:58:40.805Z
paper_authors._id: 64709e80d9360cd9d8e7073d
paper_authors.avatarUrl: /avatars/ec37977b9a10e823e0d5bb73d152f253.svg
paper_authors.fullname: tang xiangrun
paper_authors.type: user
paper_authors.user: stxrun
paper_authors._id: 64dacfd68da011d656036324
paper_authors.name: Leandro von Werra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:59:12.417Z
paper_authors._id: 5e48005437cb5b49818287a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
paper_authors.fullname: Leandro von Werra
paper_authors.type: user
paper_authors.user: lvwerra
paper_authors._id: 64dacfd68da011d656036325
paper_authors.name: Shayne Longpre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T12:59:22.573Z
paper_authors._id: 61f4283a81c4d30f58140242
paper_authors.avatarUrl: /avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
paper_authors.fullname: Shayne Longpre
paper_authors.type: user
paper_authors.user: Shayne
paper_title: OctoPack: Instruction Tuning Code Large Language Models
paper_summary: Finetuning large language models (LLMs) on instructions leads to vast
  performance improvements on natural language tasks. We apply instruction tuning
  using code, leveraging the natural structure of Git commits, which pair code
  changes with human instructions. We compile CommitPack: 4 terabytes of Git
  commits across 350 programming languages. We benchmark CommitPack against other
  natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B
  parameter StarCoder model, and achieve state-of-the-art performance among
  models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2%
  pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark
  to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis)
  across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models,
  OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among
  all permissive models, demonstrating CommitPack's benefits in generalizing to a
  wider set of languages and natural coding tasks. Code, models and data are
  freely available at https://github.com/bigcode-project/octopack.