title: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
summary: Large language models (LLMs) for code have become indispensable in various
  domains, including code generation, reasoning tasks and agent systems.While
  open-access code LLMs are increasingly approaching the performance levels of
  proprietary models, high-quality code LLMs suitable for rigorous scientific
  investigation, particularly those with reproducible data processing pipelines
  and transparent training protocols, remain limited. The scarcity is due to
  various challenges, including resource constraints, ethical considerations, and
  the competitive advantages of keeping models advanced. To address the gap, we
  introduce OpenCoder, a top-tier code LLM that not only achieves performance
  comparable to leading models but also serves as an ``open cookbook'' for the
  research community. Unlike most prior efforts, we release not only model
  weights and inference code, but also the reproducible training data, complete
  data processing pipeline, rigorous experimental ablation results, and detailed
  training protocols for open scientific research. Through this comprehensive
  release, we identify the key ingredients for building a top-tier code LLM: (1)
  code optimized heuristic rules for data cleaning and methods for data
  deduplication, (2) recall of text corpus related to code and (3) high-quality
  synthetic data in both annealing and supervised fine-tuning stages. By offering
  this level of openness, we aim to broaden access to all aspects of a top-tier
  code LLM, with OpenCoder serving as both a powerful model and an open
  foundation to accelerate research, and enable reproducible advancements in code
  AI.
paper_authors._id: 672d8d720028ad1c50f6c7d2
paper_authors.name: Siming Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:30:08.966Z
paper_authors._id: 65bd066c696d46ec68aca0ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65bd066c696d46ec68aca0ae/EvKLmmxMPle9hMahyj7cn.jpeg
paper_authors.fullname: Siming Huang
paper_authors.type: user
paper_authors.user: Simingh
paper_authors._id: 672d8d720028ad1c50f6c7d3
paper_authors.name: Tianhao Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:30:04.697Z
paper_authors._id: 65084ce79db6e2495cc52cc6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65084ce79db6e2495cc52cc6/MbwRARtrDfjVi_DHZW6-r.jpeg
paper_authors.fullname: Tianhao Cheng
paper_authors.type: user
paper_authors.user: crazycth
paper_authors._id: 672d8d720028ad1c50f6c7d4
paper_authors.name: Jason Klein Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-16T19:47:05.609Z
paper_authors._id: 6731c04d5f55903a1d8c307c
paper_authors.avatarUrl: /avatars/704b1d628c55e3141194b08736f21267.svg
paper_authors.fullname: Jason Klein Liu
paper_authors.type: user
paper_authors.user: jasonkleinlove
paper_authors._id: 672d8d720028ad1c50f6c7d5
paper_authors.name: Jiaran Hao
paper_authors._id: 672d8d720028ad1c50f6c7d6
paper_authors.name: Liuyihan Song
paper_authors._id: 672d8d720028ad1c50f6c7d7
paper_authors.name: Yang Xu
paper_authors._id: 672d8d720028ad1c50f6c7d8
paper_authors.name: J. Yang
paper_authors._id: 672d8d720028ad1c50f6c7d9
paper_authors.name: J. H. Liu
paper_authors._id: 672d8d720028ad1c50f6c7da
paper_authors.name: Chenchen Zhang
paper_authors._id: 672d8d720028ad1c50f6c7db
paper_authors.name: Linzheng Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:30:06.532Z
paper_authors._id: 64ba096e760936217a3ad2e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ba096e760936217a3ad2e2/aNQK83Jg5PsBkY0UDg-RA.jpeg
paper_authors.fullname: Linzheng Chai
paper_authors.type: user
paper_authors.user: Challenging666
paper_authors._id: 672d8d720028ad1c50f6c7dc
paper_authors.name: Ruifeng Yuan
paper_authors._id: 672d8d720028ad1c50f6c7dd
paper_authors.name: Zhaoxiang Zhang
paper_authors._id: 672d8d720028ad1c50f6c7de
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:30:11.161Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 672d8d720028ad1c50f6c7df
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:30:55.612Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 672d8d720028ad1c50f6c7e0
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:31:20.477Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 672d8d720028ad1c50f6c7e1
paper_authors.name: Zili Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-08T04:03:02.446Z
paper_authors._id: 632c59bbea6e62428ab3b480
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c59bbea6e62428ab3b480/XroXmPnGl1fDrSnN1bQ99.jpeg
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: Zenithwang
paper_authors._id: 672d8d720028ad1c50f6c7e2
paper_authors.name: Yuan Qi
paper_authors._id: 672d8d720028ad1c50f6c7e3
paper_authors.name: Yinghui Xu
paper_authors._id: 672d8d720028ad1c50f6c7e4
paper_authors.name: Wei Chu
paper_title: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
paper_summary: Large language models (LLMs) for code have become indispensable in various
  domains, including code generation, reasoning tasks and agent systems.While
  open-access code LLMs are increasingly approaching the performance levels of
  proprietary models, high-quality code LLMs suitable for rigorous scientific
  investigation, particularly those with reproducible data processing pipelines
  and transparent training protocols, remain limited. The scarcity is due to
  various challenges, including resource constraints, ethical considerations, and
  the competitive advantages of keeping models advanced. To address the gap, we
  introduce OpenCoder, a top-tier code LLM that not only achieves performance
  comparable to leading models but also serves as an ``open cookbook'' for the
  research community. Unlike most prior efforts, we release not only model
  weights and inference code, but also the reproducible training data, complete
  data processing pipeline, rigorous experimental ablation results, and detailed
  training protocols for open scientific research. Through this comprehensive
  release, we identify the key ingredients for building a top-tier code LLM: (1)
  code optimized heuristic rules for data cleaning and methods for data
  deduplication, (2) recall of text corpus related to code and (3) high-quality
  synthetic data in both annealing and supervised fine-tuning stages. By offering
  this level of openness, we aim to broaden access to all aspects of a top-tier
  code LLM, with OpenCoder serving as both a powerful model and an open
  foundation to accelerate research, and enable reproducible advancements in code
  AI.
paper_ai_keywords: large language models (LLMs) code generation reasoning tasks agent systems open-access code LLMs proprietary models reproducible data processing pipelines transparent training protocols resource constraints ethical considerations model weights inference code reproducible training data data processing pipeline experimental ablation results training protocols top-tier code LLM code optimized heuristic rules data cleaning methods for data deduplication recall of text corpus related to code high-quality synthetic data annealing supervised fine-tuning open cookbook open foundation reproducible advancements code AI