title: OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
summary: Since the advent of reasoning-based large language models, many have found
  great success from distilling reasoning capabilities into student models. Such
  techniques have significantly bridged the gap between reasoning and standard
  LLMs on coding tasks. Despite this, much of the progress on distilling
  reasoning models remains locked behind proprietary datasets or lacks details on
  data curation, filtering and subsequent training. To address this, we construct
  a superior supervised fine-tuning (SFT) dataset that we use to achieve
  state-of-the-art coding capability results in models of various sizes. Our
  distilled models use only SFT to achieve 61.8% on LiveCodeBench and 24.6% on
  CodeContests, surpassing alternatives trained with reinforcement learning. We
  then perform analysis on the data sources used to construct our dataset, the
  impact of code execution filtering, and the importance of instruction/solution
  diversity. We observe that execution filtering negatively affected benchmark
  accuracy, leading us to prioritize instruction diversity over solution
  correctness. Finally, we also analyze the token efficiency and reasoning
  patterns utilized by these models. We will open-source these datasets and
  distilled models to the community.
paper_authors._id: 67edf81f5e87fcaa485b0df5
paper_authors.name: Wasi Uddin Ahmad
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:48.768Z
paper_authors._id: 67ec45ba92673a326c067b52
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PcD60kFRczOblxgc1bWcl.png
paper_authors.fullname: Wasi Uddin Ahmad
paper_authors.type: user
paper_authors.user: wasiuddina
paper_authors._id: 67edf81f5e87fcaa485b0df6
paper_authors.name: Sean Narenthiran
paper_authors._id: 67edf81f5e87fcaa485b0df7
paper_authors.name: Somshubra Majumdar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T07:52:08.410Z
paper_authors._id: 61eb5b2f55d896e8b6c1c376
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652937892550-61eb5b2f55d896e8b6c1c376.jpeg
paper_authors.fullname: Somshubra Majumdar
paper_authors.type: user
paper_authors.user: smajumdar
paper_authors._id: 67edf81f5e87fcaa485b0df8
paper_authors.name: Aleksander Ficek
paper_authors._id: 67edf81f5e87fcaa485b0df9
paper_authors.name: Siddhartha Jain
paper_authors._id: 67edf81f5e87fcaa485b0dfa
paper_authors.name: Jocelyn Huang
paper_authors._id: 67edf81f5e87fcaa485b0dfb
paper_authors.name: Vahid Noroozi
paper_authors._id: 67edf81f5e87fcaa485b0dfc
paper_authors.name: Boris Ginsburg
paper_title: OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
paper_summary: Since the advent of reasoning-based large language models, many have found
  great success from distilling reasoning capabilities into student models. Such
  techniques have significantly bridged the gap between reasoning and standard
  LLMs on coding tasks. Despite this, much of the progress on distilling
  reasoning models remains locked behind proprietary datasets or lacks details on
  data curation, filtering and subsequent training. To address this, we construct
  a superior supervised fine-tuning (SFT) dataset that we use to achieve
  state-of-the-art coding capability results in models of various sizes. Our
  distilled models use only SFT to achieve 61.8% on LiveCodeBench and 24.6% on
  CodeContests, surpassing alternatives trained with reinforcement learning. We
  then perform analysis on the data sources used to construct our dataset, the
  impact of code execution filtering, and the importance of instruction/solution
  diversity. We observe that execution filtering negatively affected benchmark
  accuracy, leading us to prioritize instruction diversity over solution
  correctness. Finally, we also analyze the token efficiency and reasoning
  patterns utilized by these models. We will open-source these datasets and
  distilled models to the community.
paper_ai_summary: A superior SFT dataset improves coding capabilities in distilled reasoning models surpassing reinforcement learning alternatives.
paper_ai_keywords: supervised fine-tuning LiveCodeBench CodeContests reinforcement learning code execution filtering instruction diversity solution correctness token efficiency reasoning patterns