title: Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
summary: Recent studies have been increasingly demonstrating that high-quality data is
  crucial for effective pretraining of language models. However, the precise
  definition of "high-quality" remains underexplored. Focusing on the code
  domain, we introduce Arctic-SnowCoder-1.3B, a data-efficient base code model
  pretrained on 555B tokens through three phases of progressively refined data:
  (1) general pretraining with 500B standard-quality code tokens, preprocessed
  through basic filtering, deduplication, and decontamination, (2) continued
  pretraining with 50B high-quality tokens, selected from phase one by a
  BERT-style quality annotator trained to distinguish good code from random data,
  using positive examples drawn from high-quality code files, along with
  instruction data from Magicoder and StarCoder2-Instruct, and (3) enhanced
  pretraining with 5B synthetic data created by Llama-3.1-70B using phase two
  data as seeds, adapting the Magicoder approach for pretraining. Despite being
  trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art
  performance on BigCodeBench, a coding benchmark focusing on practical and
  challenging programming tasks, compared to similarly sized models trained on no
  more than 1T tokens, outperforming Phi-1.5-1.3B by 36%. Across all evaluated
  benchmarks, Arctic-SnowCoder-1.3B beats StarCoderBase-3B pretrained on 1T
  tokens. Additionally, it matches the performance of leading small base code
  models trained on trillions of tokens. For example, Arctic-SnowCoder-1.3B
  surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a
  benchmark that evaluates function-level code generation, and remains
  competitive on BigCodeBench. Our evaluation presents a comprehensive analysis
  justifying various design choices for Arctic-SnowCoder. Most importantly, we
  find that the key to high-quality data is its alignment with the distribution
  of downstream applications.
paper_authors._id: 66d9403866e970a15fff17d3
paper_authors.name: Yuxiang Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:55:03.115Z
paper_authors._id: 632a176259950c1d279d5ea7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
paper_authors.fullname: Yuxiang Wei
paper_authors.type: user
paper_authors.user: yuxiang630
paper_authors._id: 66d9403866e970a15fff17d4
paper_authors.name: Hojae Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:55:09.640Z
paper_authors._id: 64ef02b9d2b65a3b4b7790ec
paper_authors.avatarUrl: /avatars/2c65a3e65ac1980cfb46340fab51b0a3.svg
paper_authors.fullname: Hojae Han
paper_authors.type: user
paper_authors.user: stovecat
paper_authors._id: 66d9403866e970a15fff17d5
paper_authors.name: Rajhans Samdani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:55:16.759Z
paper_authors._id: 64cae133bb5d195b990ad4be
paper_authors.avatarUrl: /avatars/31f9f298c756b93871f4c011042f0abe.svg
paper_authors.fullname: Rajhans Samdani
paper_authors.type: user
paper_authors.user: rajhans-snowflake
paper_title: Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
paper_summary: Recent studies have been increasingly demonstrating that high-quality data is
  crucial for effective pretraining of language models. However, the precise
  definition of "high-quality" remains underexplored. Focusing on the code
  domain, we introduce Arctic-SnowCoder-1.3B, a data-efficient base code model
  pretrained on 555B tokens through three phases of progressively refined data:
  (1) general pretraining with 500B standard-quality code tokens, preprocessed
  through basic filtering, deduplication, and decontamination, (2) continued
  pretraining with 50B high-quality tokens, selected from phase one by a
  BERT-style quality annotator trained to distinguish good code from random data,
  using positive examples drawn from high-quality code files, along with
  instruction data from Magicoder and StarCoder2-Instruct, and (3) enhanced
  pretraining with 5B synthetic data created by Llama-3.1-70B using phase two
  data as seeds, adapting the Magicoder approach for pretraining. Despite being
  trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art
  performance on BigCodeBench, a coding benchmark focusing on practical and
  challenging programming tasks, compared to similarly sized models trained on no
  more than 1T tokens, outperforming Phi-1.5-1.3B by 36%. Across all evaluated
  benchmarks, Arctic-SnowCoder-1.3B beats StarCoderBase-3B pretrained on 1T
  tokens. Additionally, it matches the performance of leading small base code
  models trained on trillions of tokens. For example, Arctic-SnowCoder-1.3B
  surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a
  benchmark that evaluates function-level code generation, and remains
  competitive on BigCodeBench. Our evaluation presents a comprehensive analysis
  justifying various design choices for Arctic-SnowCoder. Most importantly, we
  find that the key to high-quality data is its alignment with the distribution
  of downstream applications.
paper_ai_keywords: BERT-style quality annotator Code domain Data-efficient base code model Deduplication Decontamination Enhanced pretraining General pretraining High-quality data Instruction data Llama-3.1-70B Magicoder Magicoder approach Phase two data Pretraining Quality annotator StarCoder2-Instruct Synthesis Synthetic data