title: To Code, or Not To Code? Exploring Impact of Code in Pre-training
summary: Including code in the pre-training data mixture, even for models not
  specifically designed for code, has become a common practice in LLMs
  pre-training. While there has been anecdotal consensus among practitioners that
  code data plays a vital role in general LLMs' performance, there is only
  limited work analyzing the precise impact of code on non-code tasks. In this
  work, we systematically investigate the impact of code data on general
  performance. We ask "what is the impact of code data used in pre-training on a
  large variety of downstream tasks beyond code generation". We conduct extensive
  ablations and evaluate across a broad range of natural language reasoning
  tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for
  models with sizes ranging from 470M to 2.8B parameters. Across settings, we
  find a consistent results that code is a critical building block for
  generalization far beyond coding tasks and improvements to code quality have an
  outsized impact across all tasks. In particular, compared to text-only
  pre-training, the addition of code results in up to relative increase of 8.2%
  in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement
  in generative win-rates, and a 12x boost in code performance respectively. Our
  work suggests investments in code quality and preserving code during
  pre-training have positive impacts.
paper_authors._id: 66c556dd50f505c5451ce8e1
paper_authors.name: Viraat Aryabumi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:51:19.578Z
paper_authors._id: 633e9b78acd351fc37d69560
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633e9b78acd351fc37d69560/rCk8z-C4oOlrksp-QE2lV.jpeg
paper_authors.fullname: Viraat Aryabumi
paper_authors.type: user
paper_authors.user: viraat
paper_authors._id: 66c556dd50f505c5451ce8e2
paper_authors.name: Yixuan Su
paper_authors._id: 66c556dd50f505c5451ce8e3
paper_authors.name: Raymond Ma
paper_authors._id: 66c556dd50f505c5451ce8e4
paper_authors.name: Adrien Morisot
paper_authors._id: 66c556dd50f505c5451ce8e5
paper_authors.name: Ivan Zhang
paper_authors._id: 66c556dd50f505c5451ce8e6
paper_authors.name: Acyr Locatelli
paper_authors._id: 66c556dd50f505c5451ce8e7
paper_authors.name: Marzieh Fadaee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T14:10:34.795Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_authors._id: 66c556dd50f505c5451ce8e8
paper_authors.name: Ahmet Üstün
paper_authors._id: 66c556dd50f505c5451ce8e9
paper_authors.name: Sara Hooker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:48:46.677Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_title: To Code, or Not To Code? Exploring Impact of Code in Pre-training
paper_summary: Including code in the pre-training data mixture, even for models not
  specifically designed for code, has become a common practice in LLMs
  pre-training. While there has been anecdotal consensus among practitioners that
  code data plays a vital role in general LLMs' performance, there is only
  limited work analyzing the precise impact of code on non-code tasks. In this
  work, we systematically investigate the impact of code data on general
  performance. We ask "what is the impact of code data used in pre-training on a
  large variety of downstream tasks beyond code generation". We conduct extensive
  ablations and evaluate across a broad range of natural language reasoning
  tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for
  models with sizes ranging from 470M to 2.8B parameters. Across settings, we
  find a consistent results that code is a critical building block for
  generalization far beyond coding tasks and improvements to code quality have an
  outsized impact across all tasks. In particular, compared to text-only
  pre-training, the addition of code results in up to relative increase of 8.2%
  in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement
  in generative win-rates, and a 12x boost in code performance respectively. Our
  work suggests investments in code quality and preserving code during
  pre-training have positive impacts.
paper_ai_keywords: LLMs pre-training downstream tasks natural language reasoning world knowledge tasks code benchmarks LLM-as-a-judge parameters generalization generative win-rates code quality