There were several discussions within the BigCode community about whether to up-sample or down-
sample certain programming languages, as the amount of compute budget allocated to a data source
in a given language can significantly affect the model’s performance in that language. However,
we realized that the largest amount of available data comes from popular programming languages
and would, therefore, benefit a larger group of end-users. Moreover, after the deduplication process,
we found that several high-resource programming languages, such as C, C++, C#, Java, Javascript,
Python, and PHP, had a similar amount of data ranging from 44–87 GB. This further reinforced our
belief that we did not need to drastically re-weigh the existing data distribution. Thus, in this work,
we followed the natural distribution of data during training and sampled data sources proportionally
to their volume. However, we did make an exception for JSON, YAML, and CSS, as we only want
StarCoder_paper (1)