(2023). Modern models are trained on publicly available, open-source code. In addition, Allamanis (2019)
and Allal et al. (2023) discuss the impact of effective deduplication and of selecting code from repositories
based on the number of GitHub stars (as a proxy for popularity), while Li et al. (2023) augment their data
with GitHub issues and commits collected from BigQuery. Gunasekar et al. (2023) filter data up to only
containing “textbook”-quality code and add synthetic problems collected using GPT-3.5, following Jung
et al. (2023), in order to obtain good performance on simple benchmarks such as HumanEval and MBPP. We
follow the approach of learning from publicly available code only, without additional meta-level or temporal
information such as issues or commits. We also do not train our foundation models on additional synthetic
exercises, since we did not want to take the risk of reducing the scope of our models to simple coding exercises
CodeLlama2