anxiety among academic researchers about whether they can meaningfully contribute to new AI
breakthroughs (Togelius & Yannakakis, 2023).
Other research projects have shared their developed LLMs publicly. In this work, we refer to open-
access LLMs when the model weights are publicly available. However, we stress that the level of
openness still varies across these open-access models, specifically regarding the extent to which they
have made their training data and filtering methods available. For example, Google released the model
weights of T5 (Raffel et al., 2020) and UL2 (Tay et al., 2022) under a permissive license and released
the corresponding C4 training set (Raffel et al., 2020). On the other hand, Salesforce released the
model weights of CodeGen-Mono (Nijkamp et al., 2023) under a permissive license, but they did not
make the proprietary Python training set available. OPT (Zhang et al., 2022) and LLaMA (Touvron
StarCoder_paper (1)