4.3 Recipe of Best Open-source LLMs

Training an LLM involves complex and resource-intensive practices, including data collection and
preprocessing, model design, and training process. While there is a growing trend of releasing open-
source LLMs regularly, the detailed practices of the leading models are often kept secret unfortunately.
Below we list some best practices widely acknowledged by the community.

Data Pre-training involves the use of trillions of data tokens, often sourced from publicly accessible
sources. Ethically, it is crucial to exclude any data that includes personal information of private
individuals (Touvron et al., 2023b). Unlike pre-training data, fine-tuning data is smaller in quantity

13

but superior in quality. Fine-tuned LLMs with top-quality data have shown improved performance,
particularly in specialized areas (Philippines, 2023; Zeng et al., 2023; Xu et al., 2023d,a).
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup