Recent progress in natural language processing (NLP) has been largely propelled by scaling up
language model sizes (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a,b). Large
Language Models (LLMs) pre-trained on extensive text corpora have demonstrated their effectiveness
on a wide range of tasks (OpenAI, 2023; Touvron et al., 2023b). Some empirical studies demonstrated
emergent abilities in LLMs, abilities that may only manifest in models with a sufficiently large number
of parameters, such as few-shot prompting (Brown et al., 2020) and chain-of-thought reasoning (Wei
et al., 2022). Other studies focus on modeling the scaling behavior of LLMs (Kaplan et al., 2020;
Hoffmann et al., 2022). Hoffmann et al. (2022) suggest that, to train a compute-optimal model, the
size of the model and the amount of training data should be increased at the same rate. This provides
a guideline on how to optimally select the model size and allocate the amount of training data when
TinyLlama