Figure 2: Evolution of performance in commonsense reasoning benchmarks during pre-training. The perfor-
mance of Pythia-1.4B is also included in the figure for comparison.

Problem-solving evaluation We also evaluate TinyLlama’s problem-solving capabilities using the
InstructEval benchmark (Chia et al., 2023). This benchmark includes the following tasks:

• Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021): This task is
used to measure a model’s world knowledge and problem-solving capabilities across various
subjects. We evaluate the models in a 5-shot setting.

• BIG-Bench Hard (BBH) (Suzgun et al., 2023): This is a subset of 23 challenging tasks from
the BIG-Bench benchmark (Srivastava et al., 2022) designed to measure a language model’s
abilities in complex instruction following. The models are evaluated in a 3-shot setting.
TinyLlama