1https://huggingface.co/docs/accelerate/usage_guides/fsdp
2https://huggingface.co/mosaicml/mpt-1b-redpajama-200b
3https://github.com/Lightning-AI/lit-gpt

3

Figure 1: Comparison of the training speed of our codebase with Pythia and MPT.

β2 at 0.95. Additionally, we use a cosine learning rate schedule with maximum learning rate as
4.0 × 10−4 and minimum learning rate as 4.0 × 10−5. We use 2,000 warmup steps to facilitate
optimized learning.4 We set the batch size as 2M tokens. We assign weight decay as 0.1, and use
a gradient clipping threshold of 1.0 to regulate the gradient value. We pretrain TinyLlama with 16
A100-40G GPUs in our project.

3 Results

We evaluate TinyLlama on a wide range of commonsense reasoning and problem-solving tasks and
compare it with several existing open-source language models with similar model parameters.
TinyLlama