Performance Analysis and Comparison with Other Models The incorporation of these elements
has propelled our training throughput to 24,000 tokens per second per A100-40G GPU. When
compared with other models like Pythia-1.0B (Biderman et al., 2023) and MPT-1.3B 2, our codebase
demonstrates superior training speed. For instance, the TinyLlama-1.1B model requires only 3,456
A100 GPU hours for 300B tokens, in contrast to Pythia’s 4,830 and MPT’s 7,920 hours. This shows
the effectiveness of our optimizations and the potential for substantial time and resource savings in
large-scale model training.

2.4 Training

We build our framework based on lit-gpt.3 In adhering to Llama 2 (Touvron et al., 2023b), we employ
an autoregressive language modeling objective during the pretraining phase. Consistent with Llama
2’s settings, we utilize the AdamW optimizer (Loshchilov and Hutter, 2019), setting β1 at 0.9 and
TinyLlama