(a) Standard

(b) Deduplicated

Figure 9. LAMBADA (OpenAI) over the course of training. Left is the standard Pile, while the right is the deduplicated Pile. The dashed
line indicates where the deduplicated Pile began its second epoch.

(a) Standard

(b) Deduplicated

Figure 10. Winogrand Schema Challenge over the course of training. Left is the standard Pile, while the right is the deduplicated Pile.
The dashed line indicates where the deduplicated Pile began its second epoch.
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling