0
2
0
2
 
c
e
D
1
3

 

 
 
]
L
C
.
s
c
[
 
 

1
v
7
2
0
0
0

.

1
0
1
2
:
v
i
X
r
a

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Leo Gao

Stella Biderman

Sid Black

Laurence Golding

Travis Hoppe

Charles Foster

Jason Phang

Horace He

Anish Thite

Noa Nabeshima

Shawn Presser

Connor Leahy

EleutherAI

contact@eleuther.ai

Abstract
The Pile- An 800GB Dataset of Diverse Text for Language Modeling