Essential-Web v1.0: 24T tokens of organized web data
Data plays the most prominent role in how language models acquire skills and
knowledge. The lack of massive, well-organized pre-training datasets results in
costly and inaccessible data pipelines. We present Essential-Web v1.0, a
24-trillion-token dataset in which every document is annotated with a
twelve-category taxonomy covering topic, format, content complexity, and
quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned
0.5b-parameter model that achieves an annotator agreement within 3% of
Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain
competitive web-curated datasets in math (-8.0% relative to SOTA), web code
(+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on
HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0
6852072a0164cd1316710407
Essential AI
6852072a0164cd1316710409
Andrew Hojel
6852072a0164cd131671040a
Michael Pust
claimed_verified
2025-06-27T08:57:58.657Z
66cd078ea796074d428fde0f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qThw3H6ukqIAuRy7aJTN1.jpeg
Michael Pust
user
essentialpust
6852072a0164cd131671040b
Tim Romanski
claimed_verified
2025-06-18T12:16:31.481Z
67101a7165442ddc48cb4b07
/avatars/551777fcd1638998ad9fd16804b313ec.svg
Tim Romanski
user
tim-essential
6852072a0164cd131671040c
Yash Vanjani
6852072a0164cd131671040d
Ritvik Kapila
claimed_verified
2025-06-18T19:48:00.136Z
654bdcf2e06d25def57cc54b
/avatars/2d2612bd7072edd60876b504345fbf25.svg
Ritvik Kapila
user
rkapila
6852072a0164cd131671040e
Mohit Parmar
claimed_verified
2025-06-18T12:16:34.350Z
671aeee626d87414f59ee7d6
/avatars/131d0ff4c12bc70e74945910ce0e1d56.svg
Mohit Parmar
user
mohit-essential
6852072a0164cd131671040f
Adarsh Chaluvaraju
6852072a0164cd1316710410
Alok Tripathy
6852072a0164cd1316710411
Anil Thomas
6852072a0164cd1316710412
Ashish Tanwer
claimed_verified
2025-06-18T16:15:43.379Z
66f5bf4db839f338c5afdc9a
/avatars/0cf00615da06cb7c04c42d55e9a97e84.svg
Ashish Tanwer
user
TanwerAshish
6852072a0164cd1316710413
Darsh J Shah
6852072a0164cd1316710414
Ishaan Shah
6852072a0164cd1316710415
Karl Stratos
6852072a0164cd1316710416
Khoi Nguyen
claimed_verified
2025-06-18T12:16:36.341Z
67ed7aa9290a7f9d33113fb5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jMMTObGZktuXWJ3wVVSxj.png
Khoi Nguyen
user
KTLK
6852072a0164cd1316710417
Kurt Smith
6852072a0164cd1316710418
Michael Callahan
claimed_verified
2025-06-23T08:29:20.950Z
66f62e1aa854f31fea6c7d25
/avatars/3cf6f471e26cec430fa5623bac938fa7.svg
Michael Callahan
user
mikecallahan
6852072a0164cd1316710419
Peter Rushton
6852072a0164cd131671041a
Philip Monk
6852072a0164cd131671041b
Platon Mazarakis
6852072a0164cd131671041c
Saad Jamal
6852072a0164cd131671041d
Saurabh Srivastava
6852072a0164cd131671041e
Somanshu Singla
claimed_verified
2025-06-18T12:16:38.442Z
679bc0b23e12a166672e5275
/avatars/fe9d0e79c21c9d3594420e69e3809f0f.svg
Somanshu Singla
user
SomanshuSingla
6852072a0164cd131671041f
Ashish Vaswani
Essential-Web v1.0: 24T tokens of organized web data
Data plays the most prominent role in how language models acquire skills and
knowledge. The lack of massive, well-organized pre-training datasets results in
costly and inaccessible data pipelines. We present Essential-Web v1.0, a
24-trillion-token dataset in which every document is annotated with a
twelve-category taxonomy covering topic, format, content complexity, and
quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned
0.5b-parameter model that achieves an annotator agreement within 3% of
Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain
competitive web-curated datasets in math (-8.0% relative to SOTA), web code
(+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on
HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0
A large, 24-trillion-token Essential-Web v1.0 dataset annotated with a multi-category taxonomy outperforms or is competitive with existing datasets in various domains using simple filtering techniques.
Essential-Web EAI-Distill-0.5b Qwen2.5-32B-Instruct