🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
text
🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
new_line
new_line
Technical report: 
text
Technical report: 
https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
HuggingFaceFW/blogpost-fineweb-v1
space
resource
https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
text
new_line
Dataset: 
text
Dataset: 
https://hf.co/datasets/HuggingFaceFW/fineweb-edu
HuggingFaceFW/fineweb-edu
dataset
resource
https://hf.co/datasets/HuggingFaceFW/fineweb-edu
text
new_line
new_line
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
text
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
new_line
new_line
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
text
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
new_line
new_line
Enjoy!
text
Enjoy!
🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.

Technical report: https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1  
Dataset: https://hf.co/datasets/HuggingFaceFW/fineweb-edu 

We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 

You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.

Enjoy!
🔥
mmhamdy alielfilali01 GPT007 arjunguha hiauiarau nicolay-r privategeek24 Ariel323 maywell guipenedo neuralink asaduzzaman319
👍
dillfrescott guipenedo neuralink
🧠
neuralink louisbrulenaudet