content.raw: 🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
content.type: text
content.value: 🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
content.type: new_line
content.type: new_line
content.raw: Technical report: 
content.type: text
content.value: Technical report: 
content.raw: https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
content.id: HuggingFaceFW/blogpost-fineweb-v1
content.type: space
content.type: resource
content.url: https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
content.type: text
content.type: new_line
content.raw: Dataset: 
content.type: text
content.value: Dataset: 
content.raw: https://hf.co/datasets/HuggingFaceFW/fineweb-edu
content.id: HuggingFaceFW/fineweb-edu
content.type: dataset
content.type: resource
content.url: https://hf.co/datasets/HuggingFaceFW/fineweb-edu
content.type: text
content.type: new_line
content.type: new_line
content.raw: We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
content.type: text
content.value: We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
content.type: new_line
content.type: new_line
content.raw: You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
content.type: text
content.value: You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
content.type: new_line
content.type: new_line
content.raw: Enjoy!
content.type: text
content.value: Enjoy!
rawContent: 🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
  
  Technical report: https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1  
  Dataset: https://hf.co/datasets/HuggingFaceFW/fineweb-edu 
  
  We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
  
  You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
  
  Enjoy!
reactions.reaction: 🔥
reactions.users: mmhamdy alielfilali01 GPT007 arjunguha hiauiarau nicolay-r privategeek24 Ariel323 maywell guipenedo neuralink asaduzzaman319
reactions.reaction: 👍
reactions.users: dillfrescott guipenedo neuralink
reactions.reaction: 🧠
reactions.users: neuralink louisbrulenaudet