Proof that ablative educational dataset significantly enhances model capabilities (independent of model parameters or architecture) 🤩
text
Proof that ablative educational dataset significantly enhances model capabilities (independent of model parameters or architecture) 🤩
new_line
new_line
Yesterday, FineWeb’s technical report was published. FYI FineWeb (by 🤗) is currently the best opensource text dataset that can scale up model performance up to that of GPT-3 level. 
text
Yesterday, FineWeb’s technical report was published. FYI FineWeb (by 🤗) is currently the best opensource text dataset that can scale up model performance up to that of GPT-3 level. 
new_line
new_line
While proprietary datasets used in training models like GPT-4/Claude/LlaMA are crawled internally and never released, FineWeb builds on CommonCrawl (an open repo for crawled web data). They preprocessed the data using their custom built data preprocessing library datatrove (which they also opensourced), and then evaluate the data quality on lighteval by training small sized models “ablation models” using nanotron (a library for pretraining transformer models).
text
While proprietary datasets used in training models like GPT-4/Claude/LlaMA are crawled internally and never released, FineWeb builds on CommonCrawl (an open repo for crawled web data). They preprocessed the data using their custom built data preprocessing library datatrove (which they also opensourced), and then evaluate the data quality on lighteval by training small sized models “ablation models” using nanotron (a library for pretraining transformer models).
new_line
new_line
Of all versions of FineWeb, FineWeb-Edu outperforms all other subsets. This is thanks to a new filtering technique wherein they used synthetic data to develop classifiers for identifying educational contents.
text
Of all versions of FineWeb, FineWeb-Edu outperforms all other subsets. This is thanks to a new filtering technique wherein they used synthetic data to develop classifiers for identifying educational contents.
new_line
new_line
Turned out “Education is All You Need”:)
text
Turned out “Education is All You Need”:)
Proof that ablative educational dataset significantly enhances model capabilities (independent of model parameters or architecture) 🤩

Yesterday, FineWeb’s technical report was published. FYI FineWeb (by 🤗) is currently the best opensource text dataset that can scale up model performance up to that of GPT-3 level. 

While proprietary datasets used in training models like GPT-4/Claude/LlaMA are crawled internally and never released, FineWeb builds on CommonCrawl (an open repo for crawled web data). They preprocessed the data using their custom built data preprocessing library datatrove (which they also opensourced), and then evaluate the data quality on lighteval by training small sized models “ablation models” using nanotron (a library for pretraining transformer models).

Of all versions of FineWeb, FineWeb-Edu outperforms all other subsets. This is thanks to a new filtering technique wherein they used synthetic data to develop classifiers for identifying educational contents.

Turned out “Education is All You Need”:)
🚀
Taylor658 osanseviero valentimarco Vlansu Hev832 neuralink hynky GPT007 victor Ramikan-BR jlzhou
🧠
neuralink Ramikan-BR
👍
dashfunnydashdash Ramikan-BR
🔥
Ramikan-BR
👀
Ramikan-BR
❤️
Ramikan-BR
6438a9027de34e8ea7e4b257
https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
Jaward Sesay
Jaward
user