The 
text
The 
https://huggingface.co/datasets/data-is-better-together/fineweb-c
data-is-better-together/fineweb-c
dataset
resource
https://huggingface.co/datasets/data-is-better-together/fineweb-c
 dataset is growing! 
text
 dataset is growing! 
new_line
new_line
This week a few more languages have got 1,000 annotations for the educational quality of data from 
text
This week a few more languages have got 1,000 annotations for the educational quality of data from 
https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
HuggingFaceFW/fineweb-2
dataset
resource
https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
. 
text
. 
new_line
new_line
Why should you care? 
text
Why should you care? 
new_line
new_line
The quality of pre-training data can have a big impact on the performance of downstream language models trained on that data (
text
The quality of pre-training data can have a big impact on the performance of downstream language models trained on that data (
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
HuggingFaceFW/blogpost-fineweb-v1
space
resource
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
). 
text
). 
new_line
new_line
Being able to filter by educational quality is on way of improving the quality of the data you use for training an LLM. Very importantly this approach can also reduce the amount of data needed for pertaining. 
text
Being able to filter by educational quality is on way of improving the quality of the data you use for training an LLM. Very importantly this approach can also reduce the amount of data needed for pertaining. 
new_line
new_line
Why not use an LLM? 
text
Why not use an LLM? 
new_line
new_line
LLMs can be used to annotate educational quality for a subset of data. This data can then be used to train a smaller encoder only model to label the full dataset. However, this may not work well for languages outside of english. This is where fineweb-c (community) comes in. 
text
LLMs can be used to annotate educational quality for a subset of data. This data can then be used to train a smaller encoder only model to label the full dataset. However, this may not work well for languages outside of english. This is where fineweb-c (community) comes in. 
new_line
new_line
The community is annotating the educational quality of fineweb2 data. Currently 114 languages have some annotations. These annotations will enable a number of things:
text
The community is annotating the educational quality of fineweb2 data. Currently 114 languages have some annotations. These annotations will enable a number of things:
new_line
new_line
- Evaluate whether an LLM can label the educational quality for texts in that language well
text
- Evaluate whether an LLM can label the educational quality for texts in that language well
new_line
- Directly be used for training quality classifiers  
text
- Directly be used for training quality classifiers  
new_line
- Help discover other rules and huerisitcs for refining fineweb2 further for different languages. 
text
- Help discover other rules and huerisitcs for refining fineweb2 further for different languages. 
new_line
new_line
This week the following languages where done:
text
This week the following languages where done:
new_line
new_line
Swedish thanks to: 
text
Swedish thanks to: 
@Lauler
mention
Lauler
text
@AntonVic
mention
AntonVic
text
@ohallstrom
mention
ohallstrom
text
@bjarlestam
mention
bjarlestam
text
@menbom
mention
menbom
text
@Ekgren
mention
Ekgren
text
@apsod
mention
apsod
text
new_line
new_line
Ukrainian thanks to: 
text
Ukrainian thanks to: 
@hannayukhymenko
mention
hannayukhymenko
text
@robinhad
mention
robinhad
text
@realPivo
mention
realPivo
text
@RabotiahovDmytro
mention
RabotiahovDmytro
text
@reciprocate
mention
reciprocate
text
new_line
new_line
Assamese thanks to: 
text
Assamese thanks to: 
@moyoor97
mention
moyoor97
text
@Arpanjyoti
mention
Arpanjyoti
text
@nawaf-helmi123
mention
nawaf-helmi123
text
@pahigogoi1
mention
pahigogoi1
text
@aelhence
mention
aelhence
text
@kishorekashyap
mention
kishorekashyap
text
new_line
new_line
Want to learn more: 
text
Want to learn more: 
https://huggingface.co/blog/davanstrien/fineweb2-community
https://huggingface.co/blog/davanstrien/fineweb2-community
link
text
new_line
new_line
Contribute yourself here: 
text
Contribute yourself here: 
https://huggingface.co/spaces/data-is-better-together/fineweb-c
data-is-better-together/fineweb-c
space
resource
https://huggingface.co/spaces/data-is-better-together/fineweb-c
The https://huggingface.co/datasets/data-is-better-together/fineweb-c dataset is growing! 

This week a few more languages have got 1,000 annotations for the educational quality of data from https://huggingface.co/datasets/HuggingFaceFW/fineweb-2. 

Why should you care? 

The quality of pre-training data can have a big impact on the performance of downstream language models trained on that data (https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1). 

Being able to filter by educational quality is on way of improving the quality of the data you use for training an LLM. Very importantly this approach can also reduce the amount of data needed for pertaining. 

Why not use an LLM? 

LLMs can be used to annotate educational quality for a subset of data. This data can then be used to train a smaller encoder only model to label the full dataset. However, this may not work well for languages outside of english. This is where fineweb-c (community) comes in. 

The community is annotating the educational quality of fineweb2 data. Currently 114 languages have some annotations. These annotations will enable a number of things:

- Evaluate whether an LLM can label the educational quality for texts in that language well
- Directly be used for training quality classifiers  
- Help discover other rules and huerisitcs for refining fineweb2 further for different languages. 

This week the following languages where done:

Swedish thanks to: @Lauler @AntonVic @ohallstrom @bjarlestam @menbom @Ekgren @apsod 

Ukrainian thanks to: @hannayukhymenko @robinhad @realPivo @RabotiahovDmytro @reciprocate 

Assamese thanks to: @moyoor97 @Arpanjyoti @nawaf-helmi123 @pahigogoi1 @aelhence @kishorekashyap 

Want to learn more: https://huggingface.co/blog/davanstrien/fineweb2-community 

Contribute yourself here: https://huggingface.co/spaces/data-is-better-together/fineweb-c
🚀
hannayukhymenko dvilasuero stefan-it guipenedo davidberenstein1957 frascuchon nataliaElv John6666 reonyy anasabakar AtAndDev
❤️
davidberenstein1957 reonyy kishorekashyap AtAndDev
🤝
davidberenstein1957 AtAndDev
🔥
anasabakar AtAndDev
634ff41ff32062e9eb7b06a3
https://cdn-avatars.huggingface.co/v1/production/uploads/1677141720071-634ff41ff32062e9eb7b06a3.jpeg
David Berenstein
davidberenstein1957
user