HuggingFaceFW/fineweb-edu
Viewer
โข
Updated
โข
3B
โข
329k
โข
571
HuggingFaceFW
Enterprise
community
AI & ML interests
None defined yet.
Recent Activity
HuggingFaceFW's activity
anton-lย
updated
a
dataset
5 days ago
Post
1965
Introducing ๐๐
๐ข๐ง๐๐๐๐ญ๐ก: the best public math pre-training dataset with 50B+ tokens!
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
๐ ๏ธ carefully extracting math data from Common Crawl;
๐ iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! ๐
Weโre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
๐ ๏ธ carefully extracting math data from Common Crawl;
๐ iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! ๐
Weโre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
guipenedoย
updated
a
Space
7 days ago
guipenedoย
updated
a
dataset
7 days ago
Post
4333
We are proud to announce
HuggingFaceFW/fineweb-2: A sparkling update to
HuggingFaceFW/fineweb with 1000s of ๐ฃ๏ธlanguages.
We applied the same data-driven approach that led to SOTA English performance in๐ท FineWeb to thousands of languages.
๐ฅ FineWeb2 has 8TB of compressed text data and outperforms other multilingual datasets in our experiments.
The dataset is released under the permissive ๐ ODC-By 1.0 license, and the ๐ป code to reproduce it and our evaluations is public.
We will very soon announce a big community project, and are working on a ๐ blogpost walking you through the entire dataset creation process. Stay tuned!
In the mean time come ask us question on our chat place: HuggingFaceFW/discussion
H/t @guipenedo @hynky @lvwerra as well as @vsabolcec Bettina Messmer @negar-foroutan and @mjaggi
We applied the same data-driven approach that led to SOTA English performance in๐ท FineWeb to thousands of languages.
๐ฅ FineWeb2 has 8TB of compressed text data and outperforms other multilingual datasets in our experiments.
The dataset is released under the permissive ๐ ODC-By 1.0 license, and the ๐ป code to reproduce it and our evaluations is public.
We will very soon announce a big community project, and are working on a ๐ blogpost walking you through the entire dataset creation process. Stay tuned!
In the mean time come ask us question on our chat place: HuggingFaceFW/discussion
H/t @guipenedo @hynky @lvwerra as well as @vsabolcec Bettina Messmer @negar-foroutan and @mjaggi
- 2 replies
clefourrierย
authored
a
paper
19 days ago
garrethleeย
posted
an
update
19 days ago
Post
1887
The latest o1 model from OpenAI is still unable to answer 9.11 > 9.9 correctly ๐ค
A possible explanation? Tokenization - and our latest work investigates how it affects a model's ability to do math!
In this blog post, we discuss:
๐ข The different ways numbers are tokenized in modern LLMs
๐งช Our detailed approach in comparing these various methods
๐ฅช How we got a free boost in arithmetic performance by adding a few lines of code to the base Llama 3 tokenizer
๐ and a definitive, best tokenization method for math in LLMs!
Check out our work here: huggingface/number-tokenization-blog
A possible explanation? Tokenization - and our latest work investigates how it affects a model's ability to do math!
In this blog post, we discuss:
๐ข The different ways numbers are tokenized in modern LLMs
๐งช Our detailed approach in comparing these various methods
๐ฅช How we got a free boost in arithmetic performance by adding a few lines of code to the base Llama 3 tokenizer
๐ and a definitive, best tokenization method for math in LLMs!
Check out our work here: huggingface/number-tokenization-blog
-
- 2 replies
Post
886
Exponentially growing number of open-source AI models over the course of the past 30 months โ from a few thousands to over 1 million and more
Interactive data viz: huggingface/open-source-ai-year-in-review-2024
Interactive data viz: huggingface/open-source-ai-year-in-review-2024
Post
1356
Most liked and most downloaded open-source AI models from 2022 to 2024
Interactive viz: https://aiworld.eu/embed/model/model/treemap
Discussion: huggingface/open-source-ai-year-in-review-2024
Interactive viz: https://aiworld.eu/embed/model/model/treemap
Discussion: huggingface/open-source-ai-year-in-review-2024
garrethleeย
posted
an
update
27 days ago
Post
360
Does tokenizing numbers into single digits outperform three-digit or BPE tokenization for arithmetic tasks? We explore various tokenization methods in our upcoming blog (releasing next week ๐)!
๐น Bringing objectivity to comparisons
Existing comparisons of number tokenization methods often ignore the difference in modelsโ compute budgets: larger tokenizer vocabularies naturally lead to more parameters, which produces less objective comparisons of model performances due to more โlearningโ being done by these bigger models.
We addressed this by keeping architectures consistent but adjusting the number of hidden layers to produce roughly equal parameter counts.
๐น Key results
We trained models on the same data mix and evaluated their performance on various arithmetic tasks (digits, operations, floats vs. ints):
- When splitting evals based on operators, single-digit tokenization consistently outperformed other methods.
- Right-to-left tokenization (which I covered in a previous post) matched or exceeded left-to-right approaches in all tasks.
All in all, single-digit tokenization is best compared to other methods, and similar to our previous postโs finding, R2L works better than L2R tokenization, although not as significant as the gap between single-digit and the rest!
The wait is almost over ๐ค, the full report is coming next week - stay tuned!
๐น Bringing objectivity to comparisons
Existing comparisons of number tokenization methods often ignore the difference in modelsโ compute budgets: larger tokenizer vocabularies naturally lead to more parameters, which produces less objective comparisons of model performances due to more โlearningโ being done by these bigger models.
We addressed this by keeping architectures consistent but adjusting the number of hidden layers to produce roughly equal parameter counts.
๐น Key results
We trained models on the same data mix and evaluated their performance on various arithmetic tasks (digits, operations, floats vs. ints):
- When splitting evals based on operators, single-digit tokenization consistently outperformed other methods.
- Right-to-left tokenization (which I covered in a previous post) matched or exceeded left-to-right approaches in all tasks.
All in all, single-digit tokenization is best compared to other methods, and similar to our previous postโs finding, R2L works better than L2R tokenization, although not as significant as the gap between single-digit and the rest!
The wait is almost over ๐ค, the full report is coming next week - stay tuned!
Post
1637
Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit ๐ ๏ธ https://github.com/huggingface/smollm/
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
Post
1639
Interesting long read from
@evanmiller-anthropic
on having a better founded statistical approach to Language Model Evaluations:
https://www.anthropic.com/research/statistical-approach-to-model-evals
Worth a read if you're into LLM evaluations!
Cc @clefourrier
https://www.anthropic.com/research/statistical-approach-to-model-evals
Worth a read if you're into LLM evaluations!
Cc @clefourrier
- 1 reply
SaylorTwiftย
posted
an
update
about 1 month ago
Post
386
How do I test an LLM for my unique needs?
If you work in finance, law, or medicine, generic benchmarks are not enough.
This blog post uses Argilla, Distilllabel and ๐ค๏ธLighteval to generate evaluation dataset and evaluate models.
https://github.com/argilla-io/argilla-cookbook/blob/main/domain-eval/README.md
If you work in finance, law, or medicine, generic benchmarks are not enough.
This blog post uses Argilla, Distilllabel and ๐ค๏ธLighteval to generate evaluation dataset and evaluate models.
https://github.com/argilla-io/argilla-cookbook/blob/main/domain-eval/README.md
Post
1413
Very exciting new
mistralai/Pixtral-Large-Instruct-2411 model from Mistral-AI
Impressive performances, huge congrats @patrickvonplaten @sgvaze @pandora-s @devendrachaplot @sophiamyang and team!
Very nice to have SOTA Multilingual OCR and Chart understanding in an open-weights model
Impressive performances, huge congrats @patrickvonplaten @sgvaze @pandora-s @devendrachaplot @sophiamyang and team!
Very nice to have SOTA Multilingual OCR and Chart understanding in an open-weights model
lvwerraย
authored
a
paper
about 2 months ago
Post
4114
Parents in the 1990: Teach the kids to code
Parents now: Teach the kids to fix the code when it starts walking around ๐คโจ
Parents now: Teach the kids to fix the code when it starts walking around ๐คโจ
- 2 replies
clefourrierย
authored
2
papers
6 months ago
thomwolfย
authored
a
paper
6 months ago
lvwerraย
authored
a
paper
6 months ago