Judging Quality Across Languages: A Multilingual Approach to Pretraining
  Data Filtering with Language Models
High-quality multilingual training data is essential for effectively
pretraining large language models (LLMs). Yet, the availability of suitable
open-source multilingual datasets remains limited. Existing state-of-the-art
datasets mostly rely on heuristic filtering methods, restricting both their
cross-lingual transferability and scalability. Here, we introduce JQL, a
systematic approach that efficiently curates diverse and high-quality
multilingual data at scale while significantly reducing computational demands.
JQL distills LLMs' annotation capabilities into lightweight annotators based on
pretrained multilingual embeddings. These models exhibit robust multilingual
and cross-lingual performance, even for languages and scripts unseen during
training. Evaluated empirically across 35 languages, the resulting annotation
pipeline substantially outperforms current heuristic filtering methods like
Fineweb2. JQL notably enhances downstream model training quality and increases
data retention rates. Our research provides practical insights and valuable
resources for multilingual data curation, raising the standards of multilingual
dataset development.
683815574d9866c160e88670
Mehdi Ali
683815574d9866c160e88671
Manuel Brack
claimed_verified
2025-05-29T09:40:14.826Z
62fa1d95e8c9c532aa75331c
https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
Manuel Brack
user
mbrack
683815574d9866c160e88672
Max Lübbering
683815574d9866c160e88673
Elias Wendt
claimed_verified
2025-06-03T08:50:36.473Z
6310ebaf631a69165c076b8d
/avatars/6ed432656913bbd77162187d158830f0.svg
Elias Wendt
user
eliaswendt
683815574d9866c160e88674
Abbas Goher Khan
683815574d9866c160e88675
Richard Rutmann
683815574d9866c160e88676
Alex Jude
683815574d9866c160e88677
Maurice Kraus
claimed_verified
2025-06-02T07:48:02.697Z
6399acd4074f7c531d57cdc1
/avatars/83e89dda95e2139f95492eee0da2e471.svg
Maurice Kraus
user
mkrausio
683815574d9866c160e88678
Alexander Arno Weber
683815574d9866c160e88679
Felix Stollenwerk
683815574d9866c160e8867a
David Kaczér
683815574d9866c160e8867b
Florian Mai
683815574d9866c160e8867c
Lucie Flek
683815574d9866c160e8867d
Rafet Sifa
683815574d9866c160e8867e
Nicolas Flores-Herr
683815574d9866c160e8867f
Joachim Köhler
683815574d9866c160e88680
Patrick Schramowski
683815574d9866c160e88681
Michael Fromm
claimed_verified
2025-06-05T10:00:21.242Z
64243ad773f771f6630871e4
https://cdn-avatars.huggingface.co/v1/production/uploads/64243ad773f771f6630871e4/Ds4g_5RyBKBDWAC32BqsH.jpeg
Michael Fromm
user
mfromm
683815574d9866c160e88682
Kristian Kersting
Judging Quality Across Languages: A Multilingual Approach to Pretraining
  Data Filtering with Language Models
High-quality multilingual training data is essential for effectively
pretraining large language models (LLMs). Yet, the availability of suitable
open-source multilingual datasets remains limited. Existing state-of-the-art
datasets mostly rely on heuristic filtering methods, restricting both their
cross-lingual transferability and scalability. Here, we introduce JQL, a
systematic approach that efficiently curates diverse and high-quality
multilingual data at scale while significantly reducing computational demands.
JQL distills LLMs' annotation capabilities into lightweight annotators based on
pretrained multilingual embeddings. These models exhibit robust multilingual
and cross-lingual performance, even for languages and scripts unseen during
training. Evaluated empirically across 35 languages, the resulting annotation
pipeline substantially outperforms current heuristic filtering methods like
Fineweb2. JQL notably enhances downstream model training quality and increases
data retention rates. Our research provides practical insights and valuable
resources for multilingual data curation, raising the standards of multilingual
dataset development.
JQL systematically curates high-quality multilingual training data using pretrained multilingual embeddings, outperforming heuristic methods and improving downstream model training across diverse languages.
pretraining large language models multilingual datasets heuristic filtering methods JQL lightweight annotators multilingual embeddings cross-lingual transferability annotation pipeline data retention rates multilingual data curation