title: Judging Quality Across Languages: A Multilingual Approach to Pretraining
    Data Filtering with Language Models
summary: High-quality multilingual training data is essential for effectively
  pretraining large language models (LLMs). Yet, the availability of suitable
  open-source multilingual datasets remains limited. Existing state-of-the-art
  datasets mostly rely on heuristic filtering methods, restricting both their
  cross-lingual transferability and scalability. Here, we introduce JQL, a
  systematic approach that efficiently curates diverse and high-quality
  multilingual data at scale while significantly reducing computational demands.
  JQL distills LLMs' annotation capabilities into lightweight annotators based on
  pretrained multilingual embeddings. These models exhibit robust multilingual
  and cross-lingual performance, even for languages and scripts unseen during
  training. Evaluated empirically across 35 languages, the resulting annotation
  pipeline substantially outperforms current heuristic filtering methods like
  Fineweb2. JQL notably enhances downstream model training quality and increases
  data retention rates. Our research provides practical insights and valuable
  resources for multilingual data curation, raising the standards of multilingual
  dataset development.
paper_authors._id: 683815574d9866c160e88670
paper_authors.name: Mehdi Ali
paper_authors._id: 683815574d9866c160e88671
paper_authors.name: Manuel Brack
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T09:40:14.826Z
paper_authors._id: 62fa1d95e8c9c532aa75331c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
paper_authors.fullname: Manuel Brack
paper_authors.type: user
paper_authors.user: mbrack
paper_authors._id: 683815574d9866c160e88672
paper_authors.name: Max Lübbering
paper_authors._id: 683815574d9866c160e88673
paper_authors.name: Elias Wendt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:50:36.473Z
paper_authors._id: 6310ebaf631a69165c076b8d
paper_authors.avatarUrl: /avatars/6ed432656913bbd77162187d158830f0.svg
paper_authors.fullname: Elias Wendt
paper_authors.type: user
paper_authors.user: eliaswendt
paper_authors._id: 683815574d9866c160e88674
paper_authors.name: Abbas Goher Khan
paper_authors._id: 683815574d9866c160e88675
paper_authors.name: Richard Rutmann
paper_authors._id: 683815574d9866c160e88676
paper_authors.name: Alex Jude
paper_authors._id: 683815574d9866c160e88677
paper_authors.name: Maurice Kraus
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:02.697Z
paper_authors._id: 6399acd4074f7c531d57cdc1
paper_authors.avatarUrl: /avatars/83e89dda95e2139f95492eee0da2e471.svg
paper_authors.fullname: Maurice Kraus
paper_authors.type: user
paper_authors.user: mkrausio
paper_authors._id: 683815574d9866c160e88678
paper_authors.name: Alexander Arno Weber
paper_authors._id: 683815574d9866c160e88679
paper_authors.name: Felix Stollenwerk
paper_authors._id: 683815574d9866c160e8867a
paper_authors.name: David Kaczér
paper_authors._id: 683815574d9866c160e8867b
paper_authors.name: Florian Mai
paper_authors._id: 683815574d9866c160e8867c
paper_authors.name: Lucie Flek
paper_authors._id: 683815574d9866c160e8867d
paper_authors.name: Rafet Sifa
paper_authors._id: 683815574d9866c160e8867e
paper_authors.name: Nicolas Flores-Herr
paper_authors._id: 683815574d9866c160e8867f
paper_authors.name: Joachim Köhler
paper_authors._id: 683815574d9866c160e88680
paper_authors.name: Patrick Schramowski
paper_authors._id: 683815574d9866c160e88681
paper_authors.name: Michael Fromm
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T10:00:21.242Z
paper_authors._id: 64243ad773f771f6630871e4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64243ad773f771f6630871e4/Ds4g_5RyBKBDWAC32BqsH.jpeg
paper_authors.fullname: Michael Fromm
paper_authors.type: user
paper_authors.user: mfromm
paper_authors._id: 683815574d9866c160e88682
paper_authors.name: Kristian Kersting
paper_title: Judging Quality Across Languages: A Multilingual Approach to Pretraining
    Data Filtering with Language Models
paper_summary: High-quality multilingual training data is essential for effectively
  pretraining large language models (LLMs). Yet, the availability of suitable
  open-source multilingual datasets remains limited. Existing state-of-the-art
  datasets mostly rely on heuristic filtering methods, restricting both their
  cross-lingual transferability and scalability. Here, we introduce JQL, a
  systematic approach that efficiently curates diverse and high-quality
  multilingual data at scale while significantly reducing computational demands.
  JQL distills LLMs' annotation capabilities into lightweight annotators based on
  pretrained multilingual embeddings. These models exhibit robust multilingual
  and cross-lingual performance, even for languages and scripts unseen during
  training. Evaluated empirically across 35 languages, the resulting annotation
  pipeline substantially outperforms current heuristic filtering methods like
  Fineweb2. JQL notably enhances downstream model training quality and increases
  data retention rates. Our research provides practical insights and valuable
  resources for multilingual data curation, raising the standards of multilingual
  dataset development.
paper_ai_summary: JQL systematically curates high-quality multilingual training data using pretrained multilingual embeddings, outperforming heuristic methods and improving downstream model training across diverse languages.
paper_ai_keywords: pretraining large language models multilingual datasets heuristic filtering methods JQL lightweight annotators multilingual embeddings cross-lingual transferability annotation pipeline data retention rates multilingual data curation