Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining
  and Extracting Rare and Hidden Content
We introduce Biomed-Enriched, a biomedical text dataset constructed from
PubMed via a two-stage annotation process. In the first stage, a large language
model annotates 400K paragraphs from PubMed scientific articles, assigning
scores for their type (review, study, clinical case, other), domain (clinical,
biomedical, other), and educational quality. The educational quality score
(rated 1 to 5) estimates how useful a paragraph is for college-level learning.
These annotations are then used to fine-tune a small language model, which
propagates the labels across the full PMC-OA corpus. The resulting metadata
allows us to extract refined subsets, including 2M clinical case paragraphs
with over 450K high-quality ones from articles with commercial-use licenses,
and to construct several variants via quality filtering and domain upsampling.
Clinical text is typically difficult to access due to privacy constraints, as
hospital records cannot be publicly shared. Hence, our dataset provides an
alternative large-scale, openly available collection of clinical cases from
PubMed, making it a valuable resource for biomedical and clinical NLP.
Preliminary continual-pretraining experiments with OLMo2 suggest these curated
subsets enable targeted improvements, with clinical upsampling boosting
performance by ~5% on MMLU ProfMed and educational quality filtering improving
MedQA and MedMCQA by ~1%. Combinations of these techniques led to faster
convergence, reaching same performance with a third of training tokens,
indicating potential for more efficient and effective biomedical pretraining
strategies.
685d0b5d696820ba1f28f349
Rian Touchent
claimed_verified
2025-06-26T09:23:08.325Z
62a9b0acf6708cb85014f9dc
https://cdn-avatars.huggingface.co/v1/production/uploads/62a9b0acf6708cb85014f9dc/Sem1qcBt1lJjFEPK-xz4_.jpeg
Rian Touchent
user
rntc
685d0b5d696820ba1f28f34a
Nathan Godey
685d0b5d696820ba1f28f34b
Eric de la Clergerie
Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining
  and Extracting Rare and Hidden Content
We introduce Biomed-Enriched, a biomedical text dataset constructed from
PubMed via a two-stage annotation process. In the first stage, a large language
model annotates 400K paragraphs from PubMed scientific articles, assigning
scores for their type (review, study, clinical case, other), domain (clinical,
biomedical, other), and educational quality. The educational quality score
(rated 1 to 5) estimates how useful a paragraph is for college-level learning.
These annotations are then used to fine-tune a small language model, which
propagates the labels across the full PMC-OA corpus. The resulting metadata
allows us to extract refined subsets, including 2M clinical case paragraphs
with over 450K high-quality ones from articles with commercial-use licenses,
and to construct several variants via quality filtering and domain upsampling.
Clinical text is typically difficult to access due to privacy constraints, as
hospital records cannot be publicly shared. Hence, our dataset provides an
alternative large-scale, openly available collection of clinical cases from
PubMed, making it a valuable resource for biomedical and clinical NLP.
Preliminary continual-pretraining experiments with OLMo2 suggest these curated
subsets enable targeted improvements, with clinical upsampling boosting
performance by ~5% on MMLU ProfMed and educational quality filtering improving
MedQA and MedMCQA by ~1%. Combinations of these techniques led to faster
convergence, reaching same performance with a third of training tokens,
indicating potential for more efficient and effective biomedical pretraining
strategies.
A biomedical text dataset, constructed from PubMed, uses a two-stage annotation process involving large and small language models to fine-tune and extract subsets for clinical NLP, improving pretraining efficiency and performance.
Biomed-Enriched PubMed large language model small language model fine-tuning PMC-OA corpus educational quality clinical cases biomedical NLP continual-pretraining OLMo2 MMLU ProfMed MedQA MedMCQA training tokens