METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
We pretrain METAGENE-1, a 7-billion-parameter autoregressive transformer
model, which we refer to as a metagenomic foundation model, on a novel corpus
of diverse metagenomic DNA and RNA sequences comprising over 1.5 trillion base
pairs. This dataset is sourced from a large collection of human wastewater
samples, processed and sequenced using deep metagenomic (next-generation)
sequencing methods. Unlike genomic models that focus on individual genomes or
curated sets of specific species, the aim of METAGENE-1 is to capture the full
distribution of genomic information present within this wastewater, to aid in
tasks relevant to pandemic monitoring and pathogen detection. We carry out
byte-pair encoding (BPE) tokenization on our dataset, tailored for metagenomic
sequences, and then pretrain our model. In this paper, we first detail the
pretraining dataset, tokenization strategy, and model architecture,
highlighting the considerations and design choices that enable the effective
modeling of metagenomic data. We then show results of pretraining this model on
our metagenomic dataset, providing details about our losses, system metrics,
and training stability over the course of pretraining. Finally, we demonstrate
the performance of METAGENE-1, which achieves state-of-the-art results on a set
of genomic benchmarks and new evaluations focused on human-pathogen detection
and genomic sequence embedding, showcasing its potential for public health
applications in pandemic monitoring, biosurveillance, and early detection of
emerging health threats.
677ca28f1fbb93b90f3d2482
Ollie Liu
admin_assigned
2025-01-07T09:48:10.277Z
66197a8afeb55cbe39e50ae8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png
Ollie Liu
user
oliu-io
677ca28f1fbb93b90f3d2483
Sami Jaghouar
claimed_verified
2025-05-26T08:43:35.430Z
618287950fdbff9130ba7ae0
https://cdn-avatars.huggingface.co/v1/production/uploads/618287950fdbff9130ba7ae0/voNf-w5GSG0uSIS8xWgoK.png
samsja
user
samsja
677ca28f1fbb93b90f3d2484
Johannes Hagemann
admin_assigned
2025-01-07T09:48:26.938Z
606ae9ab0392350f35a22e37
https://cdn-avatars.huggingface.co/v1/production/uploads/606ae9ab0392350f35a22e37/7642Z9xUwSNf5oqFpGzyd.png
Johannes Hagemann
user
Johannes
677ca28f1fbb93b90f3d2485
Shangshang Wang
claimed_verified
2025-01-07T08:41:44.794Z
67469d6a8407f929491dce06
https://cdn-avatars.huggingface.co/v1/production/uploads/67469d6a8407f929491dce06/7vd7zyApmT4rXe58gqmG1.png
Shangshang Wang
user
upup-ashton-wang
677ca28f1fbb93b90f3d2486
Jason Wiemels
extracted_confirmed
2025-01-07T03:44:48.441Z
66197a8afeb55cbe39e50ae8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png
Ollie Liu
user
oliu-io
677ca28f1fbb93b90f3d2487
Jeff Kaufman
claimed_verified
2025-01-08T09:47:33.113Z
65319f3b6c7dac8c448f87a6
https://cdn-avatars.huggingface.co/v1/production/uploads/65319f3b6c7dac8c448f87a6/doj8efFaMdwdZzbjrKdK_.png
Jason Wiemels
user
jason136
677ca28f1fbb93b90f3d2488
Willie Neiswanger
claimed_verified
2025-01-07T08:41:42.587Z
644bf65522d211df6444a7f4
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
Willie Neiswanger
user
willieneis
METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
We pretrain METAGENE-1, a 7-billion-parameter autoregressive transformer
model, which we refer to as a metagenomic foundation model, on a novel corpus
of diverse metagenomic DNA and RNA sequences comprising over 1.5 trillion base
pairs. This dataset is sourced from a large collection of human wastewater
samples, processed and sequenced using deep metagenomic (next-generation)
sequencing methods. Unlike genomic models that focus on individual genomes or
curated sets of specific species, the aim of METAGENE-1 is to capture the full
distribution of genomic information present within this wastewater, to aid in
tasks relevant to pandemic monitoring and pathogen detection. We carry out
byte-pair encoding (BPE) tokenization on our dataset, tailored for metagenomic
sequences, and then pretrain our model. In this paper, we first detail the
pretraining dataset, tokenization strategy, and model architecture,
highlighting the considerations and design choices that enable the effective
modeling of metagenomic data. We then show results of pretraining this model on
our metagenomic dataset, providing details about our losses, system metrics,
and training stability over the course of pretraining. Finally, we demonstrate
the performance of METAGENE-1, which achieves state-of-the-art results on a set
of genomic benchmarks and new evaluations focused on human-pathogen detection
and genomic sequence embedding, showcasing its potential for public health
applications in pandemic monitoring, biosurveillance, and early detection of
emerging health threats.
A large autoregressive transformer model is pretrained on diverse metagenomic sequences for pandemic monitoring and pathogen detection, achieving high performance on genomic benchmarks.
autoregressive transformer metagenomic foundation model deep metagenomic sequencing byte-pair encoding metagenomic sequences pandemic monitoring pathogen detection genomic benchmarks genomic sequence embedding biosurveillance emerging health threats