title: METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
summary: We pretrain METAGENE-1, a 7-billion-parameter autoregressive transformer
  model, which we refer to as a metagenomic foundation model, on a novel corpus
  of diverse metagenomic DNA and RNA sequences comprising over 1.5 trillion base
  pairs. This dataset is sourced from a large collection of human wastewater
  samples, processed and sequenced using deep metagenomic (next-generation)
  sequencing methods. Unlike genomic models that focus on individual genomes or
  curated sets of specific species, the aim of METAGENE-1 is to capture the full
  distribution of genomic information present within this wastewater, to aid in
  tasks relevant to pandemic monitoring and pathogen detection. We carry out
  byte-pair encoding (BPE) tokenization on our dataset, tailored for metagenomic
  sequences, and then pretrain our model. In this paper, we first detail the
  pretraining dataset, tokenization strategy, and model architecture,
  highlighting the considerations and design choices that enable the effective
  modeling of metagenomic data. We then show results of pretraining this model on
  our metagenomic dataset, providing details about our losses, system metrics,
  and training stability over the course of pretraining. Finally, we demonstrate
  the performance of METAGENE-1, which achieves state-of-the-art results on a set
  of genomic benchmarks and new evaluations focused on human-pathogen detection
  and genomic sequence embedding, showcasing its potential for public health
  applications in pandemic monitoring, biosurveillance, and early detection of
  emerging health threats.
paper_authors._id: 677ca28f1fbb93b90f3d2482
paper_authors.name: Ollie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:48:10.277Z
paper_authors._id: 66197a8afeb55cbe39e50ae8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png
paper_authors.fullname: Ollie Liu
paper_authors.type: user
paper_authors.user: oliu-io
paper_authors._id: 677ca28f1fbb93b90f3d2483
paper_authors.name: Sami Jaghouar
paper_authors._id: 677ca28f1fbb93b90f3d2484
paper_authors.name: Johannes Hagemann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:48:26.938Z
paper_authors._id: 606ae9ab0392350f35a22e37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617619292851-noauth.jpeg
paper_authors.fullname: Johannes Hagemann
paper_authors.type: user
paper_authors.user: Johannes
paper_authors._id: 677ca28f1fbb93b90f3d2485
paper_authors.name: Shangshang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:41:44.794Z
paper_authors._id: 67469d6a8407f929491dce06
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67469d6a8407f929491dce06/ZJ17ItOIG6X6yqqo4FQMp.jpeg
paper_authors.fullname: Shangshang Wang
paper_authors.type: user
paper_authors.user: upup-ashton-wang
paper_authors._id: 677ca28f1fbb93b90f3d2486
paper_authors.name: Jason Wiemels
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-07T03:44:48.441Z
paper_authors._id: 66197a8afeb55cbe39e50ae8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png
paper_authors.fullname: Ollie Liu
paper_authors.type: user
paper_authors.user: oliu-io
paper_authors._id: 677ca28f1fbb93b90f3d2487
paper_authors.name: Jeff Kaufman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T09:47:33.113Z
paper_authors._id: 65319f3b6c7dac8c448f87a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65319f3b6c7dac8c448f87a6/doj8efFaMdwdZzbjrKdK_.png
paper_authors.fullname: Jason Wiemels
paper_authors.type: user
paper_authors.user: jason136
paper_authors._id: 677ca28f1fbb93b90f3d2488
paper_authors.name: Willie Neiswanger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:41:42.587Z
paper_authors._id: 644bf65522d211df6444a7f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
paper_authors.fullname: Willie Neiswanger
paper_authors.type: user
paper_authors.user: willieneis
paper_title: METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
paper_summary: We pretrain METAGENE-1, a 7-billion-parameter autoregressive transformer
  model, which we refer to as a metagenomic foundation model, on a novel corpus
  of diverse metagenomic DNA and RNA sequences comprising over 1.5 trillion base
  pairs. This dataset is sourced from a large collection of human wastewater
  samples, processed and sequenced using deep metagenomic (next-generation)
  sequencing methods. Unlike genomic models that focus on individual genomes or
  curated sets of specific species, the aim of METAGENE-1 is to capture the full
  distribution of genomic information present within this wastewater, to aid in
  tasks relevant to pandemic monitoring and pathogen detection. We carry out
  byte-pair encoding (BPE) tokenization on our dataset, tailored for metagenomic
  sequences, and then pretrain our model. In this paper, we first detail the
  pretraining dataset, tokenization strategy, and model architecture,
  highlighting the considerations and design choices that enable the effective
  modeling of metagenomic data. We then show results of pretraining this model on
  our metagenomic dataset, providing details about our losses, system metrics,
  and training stability over the course of pretraining. Finally, we demonstrate
  the performance of METAGENE-1, which achieves state-of-the-art results on a set
  of genomic benchmarks and new evaluations focused on human-pathogen detection
  and genomic sequence embedding, showcasing its potential for public health
  applications in pandemic monitoring, biosurveillance, and early detection of
  emerging health threats.
paper_ai_keywords: autoregressive transformer metagenomic foundation model byte-pair encoding (BPE) tokenization genomic data pretraining losses system metrics training stability state-of-the-art results genomic sequence embedding pandemic monitoring biosurveillance emerging health threats