title: HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide
    Resolution
summary: Genomic (DNA) sequences encode an enormous amount of information for gene
  regulation and protein synthesis. Similar to natural language models,
  researchers have proposed foundation models in genomics to learn generalizable
  features from unlabeled genome data that can then be fine-tuned for downstream
  tasks such as identifying regulatory elements. Due to the quadratic scaling of
  attention, previous Transformer-based genomic models have used 512 to 4k tokens
  as context (<0.001% of the human genome), significantly limiting the modeling
  of long-range interactions in DNA. In addition, these methods rely on
  tokenizers to aggregate meaningful DNA units, losing single nucleotide
  resolution where subtle genetic variations can completely alter protein
  function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large
  language model based on implicit convolutions was shown to match attention in
  quality while allowing longer context lengths and lower time complexity.
  Leveraging Hyenas new long-range capabilities, we present HyenaDNA, a genomic
  foundation model pretrained on the human reference genome with context lengths
  of up to 1 million tokens at the single nucleotide-level, an up to 500x
  increase over previous dense attention-based models. HyenaDNA scales
  sub-quadratically in sequence length (training up to 160x faster than
  Transformer), uses single nucleotide tokens, and has full global context at
  each layer. We explore what longer context enables - including the first use of
  in-context learning in genomics for simple adaptation to novel tasks without
  updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide
  Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 17 datasets
  using a model with orders of magnitude less parameters and pretraining data. On
  the GenomicBenchmarks, HyenaDNA surpasses SotA on all 8 datasets on average by
  +9 accuracy points.
paper_authors._id: 649cd97cc1ae48cf4d8b85aa
paper_authors.name: Eric Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-29T20:55:46.401Z
paper_authors._id: 63f87f8ada440a47e9f5b7f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f87f8ada440a47e9f5b7f0/LE4CUPwT6khOU-gib520a.jpeg
paper_authors.fullname: Eric Nguyen
paper_authors.type: user
paper_authors.user: exnx
paper_authors._id: 649cd97cc1ae48cf4d8b85ab
paper_authors.name: Michael Poli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-29T20:55:41.989Z
paper_authors._id: 62a1306bbe7fa896d2c8de44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a1306bbe7fa896d2c8de44/ndUBVtc5A3nsfUE47vood.png
paper_authors.fullname: Michael Poli
paper_authors.type: user
paper_authors.user: Zymrael
paper_authors._id: 649cd97cc1ae48cf4d8b85ac
paper_authors.name: Marjan Faizi
paper_authors._id: 649cd97cc1ae48cf4d8b85ad
paper_authors.name: Armin Thomas
paper_authors._id: 649cd97cc1ae48cf4d8b85ae
paper_authors.name: Callum Birch-Sykes
paper_authors._id: 649cd97cc1ae48cf4d8b85af
paper_authors.name: Michael Wornow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:08:16.716Z
paper_authors._id: 6025f42e0703121c822a80b4
paper_authors.avatarUrl: /avatars/5a50df2c0618b49739bb6db57d1aa2b8.svg
paper_authors.fullname: Michael Wornow
paper_authors.type: user
paper_authors.user: Miking98
paper_authors._id: 649cd97cc1ae48cf4d8b85b0
paper_authors.name: Aman Patel
paper_authors._id: 649cd97cc1ae48cf4d8b85b1
paper_authors.name: Clayton Rabideau
paper_authors._id: 649cd97cc1ae48cf4d8b85b2
paper_authors.name: Stefano Massaroli
paper_authors._id: 649cd97cc1ae48cf4d8b85b3
paper_authors.name: Yoshua Bengio
paper_authors._id: 649cd97cc1ae48cf4d8b85b4
paper_authors.name: Stefano Ermon
paper_authors._id: 649cd97cc1ae48cf4d8b85b5
paper_authors.name: Stephen A. Baccus
paper_authors._id: 649cd97cc1ae48cf4d8b85b6
paper_authors.name: Chris Ré
paper_title: HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide
    Resolution
paper_summary: Genomic (DNA) sequences encode an enormous amount of information for gene
  regulation and protein synthesis. Similar to natural language models,
  researchers have proposed foundation models in genomics to learn generalizable
  features from unlabeled genome data that can then be fine-tuned for downstream
  tasks such as identifying regulatory elements. Due to the quadratic scaling of
  attention, previous Transformer-based genomic models have used 512 to 4k tokens
  as context (<0.001% of the human genome), significantly limiting the modeling
  of long-range interactions in DNA. In addition, these methods rely on
  tokenizers to aggregate meaningful DNA units, losing single nucleotide
  resolution where subtle genetic variations can completely alter protein
  function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large
  language model based on implicit convolutions was shown to match attention in
  quality while allowing longer context lengths and lower time complexity.
  Leveraging Hyenas new long-range capabilities, we present HyenaDNA, a genomic
  foundation model pretrained on the human reference genome with context lengths
  of up to 1 million tokens at the single nucleotide-level, an up to 500x
  increase over previous dense attention-based models. HyenaDNA scales
  sub-quadratically in sequence length (training up to 160x faster than
  Transformer), uses single nucleotide tokens, and has full global context at
  each layer. We explore what longer context enables - including the first use of
  in-context learning in genomics for simple adaptation to novel tasks without
  updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide
  Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 17 datasets
  using a model with orders of magnitude less parameters and pretraining data. On
  the GenomicBenchmarks, HyenaDNA surpasses SotA on all 8 datasets on average by
  +9 accuracy points.
paper_ai_summary: HyenaDNA, a genomic foundation model based on implicit convolutions, achieves state-of-the-art performance on various benchmarks with longer context lengths, single nucleotide resolution, and fewer parameters.
paper_ai_keywords: Transformer implicit convolutions Hyena context lengths single nucleotide-level in-context learning Nucleotide Transformer GenomicBenchmarks