title: Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
summary: Large-scale sequence modeling has sparked rapid advances that now extend into
  biology and genomics. However, modeling genomic sequences introduces challenges
  such as the need to model long-range token interactions, the effects of
  upstream and downstream regions of the genome, and the reverse complementarity
  (RC) of DNA. Here, we propose an architecture motivated by these challenges
  that builds off the long-range Mamba block, and extends it to a BiMamba
  component that supports bi-directionality, and to a MambaDNA block that
  additionally supports RC equivariance. We use MambaDNA as the basis of
  Caduceus, the first family of RC equivariant bi-directional long-range DNA
  language models, and we introduce pre-training and fine-tuning strategies that
  yield Caduceus DNA foundation models. Caduceus outperforms previous long-range
  models on downstream benchmarks; on a challenging long-range variant effect
  prediction task, Caduceus exceeds the performance of 10x larger models that do
  not leverage bi-directionality or equivariance.
paper_authors._id: 65e923a00ba2a33b5a7da86f
paper_authors.name: Yair Schiff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:11:54.031Z
paper_authors._id: 6418e5e722270b3ccf15dfef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418e5e722270b3ccf15dfef/ZiammTpL9FVMuE7DN8n5B.jpeg
paper_authors.fullname: Yair Schiff
paper_authors.type: user
paper_authors.user: yairschiff
paper_authors._id: 65e923a00ba2a33b5a7da870
paper_authors.name: Chia-Hsiang Kao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:12:10.628Z
paper_authors._id: 6501eff07c3f4b17b897fa0b
paper_authors.avatarUrl: /avatars/1e47934d03927c6e9b549083d7e7df7e.svg
paper_authors.fullname: Chia-Hsiang Kao
paper_authors.type: user
paper_authors.user: AaronKao
paper_authors._id: 65e923a00ba2a33b5a7da871
paper_authors.name: Aaron Gokaslan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:12:17.121Z
paper_authors._id: 620573d0522e40b4a18d8763
paper_authors.avatarUrl: /avatars/9353c064ef8ccac84d0397411d38fa90.svg
paper_authors.fullname: Aaron Gokaslan
paper_authors.type: user
paper_authors.user: Skylion007
paper_authors._id: 65e923a00ba2a33b5a7da872
paper_authors.name: Tri Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:12:24.009Z
paper_authors._id: 64b8a6b5cf14c2fabe98159b
paper_authors.avatarUrl: /avatars/dbc009451865435bf290791beadc4723.svg
paper_authors.fullname: Tri Dao
paper_authors.type: user
paper_authors.user: tridao
paper_authors._id: 65e923a00ba2a33b5a7da873
paper_authors.name: Albert Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:12:39.524Z
paper_authors._id: 642619b62864ea9e59de834e
paper_authors.avatarUrl: /avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
paper_authors.fullname: Albert Gu
paper_authors.type: user
paper_authors.user: albertgu
paper_authors._id: 65e923a00ba2a33b5a7da874
paper_authors.name: Volodymyr Kuleshov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:12:47.046Z
paper_authors._id: 640ea1c43c82bd463ee80e19
paper_authors.avatarUrl: /avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
paper_authors.fullname: Volodymyr Kuleshov
paper_authors.type: user
paper_authors.user: kuleshov
paper_title: Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
paper_summary: Large-scale sequence modeling has sparked rapid advances that now extend into
  biology and genomics. However, modeling genomic sequences introduces challenges
  such as the need to model long-range token interactions, the effects of
  upstream and downstream regions of the genome, and the reverse complementarity
  (RC) of DNA. Here, we propose an architecture motivated by these challenges
  that builds off the long-range Mamba block, and extends it to a BiMamba
  component that supports bi-directionality, and to a MambaDNA block that
  additionally supports RC equivariance. We use MambaDNA as the basis of
  Caduceus, the first family of RC equivariant bi-directional long-range DNA
  language models, and we introduce pre-training and fine-tuning strategies that
  yield Caduceus DNA foundation models. Caduceus outperforms previous long-range
  models on downstream benchmarks; on a challenging long-range variant effect
  prediction task, Caduceus exceeds the performance of 10x larger models that do
  not leverage bi-directionality or equivariance.
paper_ai_summary: A new family of DNA language models, Caduceus, supports bi-directionality and reverse complementarity, achieving superior performance on long-range genomic sequence modeling tasks.
paper_ai_keywords: sequence modeling genomic sequences long-range token interactions reverse complementarity Mamba block BiMamba MambaDNA RC equivariant long-range DNA language models pre-training fine-tuning variant effect prediction