title: BioMamba: A Pre-trained Biomedical Language Representation Model
    Leveraging Mamba
summary: The advancement of natural language processing (NLP) in biology hinges on
  models' ability to interpret intricate biomedical literature. Traditional
  models often struggle with the complex and domain-specific language in this
  field. In this paper, we present BioMamba, a pre-trained model specifically
  designed for biomedical text mining. BioMamba builds upon the Mamba
  architecture and is pre-trained on an extensive corpus of biomedical
  literature. Our empirical studies demonstrate that BioMamba significantly
  outperforms models like BioBERT and general-domain Mamba across various
  biomedical tasks. For instance, BioMamba achieves a 100 times reduction in
  perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test
  set. We provide an overview of the model architecture, pre-training process,
  and fine-tuning techniques. Additionally, we release the code and trained model
  to facilitate further research.
paper_authors._id: 66b1f44fba6dc98b702963ff
paper_authors.name: Ling Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:41:51.097Z
paper_authors._id: 66739484a660dfbb2643eb3d
paper_authors.avatarUrl: /avatars/c9a6c6a1e295e448dacfccb1a2860e8a.svg
paper_authors.fullname: Leo Y
paper_authors.type: user
paper_authors.user: LeoYML
paper_authors._id: 66b1f44fba6dc98b70296400
paper_authors.name: Sixue Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T11:59:49.024Z
paper_authors._id: 66217de1c92239f4997a4c75
paper_authors.avatarUrl: /avatars/b1c4d74878b62ffdb07f65971b4f5501.svg
paper_authors.fullname: Sixue Xing
paper_authors.type: user
paper_authors.user: sixuexing
paper_authors._id: 66b1f44fba6dc98b70296401
paper_authors.name: Yingzhou Lu
paper_authors._id: 66b1f44fba6dc98b70296402
paper_authors.name: Tianfan Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T11:59:36.574Z
paper_authors._id: 6322429c7bb41a713daf40fb
paper_authors.avatarUrl: /avatars/e226463a33ac51d7052eb6bc6d35bed5.svg
paper_authors.fullname: Tianfan Fu
paper_authors.type: user
paper_authors.user: TianfanFu
paper_title: BioMamba: A Pre-trained Biomedical Language Representation Model
    Leveraging Mamba
paper_summary: The advancement of natural language processing (NLP) in biology hinges on
  models' ability to interpret intricate biomedical literature. Traditional
  models often struggle with the complex and domain-specific language in this
  field. In this paper, we present BioMamba, a pre-trained model specifically
  designed for biomedical text mining. BioMamba builds upon the Mamba
  architecture and is pre-trained on an extensive corpus of biomedical
  literature. Our empirical studies demonstrate that BioMamba significantly
  outperforms models like BioBERT and general-domain Mamba across various
  biomedical tasks. For instance, BioMamba achieves a 100 times reduction in
  perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test
  set. We provide an overview of the model architecture, pre-training process,
  and fine-tuning techniques. Additionally, we release the code and trained model
  to facilitate further research.
paper_ai_keywords: pre-trained model biomedical text mining Mamba architecture pre-training process fine-tuning techniques BioMamba BioBERT Biomedical tasks perplexity cross-entropy loss BioASQ BioMamba architecture trained model