title: Clinical ModernBERT: An efficient and long context encoder for
    biomedical text
summary: We introduce Clinical ModernBERT, a transformer based encoder pretrained on
  large scale biomedical literature, clinical notes, and medical ontologies,
  incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with
  their textual descriptions. Building on ModernBERT the current state of the art
  natural language text encoder featuring architectural upgrades such as rotary
  positional embeddings (RoPE), Flash Attention, and extended context length up
  to 8,192 tokens our model adapts these innovations specifically for biomedical
  and clinical domains. Clinical ModernBERT excels at producing semantically rich
  representations tailored for long context tasks. We validate this both by
  analyzing its pretrained weights and through empirical evaluation on a
  comprehensive suite of clinical NLP benchmarks.
paper_authors._id: 67f4889c7e1624ebbaef710d
paper_authors.name: Simon A. Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:51:44.271Z
paper_authors._id: 652ee41e7b0079ff035b2269
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652ee41e7b0079ff035b2269/jv-2-UPHN_e9Lhwl3Evwh.jpeg
paper_authors.fullname: Simon Lee
paper_authors.type: user
paper_authors.user: Simonlee711
paper_authors._id: 67f4889c7e1624ebbaef710e
paper_authors.name: Anthony Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:55:49.038Z
paper_authors._id: 66ae56e72bc4568096d770df
paper_authors.avatarUrl: /avatars/bcb255a24583c32719985dafc5f35e5a.svg
paper_authors.fullname: Anthony Wu
paper_authors.type: user
paper_authors.user: anthonytkwu
paper_authors._id: 67f4889c7e1624ebbaef710f
paper_authors.name: Jeffrey N. Chiang
paper_title: Clinical ModernBERT: An efficient and long context encoder for
    biomedical text
paper_summary: We introduce Clinical ModernBERT, a transformer based encoder pretrained on
  large scale biomedical literature, clinical notes, and medical ontologies,
  incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with
  their textual descriptions. Building on ModernBERT the current state of the art
  natural language text encoder featuring architectural upgrades such as rotary
  positional embeddings (RoPE), Flash Attention, and extended context length up
  to 8,192 tokens our model adapts these innovations specifically for biomedical
  and clinical domains. Clinical ModernBERT excels at producing semantically rich
  representations tailored for long context tasks. We validate this both by
  analyzing its pretrained weights and through empirical evaluation on a
  comprehensive suite of clinical NLP benchmarks.
paper_ai_summary: Clinical ModernBERT, a transformer-based encoder pre-trained on extensive biomedical data, incorporates innovative features like rotary positional embeddings and Flash Attention, excelling in long-context clinical NLP tasks.
paper_ai_keywords: transformer based encoder rotary positional embeddings Flash Attention biomedical literature clinical notes medical ontologies PubMed abstracts MIMIC IV clinical data medical codes semantically rich representations clinical NLP benchmarks