title: Rare Disease Differential Diagnosis with Large Language Models at Scale:
    From Abdominal Actinomycosis to Wilson's Disease
summary: Large language models (LLMs) have demonstrated impressive capabilities in
  disease diagnosis. However, their effectiveness in identifying rarer diseases,
  which are inherently more challenging to diagnose, remains an open question.
  Rare disease performance is critical with the increasing use of LLMs in
  healthcare settings. This is especially true if a primary care physician needs
  to make a rarer prognosis from only a patient conversation so that they can
  take the appropriate next step. To that end, several clinical decision support
  systems are designed to support providers in rare disease identification. Yet
  their utility is limited due to their lack of knowledge of common disorders and
  difficulty of use.
    In this paper, we propose RareScale to combine the knowledge LLMs with expert
  systems. We use jointly use an expert system and LLM to simulate rare disease
  chats. This data is used to train a rare disease candidate predictor model.
  Candidates from this smaller model are then used as additional inputs to
  black-box LLM to make the final differential diagnosis. Thus, RareScale allows
  for a balance between rare and common diagnoses. We present results on over 575
  rare diseases, beginning with Abdominal Actinomycosis and ending with Wilson's
  Disease. Our approach significantly improves the baseline performance of
  black-box LLMs by over 17% in Top-5 accuracy. We also find that our candidate
  generation performance is high (e.g. 88.8% on gpt-4o generated chats).
paper_authors._id: 67bcdc646840d9686fcea432
paper_authors.name: Elliot Schumacher
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-24T20:54:12.397Z
paper_authors._id: 64f74beb4db24c1ca9379afc
paper_authors.avatarUrl: /avatars/c29b7d000ff65c590a5aec2d3262edd9.svg
paper_authors.fullname: Elliot Schumacher
paper_authors.type: user
paper_authors.user: elliotschu
paper_authors._id: 67bcdc646840d9686fcea433
paper_authors.name: Dhruv Naik
paper_authors._id: 67bcdc646840d9686fcea434
paper_authors.name: Anitha Kannan
paper_title: Rare Disease Differential Diagnosis with Large Language Models at Scale:
    From Abdominal Actinomycosis to Wilson's Disease
paper_summary: Large language models (LLMs) have demonstrated impressive capabilities in
  disease diagnosis. However, their effectiveness in identifying rarer diseases,
  which are inherently more challenging to diagnose, remains an open question.
  Rare disease performance is critical with the increasing use of LLMs in
  healthcare settings. This is especially true if a primary care physician needs
  to make a rarer prognosis from only a patient conversation so that they can
  take the appropriate next step. To that end, several clinical decision support
  systems are designed to support providers in rare disease identification. Yet
  their utility is limited due to their lack of knowledge of common disorders and
  difficulty of use.
    In this paper, we propose RareScale to combine the knowledge LLMs with expert
  systems. We use jointly use an expert system and LLM to simulate rare disease
  chats. This data is used to train a rare disease candidate predictor model.
  Candidates from this smaller model are then used as additional inputs to
  black-box LLM to make the final differential diagnosis. Thus, RareScale allows
  for a balance between rare and common diagnoses. We present results on over 575
  rare diseases, beginning with Abdominal Actinomycosis and ending with Wilson's
  Disease. Our approach significantly improves the baseline performance of
  black-box LLMs by over 17% in Top-5 accuracy. We also find that our candidate
  generation performance is high (e.g. 88.8% on gpt-4o generated chats).
paper_ai_keywords: large language models (LLMs) rare disease performance clinical decision support systems expert systems rare disease chats rare disease candidate predictor model black-box LLM Top-5 accuracy candidate generationperformance