title: ELIXR: Towards a general purpose X-ray artificial intelligence system
    through alignment of large language models and radiology vision encoders
summary: Our approach, which we call Embeddings for Language/Image-aligned X-Rays, or
  ELIXR, leverages a language-aligned image encoder combined or grafted onto a
  fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight
  adapter architecture using images paired with corresponding free-text radiology
  reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance
  on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13
  findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898
  across five findings (atelectasis, cardiomegaly, consolidation, pleural
  effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images)
  training data), and semantic search (0.76 normalized discounted cumulative gain
  (NDCG) across nineteen queries, including perfect retrieval on twelve of them).
  Compared to existing data-efficient methods including supervised contrastive
  learning (SupCon), ELIXR required two orders of magnitude less data to reach
  similar performance. ELIXR also showed promise on CXR vision-language tasks,
  demonstrating overall accuracies of 58.7% and 62.5% on visual question
  answering and report quality assurance tasks, respectively. These results
  suggest that ELIXR is a robust and versatile approach to CXR AI.
paper_authors._id: 64cb08e778fad0f5b671c25a
paper_authors.name: Shawn Xu
paper_authors._id: 64cb08e778fad0f5b671c25b
paper_authors.name: Lin Yang
paper_authors._id: 64cb08e778fad0f5b671c25c
paper_authors.name: Christopher Kelly
paper_authors._id: 64cb08e778fad0f5b671c25d
paper_authors.name: Marcin Sieniek
paper_authors._id: 64cb08e778fad0f5b671c25e
paper_authors.name: Timo Kohlberger
paper_authors._id: 64cb08e778fad0f5b671c25f
paper_authors.name: Martin Ma
paper_authors._id: 64cb08e778fad0f5b671c260
paper_authors.name: Wei-Hung Weng
paper_authors._id: 64cb08e778fad0f5b671c261
paper_authors.name: Attila Kiraly
paper_authors._id: 64cb08e778fad0f5b671c262
paper_authors.name: Sahar Kazemzadeh
paper_authors._id: 64cb08e778fad0f5b671c263
paper_authors.name: Zakkai Melamed
paper_authors._id: 64cb08e778fad0f5b671c264
paper_authors.name: Jungyeon Park
paper_authors._id: 64cb08e778fad0f5b671c265
paper_authors.name: Patricia Strachan
paper_authors._id: 64cb08e778fad0f5b671c266
paper_authors.name: Yun Liu
paper_authors._id: 64cb08e778fad0f5b671c267
paper_authors.name: Chuck Lau
paper_authors._id: 64cb08e778fad0f5b671c268
paper_authors.name: Preeti Singh
paper_authors._id: 64cb08e778fad0f5b671c269
paper_authors.name: Christina Chen
paper_authors._id: 64cb08e778fad0f5b671c26a
paper_authors.name: Mozziyar Etemadi
paper_authors._id: 64cb08e778fad0f5b671c26b
paper_authors.name: Sreenivasa Raju Kalidindi
paper_authors._id: 64cb08e778fad0f5b671c26c
paper_authors.name: Yossi Matias
paper_authors._id: 64cb08e778fad0f5b671c26d
paper_authors.name: Katherine Chou
paper_authors._id: 64cb08e778fad0f5b671c26e
paper_authors.name: Greg S. Corrado
paper_authors._id: 64cb08e778fad0f5b671c26f
paper_authors.name: Shravya Shetty
paper_authors._id: 64cb08e778fad0f5b671c270
paper_authors.name: Daniel Tse
paper_authors._id: 64cb08e778fad0f5b671c271
paper_authors.name: Shruthi Prabhakara
paper_authors._id: 64cb08e778fad0f5b671c272
paper_authors.name: Daniel Golden
paper_authors._id: 64cb08e778fad0f5b671c273
paper_authors.name: Rory Pilgrim
paper_authors._id: 64cb08e778fad0f5b671c274
paper_authors.name: Krish Eswaran
paper_authors._id: 64cb08e778fad0f5b671c275
paper_authors.name: Andrew Sellergren
paper_title: ELIXR: Towards a general purpose X-ray artificial intelligence system
    through alignment of large language models and radiology vision encoders
paper_summary: Our approach, which we call Embeddings for Language/Image-aligned X-Rays, or
  ELIXR, leverages a language-aligned image encoder combined or grafted onto a
  fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight
  adapter architecture using images paired with corresponding free-text radiology
  reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance
  on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13
  findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898
  across five findings (atelectasis, cardiomegaly, consolidation, pleural
  effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images)
  training data), and semantic search (0.76 normalized discounted cumulative gain
  (NDCG) across nineteen queries, including perfect retrieval on twelve of them).
  Compared to existing data-efficient methods including supervised contrastive
  learning (SupCon), ELIXR required two orders of magnitude less data to reach
  similar performance. ELIXR also showed promise on CXR vision-language tasks,
  demonstrating overall accuracies of 58.7% and 62.5% on visual question
  answering and report quality assurance tasks, respectively. These results
  suggest that ELIXR is a robust and versatile approach to CXR AI.
paper_ai_summary: ELIXR, a lightweight adapter architecture combining a language-aligned image encoder with PaLM 2, achieves state-of-the-art performance in zero-shot and data-efficient chest X-ray classification and semantic search, requiring significantly less data compared to existing methods.
paper_ai_keywords: language-aligned image encoder LLM PaLM 2 adapter architecture MIMIC-CXR dataset zero-shot chest X-ray classification data-efficient CXR classification mean AUC NDCG supervised contrastive learning (SupCon) visual question answering report quality assurance