ELIXR: Towards a general purpose X-ray artificial intelligence system
  through alignment of large language models and radiology vision encoders
Our approach, which we call Embeddings for Language/Image-aligned X-Rays, or
ELIXR, leverages a language-aligned image encoder combined or grafted onto a
fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight
adapter architecture using images paired with corresponding free-text radiology
reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance
on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13
findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898
across five findings (atelectasis, cardiomegaly, consolidation, pleural
effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images)
training data), and semantic search (0.76 normalized discounted cumulative gain
(NDCG) across nineteen queries, including perfect retrieval on twelve of them).
Compared to existing data-efficient methods including supervised contrastive
learning (SupCon), ELIXR required two orders of magnitude less data to reach
similar performance. ELIXR also showed promise on CXR vision-language tasks,
demonstrating overall accuracies of 58.7% and 62.5% on visual question
answering and report quality assurance tasks, respectively. These results
suggest that ELIXR is a robust and versatile approach to CXR AI.
64cb08e778fad0f5b671c25a
Shawn Xu
64cb08e778fad0f5b671c25b
Lin Yang
64cb08e778fad0f5b671c25c
Christopher Kelly
64cb08e778fad0f5b671c25d
Marcin Sieniek
64cb08e778fad0f5b671c25e
Timo Kohlberger
64cb08e778fad0f5b671c25f
Martin Ma
64cb08e778fad0f5b671c260
Wei-Hung Weng
64cb08e778fad0f5b671c261
Attila Kiraly
64cb08e778fad0f5b671c262
Sahar Kazemzadeh
64cb08e778fad0f5b671c263
Zakkai Melamed
64cb08e778fad0f5b671c264
Jungyeon Park
64cb08e778fad0f5b671c265
Patricia Strachan
64cb08e778fad0f5b671c266
Yun Liu
64cb08e778fad0f5b671c267
Chuck Lau
64cb08e778fad0f5b671c268
Preeti Singh
64cb08e778fad0f5b671c269
Christina Chen
64cb08e778fad0f5b671c26a
Mozziyar Etemadi
64cb08e778fad0f5b671c26b
Sreenivasa Raju Kalidindi
64cb08e778fad0f5b671c26c
Yossi Matias
64cb08e778fad0f5b671c26d
Katherine Chou
64cb08e778fad0f5b671c26e
Greg S. Corrado
64cb08e778fad0f5b671c26f
Shravya Shetty
64cb08e778fad0f5b671c270
Daniel Tse
64cb08e778fad0f5b671c271
Shruthi Prabhakara
64cb08e778fad0f5b671c272
Daniel Golden
64cb08e778fad0f5b671c273
Rory Pilgrim
64cb08e778fad0f5b671c274
Krish Eswaran
64cb08e778fad0f5b671c275
Andrew Sellergren
ELIXR: Towards a general purpose X-ray artificial intelligence system
  through alignment of large language models and radiology vision encoders
Our approach, which we call Embeddings for Language/Image-aligned X-Rays, or
ELIXR, leverages a language-aligned image encoder combined or grafted onto a
fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight
adapter architecture using images paired with corresponding free-text radiology
reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance
on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13
findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898
across five findings (atelectasis, cardiomegaly, consolidation, pleural
effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images)
training data), and semantic search (0.76 normalized discounted cumulative gain
(NDCG) across nineteen queries, including perfect retrieval on twelve of them).
Compared to existing data-efficient methods including supervised contrastive
learning (SupCon), ELIXR required two orders of magnitude less data to reach
similar performance. ELIXR also showed promise on CXR vision-language tasks,
demonstrating overall accuracies of 58.7% and 62.5% on visual question
answering and report quality assurance tasks, respectively. These results
suggest that ELIXR is a robust and versatile approach to CXR AI.
ELIXR, a lightweight adapter architecture combining a language-aligned image encoder with PaLM 2, achieves state-of-the-art performance in zero-shot and data-efficient chest X-ray classification and semantic search, requiring significantly less data compared to existing methods.
language-aligned image encoder LLM PaLM 2 adapter architecture MIMIC-CXR dataset zero-shot chest X-ray classification data-efficient CXR classification mean AUC NDCG supervised contrastive learning (SupCon) visual question answering report quality assurance