title: Towards Generalist Biomedical AI
summary: Medicine is inherently multimodal, with rich data modalities spanning text,
  imaging, genomics, and more. Generalist biomedical artificial intelligence (AI)
  systems that flexibly encode, integrate, and interpret this data at scale can
  potentially enable impactful applications ranging from scientific discovery to
  care delivery. To enable the development of these models, we first curate
  MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses
  14 diverse tasks such as medical question answering, mammography and
  dermatology image interpretation, radiology report generation and
  summarization, and genomic variant calling. We then introduce Med-PaLM
  Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI
  system. Med-PaLM M is a large multimodal generative model that flexibly encodes
  and interprets biomedical data including clinical language, imaging, and
  genomics with the same set of model weights. Med-PaLM M reaches performance
  competitive with or exceeding the state of the art on all MultiMedBench tasks,
  often surpassing specialist models by a wide margin. We also report examples of
  zero-shot generalization to novel medical concepts and tasks, positive transfer
  learning across tasks, and emergent zero-shot medical reasoning. To further
  probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist
  evaluation of model-generated (and human) chest X-ray reports and observe
  encouraging performance across model scales. In a side-by-side ranking on 246
  retrospective chest X-rays, clinicians express a pairwise preference for
  Med-PaLM M reports over those produced by radiologists in up to 40.50% of
  cases, suggesting potential clinical utility. While considerable work is needed
  to validate these models in real-world use cases, our results represent a
  milestone towards the development of generalist biomedical AI systems.
paper_authors._id: 64c21c46245c55a21c79e14f
paper_authors.name: Tao Tu
paper_authors._id: 64c21c46245c55a21c79e150
paper_authors.name: Shekoofeh Azizi
paper_authors._id: 64c21c46245c55a21c79e151
paper_authors.name: Danny Driess
paper_authors._id: 64c21c46245c55a21c79e152
paper_authors.name: Mike Schaekermann
paper_authors._id: 64c21c46245c55a21c79e153
paper_authors.name: Mohamed Amin
paper_authors._id: 64c21c46245c55a21c79e154
paper_authors.name: Pi-Chuan Chang
paper_authors._id: 64c21c46245c55a21c79e155
paper_authors.name: Andrew Carroll
paper_authors._id: 64c21c46245c55a21c79e156
paper_authors.name: Chuck Lau
paper_authors._id: 64c21c46245c55a21c79e157
paper_authors.name: Ryutaro Tanno
paper_authors._id: 64c21c46245c55a21c79e158
paper_authors.name: Ira Ktena
paper_authors._id: 64c21c46245c55a21c79e159
paper_authors.name: Basil Mustafa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:33:25.021Z
paper_authors._id: 63cfcad6e23b90128c66685c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
paper_authors.fullname: Basil Mustafa
paper_authors.type: user
paper_authors.user: BasilMustafa
paper_authors._id: 64c21c46245c55a21c79e15a
paper_authors.name: Aakanksha Chowdhery
paper_authors._id: 64c21c46245c55a21c79e15b
paper_authors.name: Yun Liu
paper_authors._id: 64c21c46245c55a21c79e15c
paper_authors.name: Simon Kornblith
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:35:14.471Z
paper_authors._id: 6443ffc2af034cdfd69c5c00
paper_authors.avatarUrl: /avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
paper_authors.fullname: Simon Kornblith
paper_authors.type: user
paper_authors.user: simonsterk
paper_authors._id: 64c21c46245c55a21c79e15d
paper_authors.name: David Fleet
paper_authors._id: 64c21c46245c55a21c79e15e
paper_authors.name: Philip Mansfield
paper_authors._id: 64c21c46245c55a21c79e15f
paper_authors.name: Sushant Prakash
paper_authors._id: 64c21c46245c55a21c79e160
paper_authors.name: Renee Wong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:35:48.270Z
paper_authors._id: 6397fb67a51b5dbfc93ca0d1
paper_authors.avatarUrl: /avatars/b254d4df8e4158e2dbd047ee077440e3.svg
paper_authors.fullname: Wong
paper_authors.type: user
paper_authors.user: reneewong60
paper_authors._id: 64c21c46245c55a21c79e161
paper_authors.name: Sunny Virmani
paper_authors._id: 64c21c46245c55a21c79e162
paper_authors.name: Christopher Semturs
paper_authors._id: 64c21c46245c55a21c79e163
paper_authors.name: S Sara Mahdavi
paper_authors._id: 64c21c46245c55a21c79e164
paper_authors.name: Bradley Green
paper_authors._id: 64c21c46245c55a21c79e165
paper_authors.name: Ewa Dominowska
paper_authors._id: 64c21c46245c55a21c79e166
paper_authors.name: Blaise Aguera y Arcas
paper_authors._id: 64c21c46245c55a21c79e167
paper_authors.name: Joelle Barral
paper_authors._id: 64c21c46245c55a21c79e168
paper_authors.name: Dale Webster
paper_authors._id: 64c21c46245c55a21c79e169
paper_authors.name: Greg S. Corrado
paper_authors._id: 64c21c46245c55a21c79e16a
paper_authors.name: Yossi Matias
paper_authors._id: 64c21c46245c55a21c79e16b
paper_authors.name: Karan Singhal
paper_authors._id: 64c21c46245c55a21c79e16c
paper_authors.name: Pete Florence
paper_authors._id: 64c21c46245c55a21c79e16d
paper_authors.name: Alan Karthikesalingam
paper_authors._id: 64c21c46245c55a21c79e16e
paper_authors.name: Vivek Natarajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:37:42.600Z
paper_authors._id: 617a6af2941993035fbaf29c
paper_authors.avatarUrl: /avatars/a389671494782f06ff8502c934090cc0.svg
paper_authors.fullname: Vivek Natarajan
paper_authors.type: user
paper_authors.user: vivnat
paper_title: Towards Generalist Biomedical AI
paper_summary: Medicine is inherently multimodal, with rich data modalities spanning text,
  imaging, genomics, and more. Generalist biomedical artificial intelligence (AI)
  systems that flexibly encode, integrate, and interpret this data at scale can
  potentially enable impactful applications ranging from scientific discovery to
  care delivery. To enable the development of these models, we first curate
  MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses
  14 diverse tasks such as medical question answering, mammography and
  dermatology image interpretation, radiology report generation and
  summarization, and genomic variant calling. We then introduce Med-PaLM
  Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI
  system. Med-PaLM M is a large multimodal generative model that flexibly encodes
  and interprets biomedical data including clinical language, imaging, and
  genomics with the same set of model weights. Med-PaLM M reaches performance
  competitive with or exceeding the state of the art on all MultiMedBench tasks,
  often surpassing specialist models by a wide margin. We also report examples of
  zero-shot generalization to novel medical concepts and tasks, positive transfer
  learning across tasks, and emergent zero-shot medical reasoning. To further
  probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist
  evaluation of model-generated (and human) chest X-ray reports and observe
  encouraging performance across model scales. In a side-by-side ranking on 246
  retrospective chest X-rays, clinicians express a pairwise preference for
  Med-PaLM M reports over those produced by radiologists in up to 40.50% of
  cases, suggesting potential clinical utility. While considerable work is needed
  to validate these models in real-world use cases, our results represent a
  milestone towards the development of generalist biomedical AI systems.
paper_ai_summary: A large multimodal generative model demonstrates competitive performance across diverse biomedical tasks, including zero-shot generalization and transfer learning, suggesting potential clinical utility.
paper_ai_keywords: multimodal biomedical AI generative model medical question answering mammography dermatology radiology report generation genomic variant calling zero-shot generalization transfer learning emergent reasoning chest X-ray reports radiologist evaluation