Towards Generalist Biomedical AI
Medicine is inherently multimodal, with rich data modalities spanning text,
imaging, genomics, and more. Generalist biomedical artificial intelligence (AI)
systems that flexibly encode, integrate, and interpret this data at scale can
potentially enable impactful applications ranging from scientific discovery to
care delivery. To enable the development of these models, we first curate
MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses
14 diverse tasks such as medical question answering, mammography and
dermatology image interpretation, radiology report generation and
summarization, and genomic variant calling. We then introduce Med-PaLM
Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI
system. Med-PaLM M is a large multimodal generative model that flexibly encodes
and interprets biomedical data including clinical language, imaging, and
genomics with the same set of model weights. Med-PaLM M reaches performance
competitive with or exceeding the state of the art on all MultiMedBench tasks,
often surpassing specialist models by a wide margin. We also report examples of
zero-shot generalization to novel medical concepts and tasks, positive transfer
learning across tasks, and emergent zero-shot medical reasoning. To further
probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist
evaluation of model-generated (and human) chest X-ray reports and observe
encouraging performance across model scales. In a side-by-side ranking on 246
retrospective chest X-rays, clinicians express a pairwise preference for
Med-PaLM M reports over those produced by radiologists in up to 40.50% of
cases, suggesting potential clinical utility. While considerable work is needed
to validate these models in real-world use cases, our results represent a
milestone towards the development of generalist biomedical AI systems.
64c21c46245c55a21c79e14f
Tao Tu
64c21c46245c55a21c79e150
Shekoofeh Azizi
64c21c46245c55a21c79e151
Danny Driess
64c21c46245c55a21c79e152
Mike Schaekermann
64c21c46245c55a21c79e153
Mohamed Amin
64c21c46245c55a21c79e154
Pi-Chuan Chang
64c21c46245c55a21c79e155
Andrew Carroll
64c21c46245c55a21c79e156
Chuck Lau
64c21c46245c55a21c79e157
Ryutaro Tanno
64c21c46245c55a21c79e158
Ira Ktena
64c21c46245c55a21c79e159
Basil Mustafa
admin_assigned
2023-07-27T11:33:25.021Z
63cfcad6e23b90128c66685c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
Basil Mustafa
user
BasilMustafa
64c21c46245c55a21c79e15a
Aakanksha Chowdhery
64c21c46245c55a21c79e15b
Yun Liu
64c21c46245c55a21c79e15c
Simon Kornblith
admin_assigned
2023-07-27T11:35:14.471Z
6443ffc2af034cdfd69c5c00
/avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
Simon Kornblith
user
simonsterk
64c21c46245c55a21c79e15d
David Fleet
64c21c46245c55a21c79e15e
Philip Mansfield
64c21c46245c55a21c79e15f
Sushant Prakash
64c21c46245c55a21c79e160
Renee Wong
admin_assigned
2023-07-27T11:35:48.270Z
6397fb67a51b5dbfc93ca0d1
/avatars/b254d4df8e4158e2dbd047ee077440e3.svg
Wong
user
reneewong60
64c21c46245c55a21c79e161
Sunny Virmani
64c21c46245c55a21c79e162
Christopher Semturs
64c21c46245c55a21c79e163
S Sara Mahdavi
64c21c46245c55a21c79e164
Bradley Green
64c21c46245c55a21c79e165
Ewa Dominowska
64c21c46245c55a21c79e166
Blaise Aguera y Arcas
64c21c46245c55a21c79e167
Joelle Barral
64c21c46245c55a21c79e168
Dale Webster
64c21c46245c55a21c79e169
Greg S. Corrado
64c21c46245c55a21c79e16a
Yossi Matias
64c21c46245c55a21c79e16b
Karan Singhal
64c21c46245c55a21c79e16c
Pete Florence
64c21c46245c55a21c79e16d
Alan Karthikesalingam
64c21c46245c55a21c79e16e
Vivek Natarajan
admin_assigned
2023-07-27T11:37:42.600Z
617a6af2941993035fbaf29c
/avatars/a389671494782f06ff8502c934090cc0.svg
Vivek Natarajan
user
vivnat
Towards Generalist Biomedical AI
Medicine is inherently multimodal, with rich data modalities spanning text,
imaging, genomics, and more. Generalist biomedical artificial intelligence (AI)
systems that flexibly encode, integrate, and interpret this data at scale can
potentially enable impactful applications ranging from scientific discovery to
care delivery. To enable the development of these models, we first curate
MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses
14 diverse tasks such as medical question answering, mammography and
dermatology image interpretation, radiology report generation and
summarization, and genomic variant calling. We then introduce Med-PaLM
Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI
system. Med-PaLM M is a large multimodal generative model that flexibly encodes
and interprets biomedical data including clinical language, imaging, and
genomics with the same set of model weights. Med-PaLM M reaches performance
competitive with or exceeding the state of the art on all MultiMedBench tasks,
often surpassing specialist models by a wide margin. We also report examples of
zero-shot generalization to novel medical concepts and tasks, positive transfer
learning across tasks, and emergent zero-shot medical reasoning. To further
probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist
evaluation of model-generated (and human) chest X-ray reports and observe
encouraging performance across model scales. In a side-by-side ranking on 246
retrospective chest X-rays, clinicians express a pairwise preference for
Med-PaLM M reports over those produced by radiologists in up to 40.50% of
cases, suggesting potential clinical utility. While considerable work is needed
to validate these models in real-world use cases, our results represent a
milestone towards the development of generalist biomedical AI systems.
A large multimodal generative model demonstrates competitive performance across diverse biomedical tasks, including zero-shot generalization and transfer learning, suggesting potential clinical utility.
multimodal biomedical AI generative model medical question answering mammography dermatology radiology report generation genomic variant calling zero-shot generalization transfer learning emergent reasoning chest X-ray reports radiologist evaluation