title: Capabilities of Gemini Models in Medicine
summary: Excellence in a wide variety of medical applications poses considerable
  challenges for AI, requiring advanced reasoning, access to up-to-date medical
  knowledge and understanding of complex multimodal data. Gemini models, with
  strong general capabilities in multimodal and long-context reasoning, offer
  exciting possibilities in medicine. Building on these core strengths of Gemini,
  we introduce Med-Gemini, a family of highly capable multimodal models that are
  specialized in medicine with the ability to seamlessly use web search, and that
  can be efficiently tailored to novel modalities using custom encoders. We
  evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
  (SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
  benchmark where a direct comparison is viable, often by a wide margin. On the
  popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
  SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
  strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
  (health & medicine), Med-Gemini improves over GPT-4V by an average relative
  margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
  capabilities through SoTA performance on a needle-in-a-haystack retrieval task
  from long de-identified health records and medical video question answering,
  surpassing prior bespoke methods using only in-context learning. Finally,
  Med-Gemini's performance suggests real-world utility by surpassing human
  experts on tasks such as medical text summarization, alongside demonstrations
  of promising potential for multimodal medical dialogue, medical research and
  education. Taken together, our results offer compelling evidence for
  Med-Gemini's potential, although further rigorous evaluation will be crucial
  before real-world deployment in this safety-critical domain.
paper_authors._id: 663059ade649907fb14aebee
paper_authors.name: Khaled Saab
paper_authors._id: 663059ade649907fb14aebef
paper_authors.name: Tao Tu
paper_authors._id: 663059ade649907fb14aebf0
paper_authors.name: Wei-Hung Weng
paper_authors._id: 663059ade649907fb14aebf1
paper_authors.name: Ryutaro Tanno
paper_authors._id: 663059ade649907fb14aebf2
paper_authors.name: David Stutz
paper_authors._id: 663059ade649907fb14aebf3
paper_authors.name: Ellery Wulczyn
paper_authors._id: 663059ade649907fb14aebf4
paper_authors.name: Fan Zhang
paper_authors._id: 663059ade649907fb14aebf5
paper_authors.name: Tim Strother
paper_authors._id: 663059ade649907fb14aebf6
paper_authors.name: Chunjong Park
paper_authors._id: 663059ade649907fb14aebf7
paper_authors.name: Elahe Vedadi
paper_authors._id: 663059ade649907fb14aebf8
paper_authors.name: Juanma Zambrano Chaves
paper_authors._id: 663059ade649907fb14aebf9
paper_authors.name: Szu-Yeu Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:10:37.627Z
paper_authors._id: 6626acc7b24310dd5feee335
paper_authors.avatarUrl: /avatars/b2a950ae946887935233d54d9374118d.svg
paper_authors.fullname: Szu-Yeu Hu
paper_authors.type: user
paper_authors.user: sdcjimmy
paper_authors._id: 663059ade649907fb14aebfa
paper_authors.name: Mike Schaekermann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:10:43.861Z
paper_authors._id: 64ebb75f171e6c9862d261c9
paper_authors.avatarUrl: /avatars/b7750f5d6281f906cfa5c717a44e2e13.svg
paper_authors.fullname: Mike Schaekermann
paper_authors.type: user
paper_authors.user: mikeschaekermann
paper_authors._id: 663059ade649907fb14aebfb
paper_authors.name: Aishwarya Kamath
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:10:52.656Z
paper_authors._id: 636996b9d322a76e1ea7e59d
paper_authors.avatarUrl: /avatars/100cee613df9914c414f51be0ca86810.svg
paper_authors.fullname: Aishwarya Kamath
paper_authors.type: user
paper_authors.user: ashkamath
paper_authors._id: 663059ade649907fb14aebfc
paper_authors.name: Yong Cheng
paper_authors._id: 663059ade649907fb14aebfd
paper_authors.name: David G. T. Barrett
paper_authors._id: 663059ade649907fb14aebfe
paper_authors.name: Cathy Cheung
paper_authors._id: 663059ade649907fb14aebff
paper_authors.name: Basil Mustafa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:11:50.424Z
paper_authors._id: 63cfcad6e23b90128c66685c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
paper_authors.fullname: Basil Mustafa
paper_authors.type: user
paper_authors.user: BasilMustafa
paper_authors._id: 663059ade649907fb14aec00
paper_authors.name: Anil Palepu
paper_authors._id: 663059ade649907fb14aec01
paper_authors.name: Daniel McDuff
paper_authors._id: 663059ade649907fb14aec02
paper_authors.name: Le Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:12:39.746Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 663059ade649907fb14aec03
paper_authors.name: Tomer Golany
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:12:46.302Z
paper_authors._id: 64f06d58a5aeea35fcbf4bae
paper_authors.avatarUrl: /avatars/4966ab54ca7074c220b706a0d159c9b7.svg
paper_authors.fullname: Tomer Golany
paper_authors.type: user
paper_authors.user: tomergolany
paper_authors._id: 663059ade649907fb14aec04
paper_authors.name: Luyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:13:33.141Z
paper_authors._id: 65d8fa4e16cebdd689fa5587
paper_authors.avatarUrl: /avatars/9646f22ae16653af47ea38af0839ae7b.svg
paper_authors.fullname: Luyang Liu
paper_authors.type: user
paper_authors.user: luyangl
paper_authors._id: 663059ade649907fb14aec05
paper_authors.name: Jean-baptiste Alayrac
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:13:44.292Z
paper_authors._id: 6253ee39e4e98393660b5c35
paper_authors.avatarUrl: /avatars/9c032f6a0729bfe5c16b3affe190834d.svg
paper_authors.fullname: Jean-Baptiste Alayrac
paper_authors.type: user
paper_authors.user: jalayrac
paper_authors._id: 663059ade649907fb14aec06
paper_authors.name: Neil Houlsby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:13:51.325Z
paper_authors._id: 64a2c7f263f69fb98d3bfdb4
paper_authors.avatarUrl: /avatars/a362a236c0654b7605dcb7673e309335.svg
paper_authors.fullname: Neil Houlsby
paper_authors.type: user
paper_authors.user: neilhoulsby
paper_authors._id: 663059ade649907fb14aec07
paper_authors.name: Nenad Tomasev
paper_authors._id: 663059ade649907fb14aec08
paper_authors.name: Jan Freyberg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:14:02.247Z
paper_authors._id: 629656557d586f66c39206e7
paper_authors.avatarUrl: /avatars/91245f3b6a2eccebef872ed00bdfea73.svg
paper_authors.fullname: Jan Freyberg
paper_authors.type: user
paper_authors.user: jfreyberg
paper_authors._id: 663059ade649907fb14aec09
paper_authors.name: Charles Lau
paper_authors._id: 663059ade649907fb14aec0a
paper_authors.name: Jonas Kemp
paper_authors._id: 663059ade649907fb14aec0b
paper_authors.name: Jeremy Lai
paper_authors._id: 663059ade649907fb14aec0c
paper_authors.name: Shekoofeh Azizi
paper_authors._id: 663059ade649907fb14aec0d
paper_authors.name: Kimberly Kanada
paper_authors._id: 663059ade649907fb14aec0e
paper_authors.name: SiWai Man
paper_authors._id: 663059ade649907fb14aec0f
paper_authors.name: Kavita Kulkarni
paper_authors._id: 663059ade649907fb14aec10
paper_authors.name: Ruoxi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:15:21.516Z
paper_authors._id: 653093ad471261a41241e048
paper_authors.avatarUrl: /avatars/54263ce75b102b81211d27b696920870.svg
paper_authors.fullname: Ruoxi Sun
paper_authors.type: user
paper_authors.user: brycesun
paper_authors._id: 663059ade649907fb14aec11
paper_authors.name: Siamak Shakeri
paper_authors._id: 663059ade649907fb14aec12
paper_authors.name: Luheng He
paper_authors._id: 663059ade649907fb14aec13
paper_authors.name: Ben Caine
paper_authors._id: 663059ade649907fb14aec14
paper_authors.name: Albert Webson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:15:46.876Z
paper_authors._id: 617ada70ff3db6021d069798
paper_authors.avatarUrl: /avatars/f14c59fd56a49dd9a925285a4fb66fff.svg
paper_authors.fullname: Albert Webson
paper_authors.type: user
paper_authors.user: awebson
paper_authors._id: 663059ade649907fb14aec15
paper_authors.name: Natasha Latysheva
paper_authors._id: 663059ade649907fb14aec16
paper_authors.name: Melvin Johnson
paper_authors._id: 663059ade649907fb14aec17
paper_authors.name: Philip Mansfield
paper_authors._id: 663059ade649907fb14aec18
paper_authors.name: Jian Lu
paper_authors._id: 663059ade649907fb14aec19
paper_authors.name: Ehud Rivlin
paper_authors._id: 663059ade649907fb14aec1a
paper_authors.name: Jesper Anderson
paper_authors._id: 663059ade649907fb14aec1b
paper_authors.name: Bradley Green
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:16:36.781Z
paper_authors._id: 63417e33dd8853dc7e33c6a1
paper_authors.avatarUrl: /avatars/3a0bcd81d2dc1b032e445997485ebb49.svg
paper_authors.fullname: bradley green
paper_authors.type: user
paper_authors.user: cybercow2077
paper_authors._id: 663059ade649907fb14aec1c
paper_authors.name: Renee Wong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:16:45.053Z
paper_authors._id: 6397fb67a51b5dbfc93ca0d1
paper_authors.avatarUrl: /avatars/b254d4df8e4158e2dbd047ee077440e3.svg
paper_authors.fullname: Wong
paper_authors.type: user
paper_authors.user: reneewong60
paper_authors._id: 663059ade649907fb14aec1d
paper_authors.name: Jonathan Krause
paper_authors._id: 663059ade649907fb14aec1e
paper_authors.name: Jonathon Shlens
paper_authors._id: 663059ade649907fb14aec1f
paper_authors.name: Ewa Dominowska
paper_authors._id: 663059ade649907fb14aec20
paper_authors.name: S. M. Ali Eslami
paper_authors._id: 663059ade649907fb14aec21
paper_authors.name: Claire Cui
paper_authors._id: 663059ade649907fb14aec22
paper_authors.name: Oriol Vinyals
paper_authors._id: 663059ade649907fb14aec23
paper_authors.name: Koray Kavukcuoglu
paper_authors._id: 663059ade649907fb14aec24
paper_authors.name: James Manyika
paper_authors._id: 663059ade649907fb14aec25
paper_authors.name: Jeff Dean
paper_authors._id: 663059ade649907fb14aec26
paper_authors.name: Demis Hassabis
paper_authors._id: 663059ade649907fb14aec27
paper_authors.name: Yossi Matias
paper_authors._id: 663059ade649907fb14aec28
paper_authors.name: Dale Webster
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:18:04.482Z
paper_authors._id: 63124715bbaa385279cebcec
paper_authors.avatarUrl: /avatars/120869c6dd8ed8cfdd4facb688a8f0af.svg
paper_authors.fullname: Dale Webster
paper_authors.type: user
paper_authors.user: allany
paper_authors._id: 663059ade649907fb14aec29
paper_authors.name: Joelle Barral
paper_authors._id: 663059ade649907fb14aec2a
paper_authors.name: Greg Corrado
paper_authors._id: 663059ade649907fb14aec2b
paper_authors.name: Christopher Semturs
paper_authors._id: 663059ade649907fb14aec2c
paper_authors.name: S. Sara Mahdavi
paper_authors._id: 663059ade649907fb14aec2d
paper_authors.name: Juraj Gottweis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:17:30.136Z
paper_authors._id: 6381e100cae34727b9ca33ce
paper_authors.avatarUrl: /avatars/f4e047e82e41ec507c6f51aef7560d87.svg
paper_authors.fullname: Juraj Gottweis
paper_authors.type: user
paper_authors.user: juro
paper_authors._id: 663059ade649907fb14aec2e
paper_authors.name: Alan Karthikesalingam
paper_authors._id: 663059ade649907fb14aec2f
paper_authors.name: Vivek Natarajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:17:17.630Z
paper_authors._id: 617a6af2941993035fbaf29c
paper_authors.avatarUrl: /avatars/a389671494782f06ff8502c934090cc0.svg
paper_authors.fullname: Vivek Natarajan
paper_authors.type: user
paper_authors.user: vivnat
paper_title: Capabilities of Gemini Models in Medicine
paper_summary: Excellence in a wide variety of medical applications poses considerable
  challenges for AI, requiring advanced reasoning, access to up-to-date medical
  knowledge and understanding of complex multimodal data. Gemini models, with
  strong general capabilities in multimodal and long-context reasoning, offer
  exciting possibilities in medicine. Building on these core strengths of Gemini,
  we introduce Med-Gemini, a family of highly capable multimodal models that are
  specialized in medicine with the ability to seamlessly use web search, and that
  can be efficiently tailored to novel modalities using custom encoders. We
  evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
  (SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
  benchmark where a direct comparison is viable, often by a wide margin. On the
  popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
  SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
  strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
  (health & medicine), Med-Gemini improves over GPT-4V by an average relative
  margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
  capabilities through SoTA performance on a needle-in-a-haystack retrieval task
  from long de-identified health records and medical video question answering,
  surpassing prior bespoke methods using only in-context learning. Finally,
  Med-Gemini's performance suggests real-world utility by surpassing human
  experts on tasks such as medical text summarization, alongside demonstrations
  of promising potential for multimodal medical dialogue, medical research and
  education. Taken together, our results offer compelling evidence for
  Med-Gemini's potential, although further rigorous evaluation will be crucial
  before real-world deployment in this safety-critical domain.
paper_ai_summary: Med-Gemini, a family of specialized multimodal AI models, achieves state-of-the-art performance across various medical benchmarks, surpassing GPT-4 and demonstrating superior long-context and uncertainty-guided search capabilities.
paper_ai_keywords: multimodal diamond models long-context reasoning Med-Gemini web search custom encoders state-of-the-art SoTA MedQA uncertainty-guided search NEJM Image Challenges MMMU GPT-4 needle-in-a-haystack retrieval medical video question answering in-context learning medical text summarization multimodal medical dialogue medical research medical education