title: Exploring the Boundaries of GPT-4 in Radiology
summary: The recent success of general-domain large language models (LLMs) has
  significantly changed the natural language processing paradigm towards a
  unified foundation model across domains and applications. In this paper, we
  focus on assessing the performance of GPT-4, the most capable LLM so far, on
  the text-based applications for radiology reports, comparing against
  state-of-the-art (SOTA) radiology-specific models. Exploring various prompting
  strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and
  we found GPT-4 either outperforms or is on par with current SOTA radiology
  models. With zero-shot prompting, GPT-4 already obtains substantial gains
  (approx 10% absolute improvement) over radiology models in temporal sentence
  similarity classification (accuracy) and natural language inference (F_1).
  For tasks that require learning dataset-specific style or schema (e.g. findings
  summarisation), GPT-4 improves with example-based prompting and matches
  supervised SOTA. Our extensive error analysis with a board-certified
  radiologist shows GPT-4 has a sufficient level of radiology knowledge with only
  occasional errors in complex context that require nuanced domain knowledge. For
  findings summarisation, GPT-4 outputs are found to be overall comparable with
  existing manually-written impressions.
paper_authors._id: 65372de5e5ad8b46d53949b9
paper_authors.name: Qianchu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:41:09.123Z
paper_authors._id: 617e7729129c9e67703ffe61
paper_authors.avatarUrl: /avatars/f47ee9f2f0e2b1075bebf3682ee2f817.svg
paper_authors.fullname: qianchu liu
paper_authors.type: user
paper_authors.user: qianchu
paper_authors._id: 65372de5e5ad8b46d53949ba
paper_authors.name: Stephanie Hyland
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T11:47:08.860Z
paper_authors._id: 627554587aef4f8142f18bf9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/627554587aef4f8142f18bf9/J8xOcG42lSCDDiewpzYob.jpeg
paper_authors.fullname: Stephanie
paper_authors.type: user
paper_authors.user: sthyland
paper_authors._id: 65372de5e5ad8b46d53949bb
paper_authors.name: Shruthi Bannur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:41:28.894Z
paper_authors._id: 6274c513d400831ee0b8267c
paper_authors.avatarUrl: /avatars/b4d183b7ab66cb7b16d6802c4850d8e8.svg
paper_authors.fullname: Shruthi Bannur
paper_authors.type: user
paper_authors.user: shruthib
paper_authors._id: 65372de5e5ad8b46d53949bc
paper_authors.name: Kenza Bouzid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:41:36.616Z
paper_authors._id: 6511a4fbea5ccf1113bb545e
paper_authors.avatarUrl: /avatars/8551b109249dd9c303ab17813297e09f.svg
paper_authors.fullname: Kenza Bouzid
paper_authors.type: user
paper_authors.user: kenza-bouzid
paper_authors._id: 65372de5e5ad8b46d53949bd
paper_authors.name: Daniel C. Castro
paper_authors._id: 65372de5e5ad8b46d53949be
paper_authors.name: Maria Teodora Wetscherek
paper_authors._id: 65372de5e5ad8b46d53949bf
paper_authors.name: Robert Tinn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:42:20.734Z
paper_authors._id: 5f807ea93e94f16a85448947
paper_authors.avatarUrl: /avatars/57f0d40178614e690864914015070fe0.svg
paper_authors.fullname: robert tinn
paper_authors.type: user
paper_authors.user: roberttinn
paper_authors._id: 65372de5e5ad8b46d53949c0
paper_authors.name: Harshita Sharma
paper_authors._id: 65372de5e5ad8b46d53949c1
paper_authors.name: Fernando Pérez-García
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:42:55.329Z
paper_authors._id: 6278fbec17c069c72a11f35d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669369101851-6278fbec17c069c72a11f35d.jpeg
paper_authors.fullname: Fernando Pérez-García
paper_authors.type: user
paper_authors.user: fepegar
paper_authors._id: 65372de5e5ad8b46d53949c2
paper_authors.name: Anton Schwaighofer
paper_authors._id: 65372de5e5ad8b46d53949c3
paper_authors.name: Pranav Rajpurkar
paper_authors._id: 65372de5e5ad8b46d53949c4
paper_authors.name: Sameer Tajdin Khanna
paper_authors._id: 65372de5e5ad8b46d53949c5
paper_authors.name: Hoifung Poon
paper_authors._id: 65372de5e5ad8b46d53949c6
paper_authors.name: Naoto Usuyama
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T07:29:19.653Z
paper_authors._id: 5f243561e923d665e616263f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
paper_authors.fullname: Naoto Usuyama
paper_authors.type: user
paper_authors.user: naotous
paper_authors._id: 65372de5e5ad8b46d53949c7
paper_authors.name: Anja Thieme
paper_authors._id: 65372de5e5ad8b46d53949c8
paper_authors.name: Aditya V. Nori
paper_authors._id: 65372de5e5ad8b46d53949c9
paper_authors.name: Matthew P. Lungren
paper_authors._id: 65372de5e5ad8b46d53949ca
paper_authors.name: Ozan Oktay
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:43:43.615Z
paper_authors._id: 6274dd3eb43ccfeef481ce29
paper_authors.avatarUrl: /avatars/9f5d2a25a74af3cb0a227f946041206d.svg
paper_authors.fullname: Ozan Oktay
paper_authors.type: user
paper_authors.user: ozanoktay
paper_authors._id: 65372de5e5ad8b46d53949cb
paper_authors.name: Javier Alvarez-Valle
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-24T02:37:25.545Z
paper_authors._id: 6278d875bba7c62cbaaf7693
paper_authors.avatarUrl: /avatars/8d93e7c33acdfac853317d8fb5b0d485.svg
paper_authors.fullname: Javier Alvarez Valle
paper_authors.type: user
paper_authors.user: jaalvare
paper_title: Exploring the Boundaries of GPT-4 in Radiology
paper_summary: The recent success of general-domain large language models (LLMs) has
  significantly changed the natural language processing paradigm towards a
  unified foundation model across domains and applications. In this paper, we
  focus on assessing the performance of GPT-4, the most capable LLM so far, on
  the text-based applications for radiology reports, comparing against
  state-of-the-art (SOTA) radiology-specific models. Exploring various prompting
  strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and
  we found GPT-4 either outperforms or is on par with current SOTA radiology
  models. With zero-shot prompting, GPT-4 already obtains substantial gains
  (approx 10% absolute improvement) over radiology models in temporal sentence
  similarity classification (accuracy) and natural language inference (F_1).
  For tasks that require learning dataset-specific style or schema (e.g. findings
  summarisation), GPT-4 improves with example-based prompting and matches
  supervised SOTA. Our extensive error analysis with a board-certified
  radiologist shows GPT-4 has a sufficient level of radiology knowledge with only
  occasional errors in complex context that require nuanced domain knowledge. For
  findings summarisation, GPT-4 outputs are found to be overall comparable with
  existing manually-written impressions.
paper_ai_keywords: large language models (LLMs) unified foundation model GPT-4 state-of-the-art (SOTA) radiology reports prompting strategies radiology-specific models temporal sentence similarity classification natural language inference ($F_1$) findings summarisation zero-shot prompting example-based prompting error analysis board-certified radiologist radiology knowledge