title: CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation
summary: Chest X-rays (CXRs) are the most frequently performed imaging test in
  clinical practice. Recent advances in the development of vision-language
  foundation models (FMs) give rise to the possibility of performing automated
  CXR interpretation, which can assist physicians with clinical decision-making
  and improve patient outcomes. However, developing FMs that can accurately
  interpret CXRs is challenging due to the (1) limited availability of
  large-scale vision-language datasets in the medical image domain, (2) lack of
  vision and language encoders that can capture the complexities of medical data,
  and (3) absence of evaluation frameworks for benchmarking the abilities of FMs
  on CXR interpretation. In this work, we address these challenges by first
  introducing CheXinstruct - a large-scale instruction-tuning dataset
  curated from 28 publicly-available datasets. We then present CheXagent -
  an instruction-tuned FM capable of analyzing and summarizing CXRs. To build
  CheXagent, we design a clinical large language model (LLM) for parsing
  radiology reports, a vision encoder for representing CXR images, and a network
  to bridge the vision and language modalities. Finally, we introduce
  CheXbench - a novel benchmark designed to systematically evaluate FMs
  across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative
  evaluations and qualitative reviews with five expert radiologists demonstrate
  that CheXagent outperforms previously-developed general- and medical-domain FMs
  on CheXbench tasks. Furthermore, in an effort to improve model transparency, we
  perform a fairness evaluation across factors of sex, race and age to highlight
  potential performance disparities. Our project is at
  https://stanford-aimi.github.io/chexagent.html.
paper_authors._id: 65af4222755c534def07a08b
paper_authors.name: Zhihong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:15:17.882Z
paper_authors._id: 62e68ecff18bd3c3b532ddfd
paper_authors.avatarUrl: /avatars/60ddbde83ab4be0277272a94e2e72e3a.svg
paper_authors.fullname: Zhihong Chen
paper_authors.type: user
paper_authors.user: zhjohnchan
paper_authors._id: 65af4222755c534def07a08c
paper_authors.name: Maya Varma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:16:02.233Z
paper_authors._id: 6180c6e5000231f499c547c7
paper_authors.avatarUrl: /avatars/0255b430be744a8da236860e2a00307e.svg
paper_authors.fullname: Maya Varma
paper_authors.type: user
paper_authors.user: mvarma
paper_authors._id: 65af4222755c534def07a08d
paper_authors.name: Jean-Benoit Delbrouck
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:09:47.549Z
paper_authors._id: 62716952bcef985363db8485
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62716952bcef985363db8485/zJPPo5xlwZRJdEuwYsYKp.jpeg
paper_authors.fullname: JB D.
paper_authors.type: user
paper_authors.user: IAMJB
paper_authors._id: 65af4222755c534def07a08e
paper_authors.name: Magdalini Paschali
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T08:09:11.441Z
paper_authors._id: 65b00ea3399c0430e82ae284
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4Iovd06vaVhsZytvG066d.jpeg
paper_authors.fullname: Magda Paschali
paper_authors.type: user
paper_authors.user: magda-paschali
paper_authors._id: 65af4222755c534def07a08f
paper_authors.name: Louis Blankemeier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:16:22.179Z
paper_authors._id: 6377044ea3b787faca443cc0
paper_authors.avatarUrl: /avatars/5cd6cc70ad597dd83776018a452947d0.svg
paper_authors.fullname: Louis Blankemeier
paper_authors.type: user
paper_authors.user: louisblankemeier
paper_authors._id: 65af4222755c534def07a090
paper_authors.name: Dave Van Veen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:16:29.120Z
paper_authors._id: 64e36dd9618cd90997e048f2
paper_authors.avatarUrl: /avatars/2a239bed193982596114d52afea1bb97.svg
paper_authors.fullname: Dave Van Veen
paper_authors.type: user
paper_authors.user: davevanveen
paper_authors._id: 65af4222755c534def07a091
paper_authors.name: Jeya Maria Jose Valanarasu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T08:09:14.459Z
paper_authors._id: 654eb5cbc67f60a3685e49e6
paper_authors.avatarUrl: /avatars/047b2bfc768b1309b8e0363e5bcfbfbb.svg
paper_authors.fullname: Jeya Maria Jose Valanarasu
paper_authors.type: user
paper_authors.user: jmjose
paper_authors._id: 65af4222755c534def07a092
paper_authors.name: Alaa Youssef
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:16:49.347Z
paper_authors._id: 6526f2d0ad78d4a07a84cf24
paper_authors.avatarUrl: /avatars/7e477a087098477b0d10517eab7f5a4e.svg
paper_authors.fullname: Alaa Youssef
paper_authors.type: user
paper_authors.user: Alaajoer
paper_authors._id: 65af4222755c534def07a093
paper_authors.name: Joseph Paul Cohen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:16:57.580Z
paper_authors._id: 63a9d04f3453852ef53e1830
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672073530253-63a9d04f3453852ef53e1830.png
paper_authors.fullname: Joseph Paul Cohen
paper_authors.type: user
paper_authors.user: ieee8023
paper_authors._id: 65af4222755c534def07a094
paper_authors.name: Eduardo Pontes Reis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:17:04.835Z
paper_authors._id: 60bc341d85b60a3459be9cac
paper_authors.avatarUrl: /avatars/e0d3d2907504176451a0d22885eef997.svg
paper_authors.fullname: Eduardo Pontes Reis
paper_authors.type: user
paper_authors.user: edureisMD
paper_authors._id: 65af4222755c534def07a095
paper_authors.name: Emily B. Tsai
paper_authors._id: 65af4222755c534def07a096
paper_authors.name: Andrew Johnston
paper_authors._id: 65af4222755c534def07a097
paper_authors.name: Cameron Olsen
paper_authors._id: 65af4222755c534def07a098
paper_authors.name: Tanishq Mathew Abraham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:26:44.198Z
paper_authors._id: 6057b823861b9d53d9c4b8df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg
paper_authors.fullname: Tanishq Abraham
paper_authors.type: user
paper_authors.user: tmabraham
paper_authors._id: 65af4222755c534def07a099
paper_authors.name: Sergios Gatidis
paper_authors._id: 65af4222755c534def07a09a
paper_authors.name: Akshay S. Chaudhari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T07:18:37.044Z
paper_authors._id: 6236533b76c8a780323af640
paper_authors.avatarUrl: /avatars/18078ad26aa44312a4927160216e5943.svg
paper_authors.fullname: Akshay Chaudhari
paper_authors.type: user
paper_authors.user: akshaysc
paper_authors._id: 65af4222755c534def07a09b
paper_authors.name: Curtis Langlotz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T07:18:38.658Z
paper_authors._id: 62867edd1d99648808888d7b
paper_authors.avatarUrl: /avatars/22d764782c5169634deec6f43ece7500.svg
paper_authors.fullname: Curt Langlotz
paper_authors.type: user
paper_authors.user: cplanglotz
paper_title: CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation
paper_summary: Chest X-rays (CXRs) are the most frequently performed imaging test in
  clinical practice. Recent advances in the development of vision-language
  foundation models (FMs) give rise to the possibility of performing automated
  CXR interpretation, which can assist physicians with clinical decision-making
  and improve patient outcomes. However, developing FMs that can accurately
  interpret CXRs is challenging due to the (1) limited availability of
  large-scale vision-language datasets in the medical image domain, (2) lack of
  vision and language encoders that can capture the complexities of medical data,
  and (3) absence of evaluation frameworks for benchmarking the abilities of FMs
  on CXR interpretation. In this work, we address these challenges by first
  introducing CheXinstruct - a large-scale instruction-tuning dataset
  curated from 28 publicly-available datasets. We then present CheXagent -
  an instruction-tuned FM capable of analyzing and summarizing CXRs. To build
  CheXagent, we design a clinical large language model (LLM) for parsing
  radiology reports, a vision encoder for representing CXR images, and a network
  to bridge the vision and language modalities. Finally, we introduce
  CheXbench - a novel benchmark designed to systematically evaluate FMs
  across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative
  evaluations and qualitative reviews with five expert radiologists demonstrate
  that CheXagent outperforms previously-developed general- and medical-domain FMs
  on CheXbench tasks. Furthermore, in an effort to improve model transparency, we
  perform a fairness evaluation across factors of sex, race and age to highlight
  potential performance disparities. Our project is at
  https://stanford-aimi.github.io/chexagent.html.
paper_ai_keywords: vision-language foundation models (FMs) automated CXR interpretation clinical decision-making patient outcomes instruction-tuning dataset CheXinstruct CheXagent clinical large language model (LLM) vision encoder network to model vision and language modalities CheXbench clinically-relevant CXR interpretation tasks fairness evaluation