CXReasonBench: A Benchmark for Evaluating Structured Diagnostic
  Reasoning in Chest X-rays
Recent progress in Large Vision-Language Models (LVLMs) has enabled promising
applications in medical tasks, such as report generation and visual question
answering. However, existing benchmarks focus mainly on the final diagnostic
answer, offering limited insight into whether models engage in clinically
meaningful reasoning. To address this, we present CheXStruct and CXReasonBench,
a structured pipeline and benchmark built on the publicly available
MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of
intermediate reasoning steps directly from chest X-rays, such as segmenting
anatomical regions, deriving anatomical landmarks and diagnostic measurements,
computing diagnostic indices, and applying clinical thresholds. CXReasonBench
leverages this pipeline to evaluate whether models can perform clinically valid
reasoning steps and to what extent they can learn from structured guidance,
enabling fine-grained and transparent assessment of diagnostic reasoning. The
benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases,
each paired with up to 4 visual inputs, and supports multi-path, multi-stage
evaluation including visual grounding via anatomical region selection and
diagnostic measurements. Even the strongest of 10 evaluated LVLMs struggle with
structured reasoning and generalization, often failing to link abstract
knowledge with anatomically grounded visual interpretation. The code is
available at https://github.com/ttumyche/CXReasonBench
68351b91037632c04211688d
Hyungyung Lee
claimed_verified
2025-05-29T07:44:18.221Z
6837b9a63ed37b18326c7fff
/avatars/7aaf5e1ff783aad45946c937ac36bdd8.svg
Hyungyung Lee
user
ttumyche
68351b91037632c04211688e
Geon Choi
claimed_verified
2025-07-16T08:16:19.535Z
66714579562ee91c16be977c
/avatars/fd1a386c1e06fcbc9c06017d18e4fe5f.svg
Geon Choi
user
checkone
68351b91037632c04211688f
Jung-Oh Lee
68351b91037632c042116890
Hangyul Yoon
68351b91037632c042116891
Hyuk Gi Hong
68351b91037632c042116892
Edward Choi
CXReasonBench: A Benchmark for Evaluating Structured Diagnostic
  Reasoning in Chest X-rays
Recent progress in Large Vision-Language Models (LVLMs) has enabled promising
applications in medical tasks, such as report generation and visual question
answering. However, existing benchmarks focus mainly on the final diagnostic
answer, offering limited insight into whether models engage in clinically
meaningful reasoning. To address this, we present CheXStruct and CXReasonBench,
a structured pipeline and benchmark built on the publicly available
MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of
intermediate reasoning steps directly from chest X-rays, such as segmenting
anatomical regions, deriving anatomical landmarks and diagnostic measurements,
computing diagnostic indices, and applying clinical thresholds. CXReasonBench
leverages this pipeline to evaluate whether models can perform clinically valid
reasoning steps and to what extent they can learn from structured guidance,
enabling fine-grained and transparent assessment of diagnostic reasoning. The
benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases,
each paired with up to 4 visual inputs, and supports multi-path, multi-stage
evaluation including visual grounding via anatomical region selection and
diagnostic measurements. Even the strongest of 10 evaluated LVLMs struggle with
structured reasoning and generalization, often failing to link abstract
knowledge with anatomically grounded visual interpretation. The code is
available at https://github.com/ttumyche/CXReasonBench
CheXStruct and CXReasonBench evaluate Large Vision-Language Models in clinical diagnosis by assessing structured reasoning, visual grounding, and generalization using the MIMIC-CXR-JPG dataset.
Large Vision-Language Models LVLMs CheXStruct CXReasonBench MIMIC-CXR-JPG structured reasoning visual question answering medical tasks report generation anatomical regions diagnostic measurements diagnostic indices clinical thresholds visual grounding diagnostic reasoning multi-path multi-stage evaluation