CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of
  Synthetic Chest Radiographs
We introduce CheXGenBench, a rigorous and multifaceted evaluation framework
for synthetic chest radiograph generation that simultaneously assesses
fidelity, privacy risks, and clinical utility across state-of-the-art
text-to-image generative models. Despite rapid advancements in generative AI
for real-world imagery, medical domain evaluations have been hindered by
methodological inconsistencies, outdated architectural comparisons, and
disconnected assessment criteria that rarely address the practical clinical
value of synthetic samples. CheXGenBench overcomes these limitations through
standardised data partitioning and a unified evaluation protocol comprising
over 20 quantitative metrics that systematically analyse generation quality,
potential privacy vulnerabilities, and downstream clinical applicability across
11 leading text-to-image architectures. Our results reveal critical
inefficiencies in the existing evaluation protocols, particularly in assessing
generative fidelity, leading to inconsistent and uninformative comparisons. Our
framework establishes a standardised benchmark for the medical AI community,
enabling objective and reproducible comparisons while facilitating seamless
integration of both existing and future generative models. Additionally, we
release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K
radiographs generated by the top-performing model (Sana 0.6B) in our benchmark
to support further research in this critical domain. Through CheXGenBench, we
establish a new state-of-the-art and release our framework, models, and
SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/
6827c35bbc09f10228e2315e
Raman Dutt
admin_assigned
2025-05-19T14:22:37.406Z
655cc4fd905b1390a44b19b8
/avatars/bbf5144468ba96317b6dc10e2d465d05.svg
Raman Dutt
user
raman07
6827c35bbc09f10228e2315f
Pedro Sanchez
admin_assigned
2025-05-19T14:22:47.584Z
633bf51720ceb9fcae5abda3
https://cdn-avatars.huggingface.co/v1/production/uploads/633bf51720ceb9fcae5abda3/2GlqQb-EPv_MiR3xraBen.jpeg
Pedro Sanchez
user
PedroSanchez
6827c35bbc09f10228e23160
Yongchen Yao
6827c35bbc09f10228e23161
Steven McDonagh
admin_assigned
2025-05-19T14:23:02.159Z
67373673d50f5c2e018b4ed6
/avatars/5ebf329d917eac0be02dc5e7fb16a7b4.svg
Steven McDonagh
user
smcdonagh
6827c35bbc09f10228e23162
Sotirios A. Tsaftaris
6827c35bbc09f10228e23163
Timothy Hospedales
CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of
  Synthetic Chest Radiographs
We introduce CheXGenBench, a rigorous and multifaceted evaluation framework
for synthetic chest radiograph generation that simultaneously assesses
fidelity, privacy risks, and clinical utility across state-of-the-art
text-to-image generative models. Despite rapid advancements in generative AI
for real-world imagery, medical domain evaluations have been hindered by
methodological inconsistencies, outdated architectural comparisons, and
disconnected assessment criteria that rarely address the practical clinical
value of synthetic samples. CheXGenBench overcomes these limitations through
standardised data partitioning and a unified evaluation protocol comprising
over 20 quantitative metrics that systematically analyse generation quality,
potential privacy vulnerabilities, and downstream clinical applicability across
11 leading text-to-image architectures. Our results reveal critical
inefficiencies in the existing evaluation protocols, particularly in assessing
generative fidelity, leading to inconsistent and uninformative comparisons. Our
framework establishes a standardised benchmark for the medical AI community,
enabling objective and reproducible comparisons while facilitating seamless
integration of both existing and future generative models. Additionally, we
release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K
radiographs generated by the top-performing model (Sana 0.6B) in our benchmark
to support further research in this critical domain. Through CheXGenBench, we
establish a new state-of-the-art and release our framework, models, and
SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/
CheXGenBench is a comprehensive evaluation framework for synthetic chest radiographs that assesses fidelity, privacy, and clinical utility of text-to-image generative models using standardized metrics and a unified protocol.
text-to-image generative models CheXGenBench synthetic chest radiograph evaluation framework generation quality privacy vulnerabilities clinical applicability standardised data partitioning evaluation protocol quantitative metrics Sana 0.6B SynthCheX-75K