title: CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of
    Synthetic Chest Radiographs
summary: We introduce CheXGenBench, a rigorous and multifaceted evaluation framework
  for synthetic chest radiograph generation that simultaneously assesses
  fidelity, privacy risks, and clinical utility across state-of-the-art
  text-to-image generative models. Despite rapid advancements in generative AI
  for real-world imagery, medical domain evaluations have been hindered by
  methodological inconsistencies, outdated architectural comparisons, and
  disconnected assessment criteria that rarely address the practical clinical
  value of synthetic samples. CheXGenBench overcomes these limitations through
  standardised data partitioning and a unified evaluation protocol comprising
  over 20 quantitative metrics that systematically analyse generation quality,
  potential privacy vulnerabilities, and downstream clinical applicability across
  11 leading text-to-image architectures. Our results reveal critical
  inefficiencies in the existing evaluation protocols, particularly in assessing
  generative fidelity, leading to inconsistent and uninformative comparisons. Our
  framework establishes a standardised benchmark for the medical AI community,
  enabling objective and reproducible comparisons while facilitating seamless
  integration of both existing and future generative models. Additionally, we
  release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K
  radiographs generated by the top-performing model (Sana 0.6B) in our benchmark
  to support further research in this critical domain. Through CheXGenBench, we
  establish a new state-of-the-art and release our framework, models, and
  SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/
paper_authors._id: 6827c35bbc09f10228e2315e
paper_authors.name: Raman Dutt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T14:22:37.406Z
paper_authors._id: 655cc4fd905b1390a44b19b8
paper_authors.avatarUrl: /avatars/bbf5144468ba96317b6dc10e2d465d05.svg
paper_authors.fullname: Raman Dutt
paper_authors.type: user
paper_authors.user: raman07
paper_authors._id: 6827c35bbc09f10228e2315f
paper_authors.name: Pedro Sanchez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T14:22:47.584Z
paper_authors._id: 633bf51720ceb9fcae5abda3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633bf51720ceb9fcae5abda3/2GlqQb-EPv_MiR3xraBen.jpeg
paper_authors.fullname: Pedro Sanchez
paper_authors.type: user
paper_authors.user: PedroSanchez
paper_authors._id: 6827c35bbc09f10228e23160
paper_authors.name: Yongchen Yao
paper_authors._id: 6827c35bbc09f10228e23161
paper_authors.name: Steven McDonagh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T14:23:02.159Z
paper_authors._id: 67373673d50f5c2e018b4ed6
paper_authors.avatarUrl: /avatars/5ebf329d917eac0be02dc5e7fb16a7b4.svg
paper_authors.fullname: Steven McDonagh
paper_authors.type: user
paper_authors.user: smcdonagh
paper_authors._id: 6827c35bbc09f10228e23162
paper_authors.name: Sotirios A. Tsaftaris
paper_authors._id: 6827c35bbc09f10228e23163
paper_authors.name: Timothy Hospedales
paper_title: CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of
    Synthetic Chest Radiographs
paper_summary: We introduce CheXGenBench, a rigorous and multifaceted evaluation framework
  for synthetic chest radiograph generation that simultaneously assesses
  fidelity, privacy risks, and clinical utility across state-of-the-art
  text-to-image generative models. Despite rapid advancements in generative AI
  for real-world imagery, medical domain evaluations have been hindered by
  methodological inconsistencies, outdated architectural comparisons, and
  disconnected assessment criteria that rarely address the practical clinical
  value of synthetic samples. CheXGenBench overcomes these limitations through
  standardised data partitioning and a unified evaluation protocol comprising
  over 20 quantitative metrics that systematically analyse generation quality,
  potential privacy vulnerabilities, and downstream clinical applicability across
  11 leading text-to-image architectures. Our results reveal critical
  inefficiencies in the existing evaluation protocols, particularly in assessing
  generative fidelity, leading to inconsistent and uninformative comparisons. Our
  framework establishes a standardised benchmark for the medical AI community,
  enabling objective and reproducible comparisons while facilitating seamless
  integration of both existing and future generative models. Additionally, we
  release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K
  radiographs generated by the top-performing model (Sana 0.6B) in our benchmark
  to support further research in this critical domain. Through CheXGenBench, we
  establish a new state-of-the-art and release our framework, models, and
  SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/
paper_ai_summary: CheXGenBench is a comprehensive evaluation framework for synthetic chest radiographs that assesses fidelity, privacy, and clinical utility of text-to-image generative models using standardized metrics and a unified protocol.
paper_ai_keywords: text-to-image generative models CheXGenBench synthetic chest radiograph evaluation framework generation quality privacy vulnerabilities clinical applicability standardised data partitioning evaluation protocol quantitative metrics Sana 0.6B SynthCheX-75K