title: BenchX: A Unified Benchmark Framework for Medical Vision-Language
    Pretraining on Chest X-Rays
summary: Medical Vision-Language Pretraining (MedVLP) shows promise in learning
  generalizable and transferable visual representations from paired and unpaired
  medical images and reports. MedVLP can provide useful features to downstream
  tasks and facilitate adapting task-specific models to new setups using fewer
  examples. However, existing MedVLP methods often differ in terms of datasets,
  preprocessing, and finetuning implementations. This pose great challenges in
  evaluating how well a MedVLP method generalizes to various clinically-relevant
  tasks due to the lack of unified, standardized, and comprehensive benchmark. To
  fill this gap, we propose BenchX, a unified benchmark framework that enables
  head-to-head comparison and systematical analysis between MedVLP methods using
  public chest X-ray datasets. Specifically, BenchX is composed of three
  components: 1) Comprehensive datasets covering nine datasets and four medical
  tasks; 2) Benchmark suites to standardize data preprocessing, train-test
  splits, and parameter selection; 3) Unified finetuning protocols that
  accommodate heterogeneous MedVLP methods for consistent task adaptation in
  classification, segmentation, and report generation, respectively. Utilizing
  BenchX, we establish baselines for nine state-of-the-art MedVLP methods and
  found that the performance of some early MedVLP methods can be enhanced to
  surpass more recent ones, prompting a revisiting of the developments and
  conclusions from prior works in MedVLP. Our code are available at
  https://github.com/yangzhou12/BenchX.
paper_authors._id: 6721eb58d3084146fa326496
paper_authors.name: Yang Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:00.565Z
paper_authors._id: 670cc0b60c2a160841c8a343
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/21wJ_Eg4jBvTfaGR4Y7CY.png
paper_authors.fullname: ZHOU Yang
paper_authors.type: user
paper_authors.user: youngzhou12
paper_authors._id: 6721eb58d3084146fa326497
paper_authors.name: Tan Li Hui Faith
paper_authors._id: 6721eb58d3084146fa326498
paper_authors.name: Yanyu Xu
paper_authors._id: 6721eb58d3084146fa326499
paper_authors.name: Sicong Leng
paper_authors._id: 6721eb58d3084146fa32649a
paper_authors.name: Xinxing Xu
paper_authors._id: 6721eb58d3084146fa32649b
paper_authors.name: Yong Liu
paper_authors._id: 6721eb58d3084146fa32649c
paper_authors.name: Rick Siow Mong Goh
paper_title: BenchX: A Unified Benchmark Framework for Medical Vision-Language
    Pretraining on Chest X-Rays
paper_summary: Medical Vision-Language Pretraining (MedVLP) shows promise in learning
  generalizable and transferable visual representations from paired and unpaired
  medical images and reports. MedVLP can provide useful features to downstream
  tasks and facilitate adapting task-specific models to new setups using fewer
  examples. However, existing MedVLP methods often differ in terms of datasets,
  preprocessing, and finetuning implementations. This pose great challenges in
  evaluating how well a MedVLP method generalizes to various clinically-relevant
  tasks due to the lack of unified, standardized, and comprehensive benchmark. To
  fill this gap, we propose BenchX, a unified benchmark framework that enables
  head-to-head comparison and systematical analysis between MedVLP methods using
  public chest X-ray datasets. Specifically, BenchX is composed of three
  components: 1) Comprehensive datasets covering nine datasets and four medical
  tasks; 2) Benchmark suites to standardize data preprocessing, train-test
  splits, and parameter selection; 3) Unified finetuning protocols that
  accommodate heterogeneous MedVLP methods for consistent task adaptation in
  classification, segmentation, and report generation, respectively. Utilizing
  BenchX, we establish baselines for nine state-of-the-art MedVLP methods and
  found that the performance of some early MedVLP methods can be enhanced to
  surpass more recent ones, prompting a revisiting of the developments and
  conclusions from prior works in MedVLP. Our code are available at
  https://github.com/yangzhou12/BenchX.
paper_ai_keywords: Mot Vision-Language Pretraining MedVLP visual representations medical images reports downstream tasks task-specific models datasets preprocessing finetuning generalizes clinically-relevant tasks benchmark BenchX comprehensive datasets medical tasks benchmark suites data preprocessing train-test splits parameter selection finetuning protocols classification segmentation report generation state-of-the-art MedVLP methods