title: Towards Scalable Language-Image Pre-training for 3D Medical Imaging
summary: Language-image pre-training has demonstrated strong performance in 2D medical
  imaging, but its success in 3D modalities such as CT and MRI remains limited
  due to the high computational demands of volumetric data, which pose a
  significant barrier to training on large-scale, uncurated clinical studies. In
  this study, we introduce Hierarchical attention for Language-Image Pre-training
  (HLIP), a scalable pre-training framework for 3D medical imaging. HLIP adopts a
  lightweight hierarchical attention mechanism inspired by the natural hierarchy
  of radiology data: slice, scan, and study. This mechanism exhibits strong
  generalizability, e.g., +4.3% macro AUC on the Rad-ChestCT benchmark when
  pre-trained on CT-RATE. Moreover, the computational efficiency of HLIP enables
  direct training on uncurated datasets. Trained on 220K patients with 3.13
  million scans for brain MRI and 240K patients with 1.44 million scans for head
  CT, HLIP achieves state-of-the-art performance, e.g., +32.4% balanced ACC on
  the proposed publicly available brain MRI benchmark Pub-Brain-5; +1.4% and
  +6.9% macro AUC on head CT benchmarks RSNA and CQ500, respectively. These
  results demonstrate that, with HLIP, directly pre-training on uncurated
  clinical datasets is a scalable and effective direction for language-image
  pre-training in 3D medical imaging. The code is available at
  https://github.com/Zch0414/hlip
paper_authors._id: 68389ac5d011eaeee3d8e75a
paper_authors.name: Chenhui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:19:31.423Z
paper_authors._id: 6701671ba48acf5f915ac060
paper_authors.avatarUrl: /avatars/b830ab57457ed14780228a46d315f42f.svg
paper_authors.fullname: Chenhui Zhao
paper_authors.type: user
paper_authors.user: Zch0414
paper_authors._id: 68389ac5d011eaeee3d8e75b
paper_authors.name: Yiwei Lyu
paper_authors._id: 68389ac5d011eaeee3d8e75c
paper_authors.name: Asadur Chowdury
paper_authors._id: 68389ac5d011eaeee3d8e75d
paper_authors.name: Edward Harake
paper_authors._id: 68389ac5d011eaeee3d8e75e
paper_authors.name: Akhil Kondepudi
paper_authors._id: 68389ac5d011eaeee3d8e75f
paper_authors.name: Akshay Rao
paper_authors._id: 68389ac5d011eaeee3d8e760
paper_authors.name: Xinhai Hou
paper_authors._id: 68389ac5d011eaeee3d8e761
paper_authors.name: Honglak Lee
paper_authors._id: 68389ac5d011eaeee3d8e762
paper_authors.name: Todd Hollon
paper_title: Towards Scalable Language-Image Pre-training for 3D Medical Imaging
paper_summary: Language-image pre-training has demonstrated strong performance in 2D medical
  imaging, but its success in 3D modalities such as CT and MRI remains limited
  due to the high computational demands of volumetric data, which pose a
  significant barrier to training on large-scale, uncurated clinical studies. In
  this study, we introduce Hierarchical attention for Language-Image Pre-training
  (HLIP), a scalable pre-training framework for 3D medical imaging. HLIP adopts a
  lightweight hierarchical attention mechanism inspired by the natural hierarchy
  of radiology data: slice, scan, and study. This mechanism exhibits strong
  generalizability, e.g., +4.3% macro AUC on the Rad-ChestCT benchmark when
  pre-trained on CT-RATE. Moreover, the computational efficiency of HLIP enables
  direct training on uncurated datasets. Trained on 220K patients with 3.13
  million scans for brain MRI and 240K patients with 1.44 million scans for head
  CT, HLIP achieves state-of-the-art performance, e.g., +32.4% balanced ACC on
  the proposed publicly available brain MRI benchmark Pub-Brain-5; +1.4% and
  +6.9% macro AUC on head CT benchmarks RSNA and CQ500, respectively. These
  results demonstrate that, with HLIP, directly pre-training on uncurated
  clinical datasets is a scalable and effective direction for language-image
  pre-training in 3D medical imaging. The code is available at
  https://github.com/Zch0414/hlip
paper_ai_summary: Hierarchical attention mechanism for language-image pre-training in 3D medical imaging achieves state-of-the-art performance on uncurated clinical datasets.
paper_ai_keywords: Hierarchical attention language-image pre-training HLIP radiology data macro AUC balanced ACC Rad-ChestCT CT-RATE brain MRI head CT Pub-Brain-5 RSNA CQ500