μ^2Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation
SiyouLi
Abstract
Automated radiology report generation (RRG) aims to produce detailed textual reports from clinical imaging, such as computed tomography (CT) scans, to improve the accuracy and efficiency of diagnosis and provision of management advice. RRG is complicated by two key challenges: (1) inherent complexity in extracting relevant information from imaging data under resource constraints, and (2) difficulty in objectively evaluating discrepancies between model-generated and expert-written reports. To address these challenges, we propose mu^2LLM, a textbf{mu}ltiscale textbf{mu}ltimodal large language models for RRG tasks. The novel {mu}^2Tokenizer, as an intermediate layer, integrates multi-modal features from the multiscale visual tokenizer and the text tokenizer, then enhances report generation quality through direct preference optimization (DPO), guided by GREEN-RedLlama. Experimental results on four large CT image-report medical datasetdemonstrate that our method outperforms existing approaches, highlighting the potential of our fine-tuned mu^2LLMs on limited data for RRG tasks.
Community
🚀 Thrilled to announce that our paper “µ²Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation” has been accepted to MICCAI 2025! 🥳
In a nutshell, µ²Tokenizer is a lightweight middleware that fuses 3D ViT features with language models, delivering state-of-the-art radiology reports with just 1 B parameters—14 % of the size of typical baselines. Key wins:
• 📊 +20 % GREEN-Score boost over larger models thanks to Direct Preference Optimisation
• 🧠 Soft, differentiable token selection → richer image understanding without blowing up compute
• ⚡ Dynamic multi-scale pooling that adapts to each CT scan on the fly
• 🏆 Tested on AMOS-MM, CT-Rate & AbdomenAtlas, consistently outperforming 7B–14B LLMs
Why it matters: automating radiology reporting at scale can free up clinicians, cut reporting times, and improve patient outcomes, without requiring giant models or giant budgets.
🔗 Paper: https://u2tokenizer.github.io/static/pdfs/%CE%BC2_Tokenizer.pdf
🌍 Website: https://u2tokenizer.github.io/
📦 Github: https://github.com/Siyou-Li/u2Tokenizer
🤝 Always open to collaborations on multi-modal LLM—drop me a message or tag someone who might be interested!
#MICCAI2025 #MedAI #Radiology #LLM #Multimodal #HealthcareInnovation
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper