FITA: Fine-grained Image-Text Aligner for Radiology Report Generation
<|reference_start|>FITA: Fine-grained Image-Text Aligner for Radiology Report Generation: Radiology report generation aims to automatically generate detailed and coherent descriptive reports alongside radiology images. Previous work mainly focused on refining fine-grained image features or leveraging external knowledge. However, the precise alignment of fine-grained image features with corresponding text descriptions has not been considered. This paper presents a novel method called Fine-grained Image-Text Aligner (FITA) to construct fine-grained alignment for image and text features. It has three novel designs: Image Feature Refiner (IFR), Text Feature Refiner (TFR) and Contrastive Aligner (CA). IFR and TFR aim to learn fine-grained image and text features, respectively. We achieve this by leveraging saliency maps to effectively fuse symptoms with corresponding abnormal visual regions, and by utilizing a meticulously constructed triplet set for training. Finally, CA module aligns fine-grained image and text features using contrastive loss for precise alignment. Results show that our method surpasses existing methods on the widely used benchmark<|reference_end|>
@article{yang2024fita:,
    title={FITA: Fine-grained Image-Text Aligner for Radiology Report Generation},
    author={Honglong Yang, Hui Tang, Xiaomeng Li},
    journal={arXiv preprint arXiv:2405.00962},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00962},
    primaryClass={cs.CV}
}