mTREE: Multi-Level Text-Guided Representation End-to-End Learning for Whole Slide Image Analysis
<|reference_start|>mTREE: Multi-Level Text-Guided Representation End-to-End Learning for Whole Slide Image Analysis: Multi-modal learning adeptly integrates visual and textual data, but its application to histopathology image and text analysis remains challenging, particularly with large, high-resolution images like gigapixel Whole Slide Images (WSIs). Current methods typically rely on manual region labeling or multi-stage learning to assemble local representations (e.g., patch-level) into global features (e.g., slide-level). However, there is no effective way to integrate multi-scale image representations with text data in a seamless end-to-end process. In this study, we introduce Multi-Level Text-Guided Representation End-to-End Learning (mTREE). This novel text-guided approach effectively captures multi-scale WSI representations by utilizing information from accompanying textual pathology information. mTREE innovatively combines - the localization of key areas (global-to-local) and the development of a WSI-level image-text representation (local-to-global) - into a unified, end-to-end learning framework. In this model, textual information serves a dual purpose: firstly, functioning as an attention map to accurately identify key areas, and secondly, acting as a conduit for integrating textual features into the comprehensive representation of the image. Our study demonstrates the effectiveness of mTREE through quantitative analyses in two image-related tasks: classification and survival prediction, showcasing its remarkable superiority over baselines.<|reference_end|>
@article{liu2024mtree:,
    title={mTREE: Multi-Level Text-Guided Representation End-to-End Learning for
  Whole Slide Image Analysis},
    author={Quan Liu, Ruining Deng, Can Cui, Tianyuan Yao, Vishwesh Nath, Yucheng
  Tang, Yuankai Huo},
    journal={arXiv preprint arXiv:2405.17824},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17824},
    primaryClass={cs.CV}
}