Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification
<|reference_start|>Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification: Long Document Classification (LDC) has gained significant attention recently. However, multi-modal data in long documents such as texts and images are not being effectively utilized. Prior studies in this area have attempted to integrate texts and images in document-related tasks, but they have only focused on short text sequences and images of pages. How to classify long documents with hierarchical structure texts and embedding images is a new problem and faces multi-modal representation difficulties. In this paper, we propose a novel approach called Hierarchical Multi-modal Transformer (HMT) for cross-modal long document classification. The HMT conducts multi-modal feature interaction and fusion between images and texts in a hierarchical manner. Our approach uses a multi-modal transformer and a dynamic multi-scale multi-modal transformer to model the complex relationships between image features, and the section and sentence features. Furthermore, we introduce a new interaction strategy called the dynamic mask transfer module to integrate these two transformers by propagating features between them. To validate our approach, we conduct cross-modal LDC experiments on two newly created and two publicly available multi-modal long document datasets, and the results show that the proposed HMT outperforms state-of-the-art single-modality and multi-modality methods.<|reference_end|>
@article{liu2024hierarchical,
    title={Hierarchical Multi-modal Transformer for Cross-modal Long Document
  Classification},
    author={Tengfei Liu, Yongli Hu, Junbin Gao, Yanfeng Sun, Baocai Yin},
    journal={arXiv preprint arXiv:2407.10105},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.10105},
    primaryClass={cs.CV cs.AI}
}