Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization
<|reference_start|>Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization: The rapid increase in multimedia data has spurred advancements in Multimodal Summarization with Multimodal Output (MSMO), which aims to produce a multimodal summary that integrates both text and relevant images. The inherent heterogeneity of content within multimodal inputs and outputs presents a significant challenge to the execution of MSMO. Traditional approaches typically adopt a holistic perspective on coarse image-text data or individual visual objects, overlooking the essential connections between objects and the entities they represent. To integrate the fine-grained entity knowledge, we propose an Entity-Guided Multimodal Summarization model (EGMS). Our model, building on BART, utilizes dual multimodal encoders with shared weights to process text-image and entity-image information concurrently. A gating mechanism then combines visual data for enhanced textual summary generation, while image selection is refined through knowledge distillation from a pre-trained vision-language model. Extensive experiments on public MSMO dataset validate the superiority of the EGMS method, which also prove the necessity to incorporate entity information into MSMO problem.<|reference_end|>
@article{zhang2024leveraging,
    title={Leveraging Entity Information for Cross-Modality Correlation Learning:
  The Entity-Guided Multimodal Summarization},
    author={Yanghai Zhang, Ye Liu, Shiwei Wu, Kai Zhang, Xukai Liu, Qi Liu, Enhong
  Chen},
    journal={arXiv preprint arXiv:2408.03149},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.03149},
    primaryClass={cs.CV cs.CL}
}