Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles
<|reference_start|>Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles: Event coreference resolution (ECR) is the task of determining whether distinct mentions of events within a multi-document corpus are actually linked to the same underlying occurrence. Images of the events can help facilitate resolution when language is ambiguous. Here, we propose a multimodal cross-document event coreference resolution method that integrates visual and textual cues with a simple linear map between vision and language models. As existing ECR benchmark datasets rarely provide images for all event mentions, we augment the popular ECB+ dataset with event-centric images scraped from the internet and generated using image diffusion models. We establish three methods that incorporate images and text for coreference: 1) a standard fused model with finetuning, 2) a novel linear mapping method without finetuning and 3) an ensembling approach based on splitting mention pairs by semantic and discourse-level difficulty. We evaluate on 2 datasets: the augmented ECB+, and AIDA Phase 1. Our ensemble systems using cross-modal linear mapping establish an upper limit (91.9 CoNLL F1) on ECB+ ECR performance given the preprocessing assumptions used, and establish a novel baseline on AIDA Phase 1. Our results demonstrate the utility of multimodal information in ECR for certain challenging coreference problems, and highlight a need for more multimodal resources in the coreference resolution space.<|reference_end|>
@article{nath2024multimodal,
    title={Multimodal Cross-Document Event Coreference Resolution Using Linear
  Semantic Transfer and Mixed-Modality Ensembles},
    author={Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Baker, Rahul
  Ghosh, James H. Martin, Nathaniel Blanchard and Nikhil Krishnaswamy},
    journal={arXiv preprint arXiv:2404.08949},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.08949},
    primaryClass={cs.CL}
}