MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking
<|reference_start|>MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking: Fact-checking real-world claims often requires reviewing multiple multimodal documents to assess a claim's truthfulness, which is a highly laborious and time-consuming task. In this paper, we present a summarization model designed to generate claim-specific summaries useful for fact-checking from multimodal, multi-document datasets. The model takes inputs in the form of documents, images, and a claim, with the objective of assisting in fact-checking tasks. We introduce a dynamic perceiver-based model that can handle inputs from multiple modalities of arbitrary lengths. To train our model, we leverage a novel reinforcement learning-based entailment objective to generate summaries that provide evidence distinguishing between different truthfulness labels. To assess the efficacy of our approach, we conduct experiments on both an existing benchmark and a new dataset of multi-document claims that we contribute. Our approach outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and demonstrates strong performance on our new Multi-News-Fact-Checking dataset.<|reference_end|>
@article{chen2024metasumperceiver:,
    title={MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for
  Fact-Checking},
    author={Ting-Chih Chen, Chia-Wei Tang, Chris Thomas},
    journal={arXiv preprint arXiv:2407.13089},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.13089},
    primaryClass={cs.AI cs.CL}
}