Using Similarity to Evaluate Factual Consistency in Summaries
<|reference_start|>Using Similarity to Evaluate Factual Consistency in Summaries: Cutting-edge abstractive summarisers generate fluent summaries, but the factuality of the generated text is not guaranteed. Early summary factuality evaluation metrics are usually based on n-gram overlap and embedding similarity, but are reported fail to align with human annotations. Therefore, many techniques for detecting factual inconsistencies build pipelines around natural language inference (NLI) or question-answering (QA) models with additional supervised learning steps. In this paper, we revisit similarity-based metrics, showing that this failure stems from the comparison text selection and its granularity. We propose a new zero-shot factuality evaluation metric, Sentence-BERT Score (SBERTScore), which compares sentences between the summary and the source document. It outperforms widely-used word-word metrics including BERTScore and can compete with existing NLI and QA-based factuality metrics on the benchmark without needing any fine-tuning. Our experiments indicate that each technique has different strengths, with SBERTScore particularly effective in identifying correct summaries. We demonstrate how a combination of techniques is more effective in detecting various types of error.<|reference_end|>
@article{ye2024using,
    title={Using Similarity to Evaluate Factual Consistency in Summaries},
    author={Yuxuan Ye, Edwin Simpson, Raul Santos Rodriguez},
    journal={arXiv preprint arXiv:2409.15090},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.15090},
    primaryClass={cs.CL}
}