Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence
<|reference_start|>Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence: Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective. However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence. The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles. Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods.<|reference_end|>
@article{liu2024unlocking,
    title={Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for
  Positional Discourse Coherence},
    author={Yinhong Liu, Yixuan Su, Ehsan Shareghi and Nigel Collier},
    journal={arXiv preprint arXiv:2402.10175},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.10175},
    primaryClass={cs.CL}
}