Auditing Counterfire: Evaluating Advanced Counterargument Generation with Evidence and Style
<|reference_start|>Auditing Counterfire: Evaluating Advanced Counterargument Generation with Evidence and Style: We audited large language models (LLMs) for their ability to create evidence-based and stylistic counter-arguments to posts from the Reddit ChangeMyView dataset. We benchmarked their rhetorical quality across a host of qualitative and quantitative metrics and then ultimately evaluated them on their persuasive abilities as compared to human counter-arguments. Our evaluation is based on Counterfire: a new dataset of 32,000 counter-arguments generated from large language models (LLMs): GPT-3.5 Turbo and Koala and their fine-tuned variants, and PaLM 2, with varying prompts for evidence use and argumentative style. GPT-3.5 Turbo ranked highest in argument quality with strong paraphrasing and style adherence, particularly in `reciprocity' style arguments. However, the stylistic counter-arguments still fall short of human persuasive standards, where people also preferred reciprocal to evidence-based rebuttals. The findings suggest that a balance between evidentiality and stylistic elements is vital to a compelling counter-argument. We close with a discussion of future research directions and implications for evaluating LLM outputs.<|reference_end|>
@article{verma2024auditing,
    title={Auditing Counterfire: Evaluating Advanced Counterargument Generation
  with Evidence and Style},
    author={Preetika Verma, Kokil Jaidka, Svetlana Churina},
    journal={arXiv preprint arXiv:2402.08498},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.08498},
    primaryClass={cs.CL}
}