SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation
<|reference_start|>SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation: The long-standing one-to-many problem of gold standard responses in open-domain dialogue systems presents challenges for automatic evaluation metrics. Though prior works have demonstrated some success by applying powerful Large Language Models (LLMs), existing approaches still struggle with the one-to-many problem, and exhibit subpar performance in domain-specific scenarios. We assume the commonsense reasoning biases within LLMs may hinder their performance in domainspecific evaluations. To address both issues, we propose a novel framework SLIDE (Small and Large Integrated for Dialogue Evaluation), that leverages both a small, specialised model (SLM), and LLMs for the evaluation of open domain dialogues. Our approach introduces several techniques: (1) Contrastive learning to differentiate between robust and non-robust response embeddings; (2) A novel metric for semantic sensitivity that combines embedding cosine distances with similarity learned through neural networks, and (3) a strategy for incorporating the evaluation results from both the SLM and LLMs. Our empirical results demonstrate that our approach achieves state-of-the-art performance in both the classification and evaluation tasks, and additionally the SLIDE evaluator exhibits better correlation with human judgements. Our code is available at https:// github.com/hegehongcha/SLIDE-ACL2024.<|reference_end|>
@article{zhao2024slide:,
    title={SLIDE: A Framework Integrating Small and Large Language Models for
  Open-Domain Dialogues Evaluation},
    author={Kun Zhao, Bohao Yang, Chen Tang, Chenghua Lin, Liang Zhan},
    journal={arXiv preprint arXiv:2405.15924},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.15924},
    primaryClass={cs.CL}
}