Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple Judges
Evaluating the conversational abilities of large language models (LLMs)
remains a challenging task. Current mainstream approaches primarily rely on the
``LLM-as-a-judge" paradigm, where an LLM is prompted to serve as an evaluator
to assess dialogue quality. However, such methods often suffer from various
biases, which undermine the reliability and consistency of the evaluation
results. To mitigate these biases, recent methods employ multiple LLMs as
judges and aggregate their judgments to select the optimal assessment. Although
effective, this multi-judge approach incurs significant computational overhead
during inference. In this paper, we propose an efficient multi-turn dialogue
evaluator that captures the collective wisdom of multiple LLM judges by
aggregating their preference knowledge into a single model. Our approach
preserves the advantages of diverse multi-judge feedback while drastically
reducing the evaluation cost, enabling fast and flexible dialogue quality
assessment. Extensive experiments on seven single rating and pairwise
comparison dialogue evaluation benchmarks demonstrate that our method
outperforms existing baselines across diverse scenarios, showcasing its
efficiency and robustness.
689022980a411b3b8d28d69e
Yuqi Tang
689022980a411b3b8d28d69f
Kehua Feng
689022980a411b3b8d28d6a0
Yunfeng Wang
689022980a411b3b8d28d6a1
Zhiwen Chen
689022980a411b3b8d28d6a2
Chengfei Lv
689022980a411b3b8d28d6a3
Gang Yu
689022980a411b3b8d28d6a4
Qiang Zhang
689022980a411b3b8d28d6a5
Keyan Ding
Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple Judges
Evaluating the conversational abilities of large language models (LLMs)
remains a challenging task. Current mainstream approaches primarily rely on the
``LLM-as-a-judge" paradigm, where an LLM is prompted to serve as an evaluator
to assess dialogue quality. However, such methods often suffer from various
biases, which undermine the reliability and consistency of the evaluation
results. To mitigate these biases, recent methods employ multiple LLMs as
judges and aggregate their judgments to select the optimal assessment. Although
effective, this multi-judge approach incurs significant computational overhead
during inference. In this paper, we propose an efficient multi-turn dialogue
evaluator that captures the collective wisdom of multiple LLM judges by
aggregating their preference knowledge into a single model. Our approach
preserves the advantages of diverse multi-judge feedback while drastically
reducing the evaluation cost, enabling fast and flexible dialogue quality
assessment. Extensive experiments on seven single rating and pairwise
comparison dialogue evaluation benchmarks demonstrate that our method
outperforms existing baselines across diverse scenarios, showcasing its
efficiency and robustness.
An efficient multi-turn dialogue evaluator aggregates multiple LLM judgments into a single model to assess dialogue quality with reduced computational cost.
LLM-as-a-judge multi-judge approach preference knowledge multi-turn dialogue evaluator dialogue evaluation benchmarks