Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text
<|reference_start|>Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text: The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics like BLEU and ROUGE, while useful, are increasingly inadequate for capturing the subtle semantics and contextual richness of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs-as-judges. Through experiments on three open-ended question-answering tasks, we demonstrate that combining multiple LLMs-as-judges significantly improves the reliability and accuracy of evaluations, particularly in complex tasks where a single model might struggle. Our findings reveal a strong correlation with human evaluations, establishing our method as a viable and effective alternative to traditional metrics and human judgments, particularly in the context of LLM-based chat assistants where the complexity and diversity of responses challenge existing benchmarks.<|reference_end|>
@article{badshah2024reference-guided,
    title={Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of
  Free-Form Text},
    author={Sher Badshah, Hassan Sajjad},
    journal={arXiv preprint arXiv:2408.09235},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.09235},
    primaryClass={cs.CL cs.AI}
}