Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions
<|reference_start|>Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions: Evaluating the output of generative large language models (LLMs) is challenging and difficult to scale. Most evaluations of LLMs focus on tasks such as single-choice question-answering or text classification. These tasks are not suitable for assessing open-ended question-answering capabilities, which are critical in domains where expertise is required, such as health, and where misleading or incorrect answers can have a significant impact on a user's health. Using human experts to evaluate the quality of LLM answers is generally considered the gold standard, but expert annotation is costly and slow. We present a method for evaluating LLM answers that uses ranking signals as a substitute for explicit relevance judgements. Our scoring method correlates with the preferences of human experts. We validate it by investigating the well-known fact that the quality of generated answers improves with the size of the model as well as with more sophisticated prompting strategies.<|reference_end|>
@article{heineking2024ranking,
    title={Ranking Generated Answers: On the Agreement of Retrieval Models with
  Humans on Consumer Health Questions},
    author={Sebastian Heineking, Jonas Probst, Daniel Steinbach, Martin Potthast,
  Harrisen Scells},
    journal={arXiv preprint arXiv:2408.09831},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.09831},
    primaryClass={cs.IR}
}