HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild
<|reference_start|>HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild: Hallucinations pose a significant challenge to the reliability of large language models (LLMs) in critical domains. Recent benchmarks designed to assess LLM hallucinations within conventional NLP tasks, such as knowledge-intensive question answering (QA) and summarization, are insufficient for capturing the complexities of user-LLM interactions in dynamic, real-world settings. To address this gap, we introduce HaluEval-Wild, the first benchmark specifically designed to evaluate LLM hallucinations in the wild. We meticulously collect challenging (adversarially filtered by Alpaca) user queries from ShareGPT, an existing real-world user-LLM interaction datasets, to evaluate the hallucination rates of various LLMs. Upon analyzing the collected queries, we categorize them into five distinct types, which enables a fine-grained analysis of the types of hallucinations LLMs exhibit, and synthesize the reference answers with the powerful GPT-4 model and retrieval-augmented generation (RAG). Our benchmark offers a novel approach towards enhancing our comprehension of and improving LLM reliability in scenarios reflective of real-world interactions. Our benchmark is available at https://github.com/HaluEval-Wild/HaluEval-Wild.<|reference_end|>
@article{zhu2024halueval-wild:,
    title={HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild},
    author={Zhiying Zhu, Yiming Yang, Zhiqing Sun},
    journal={arXiv preprint arXiv:2403.04307},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04307},
    primaryClass={cs.CL}
}