Rethinking Generative Large Language Model Evaluation for Semantic Comprehension
<|reference_start|>Rethinking Generative Large Language Model Evaluation for Semantic Comprehension: Despite their sophisticated capabilities, large language models (LLMs) encounter a major hurdle in effective assessment. This paper first revisits the prevalent evaluation method-multiple choice question answering (MCQA), which allows for straightforward accuracy measurement. Through a comprehensive evaluation of 24 models across 11 benchmarks, we highlight several potential drawbacks of MCQA, for instance, the inconsistency between the MCQA evaluation and the generation of open-ended responses in practical scenarios. In response, we introduce an RWQ-Elo rating system, engaging 24 LLMs such as GPT-4, GPT-3.5, Google-Gemini-Pro and LLaMA-1/-2, in a two-player competitive format, with GPT-4 serving as the judge. Each LLM receives an Elo rating thereafter. This system is designed to mirror real-world usage, and for this purpose, we have compiled a new benchmark called ``Real-world questions'' (RWQ), comprising 20,772 authentic user inquiries. Additionally, we thoroughly analyze the characteristics of our system and compare it with prior leaderboards like AlpacaEval and MT-Bench. Our analysis reveals the stability of our RWQ-Elo system, the feasibility of registering new models, and its potential to reshape LLM leaderboards.<|reference_end|>
@article{wei2024rethinking,
    title={Rethinking Generative Large Language Model Evaluation for Semantic
  Comprehension},
    author={Fangyun Wei, Xi Chen, Lin Luo},
    journal={arXiv preprint arXiv:2403.07872},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.07872},
    primaryClass={cs.CL}
}