PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation
<|reference_start|>PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation: We introduce a novel benchmark for evaluating the role-playing capabilities of language models. Our approach leverages language models themselves to emulate users in dynamic, multi-turn conversations and to assess the resulting dialogues. The framework consists of three main components: a player model assuming a specific character role, an interrogator model simulating user behavior, and a judge model evaluating conversation quality. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of model capabilities in interactive scenarios.<|reference_end|>
@article{gusev2024pingpong:,
    title={PingPong: A Benchmark for Role-Playing Language Models with User
  Emulation and Multi-Model Evaluation},
    author={Ilya Gusev},
    journal={arXiv preprint arXiv:2409.06820},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.06820},
    primaryClass={cs.CL}
}