PingPong: A Benchmark for Role-Playing Language Models with User
  Emulation and Multi-Model Evaluation
We introduce a novel benchmark for evaluating the role-playing capabilities
of language models. Our approach leverages language models themselves to
emulate users in dynamic, multi-turn conversations and to assess the resulting
dialogues. The framework consists of three main components: a player model
assuming a specific character role, an interrogator model simulating user
behavior, and a judge model evaluating conversation quality. We conducted
experiments comparing automated evaluations with human annotations to validate
our approach, demonstrating strong correlations across multiple criteria. This
work provides a foundation for a robust and dynamic evaluation of model
capabilities in interactive scenarios.
66e289df26ea157c05c054fc
Ilya Gusev
extracted_pending
2024-09-12T06:27:43.769Z
5fc2346dea82dd667bb0ffbc
https://cdn-avatars.huggingface.co/v1/production/uploads/1612371927570-5fc2346dea82dd667bb0ffbc.jpeg
Ilya Gusev
user
IlyaGusev
PingPong: A Benchmark for Role-Playing Language Models with User
  Emulation and Multi-Model Evaluation
We introduce a novel benchmark for evaluating the role-playing capabilities
of language models. Our approach leverages language models themselves to
emulate users in dynamic, multi-turn conversations and to assess the resulting
dialogues. The framework consists of three main components: a player model
assuming a specific character role, an interrogator model simulating user
behavior, and a judge model evaluating conversation quality. We conducted
experiments comparing automated evaluations with human annotations to validate
our approach, demonstrating strong correlations across multiple criteria. This
work provides a foundation for a robust and dynamic evaluation of model
capabilities in interactive scenarios.
A benchmark evaluates language models' role-playing abilities using automated components to simulate and judge dialogues.
role-playing capabilities language models player model interrogator model judge model automated evaluations human annotations conversation quality interactive scenarios