title: PingPong: A Benchmark for Role-Playing Language Models with User
    Emulation and Multi-Model Evaluation
summary: We introduce a novel benchmark for evaluating the role-playing capabilities
  of language models. Our approach leverages language models themselves to
  emulate users in dynamic, multi-turn conversations and to assess the resulting
  dialogues. The framework consists of three main components: a player model
  assuming a specific character role, an interrogator model simulating user
  behavior, and a judge model evaluating conversation quality. We conducted
  experiments comparing automated evaluations with human annotations to validate
  our approach, demonstrating strong correlations across multiple criteria. This
  work provides a foundation for a robust and dynamic evaluation of model
  capabilities in interactive scenarios.
paper_authors._id: 66e289df26ea157c05c054fc
paper_authors.name: Ilya Gusev
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-12T06:27:43.769Z
paper_authors._id: 5fc2346dea82dd667bb0ffbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1612371927570-5fc2346dea82dd667bb0ffbc.jpeg
paper_authors.fullname: Ilya Gusev
paper_authors.type: user
paper_authors.user: IlyaGusev
paper_title: PingPong: A Benchmark for Role-Playing Language Models with User
    Emulation and Multi-Model Evaluation
paper_summary: We introduce a novel benchmark for evaluating the role-playing capabilities
  of language models. Our approach leverages language models themselves to
  emulate users in dynamic, multi-turn conversations and to assess the resulting
  dialogues. The framework consists of three main components: a player model
  assuming a specific character role, an interrogator model simulating user
  behavior, and a judge model evaluating conversation quality. We conducted
  experiments comparing automated evaluations with human annotations to validate
  our approach, demonstrating strong correlations across multiple criteria. This
  work provides a foundation for a robust and dynamic evaluation of model
  capabilities in interactive scenarios.
paper_ai_summary: A benchmark evaluates language models' role-playing abilities using automated components to simulate and judge dialogues.
paper_ai_keywords: role-playing capabilities language models player model interrogator model judge model automated evaluations human annotations conversation quality interactive scenarios