title: Leveraging Word Guessing Games to Assess the Intelligence of Large
    Language Models
summary: The automatic evaluation of LLM-based agent intelligence is critical in
  developing advanced LLM-based agents. Although considerable effort has been
  devoted to developing human-annotated evaluation datasets, such as AlpacaEval,
  existing techniques are costly, time-consuming, and lack adaptability. In this
  paper, inspired by the popular language game ``Who is Spy'', we propose to use
  the word guessing game to assess the intelligence performance of LLMs. Given a
  word, the LLM is asked to describe the word and determine its identity (spy or
  not) based on its and other players' descriptions. Ideally, an advanced agent
  should possess the ability to accurately describe a given word using an
  aggressive description while concurrently maximizing confusion in the
  conservative description, enhancing its participation in the game. To this end,
  we first develop DEEP to evaluate LLMs' expression and disguising abilities.
  DEEP requires LLM to describe a word in aggressive and conservative modes. We
  then introduce SpyGame, an interactive multi-agent framework designed to assess
  LLMs' intelligence through participation in a competitive language-based board
  game. Incorporating multi-agent interaction, SpyGame requires the target LLM to
  possess linguistic skills and strategic thinking, providing a more
  comprehensive evaluation of LLMs' human-like cognitive abilities and
  adaptability in complex communication situations. The proposed evaluation
  framework is very easy to implement. We collected words from multiple sources,
  domains, and languages and used the proposed evaluation framework to conduct
  experiments. Extensive experiments demonstrate that the proposed DEEP and
  SpyGame effectively evaluate the capabilities of various LLMs, capturing their
  ability to adapt to novel situations and engage in strategic communication.
paper_authors._id: 6541c04b23e8ef68e44f7aae
paper_authors.name: Tian Liang
paper_authors._id: 6541c04b23e8ef68e44f7aaf
paper_authors.name: Zhiwei He
paper_authors._id: 6541c04b23e8ef68e44f7ab0
paper_authors.name: Jen-tes Huang
paper_authors._id: 6541c04b23e8ef68e44f7ab1
paper_authors.name: Wenxuan Wang
paper_authors._id: 6541c04b23e8ef68e44f7ab2
paper_authors.name: Wenxiang Jiao
paper_authors._id: 6541c04b23e8ef68e44f7ab3
paper_authors.name: Rui Wang
paper_authors._id: 6541c04b23e8ef68e44f7ab4
paper_authors.name: Yujiu Yang
paper_authors._id: 6541c04b23e8ef68e44f7ab5
paper_authors.name: Zhaopeng Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T11:44:40.759Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_authors._id: 6541c04b23e8ef68e44f7ab6
paper_authors.name: Shuming Shi
paper_authors._id: 6541c04b23e8ef68e44f7ab7
paper_authors.name: Xing Wang
paper_title: Leveraging Word Guessing Games to Assess the Intelligence of Large
    Language Models
paper_summary: The automatic evaluation of LLM-based agent intelligence is critical in
  developing advanced LLM-based agents. Although considerable effort has been
  devoted to developing human-annotated evaluation datasets, such as AlpacaEval,
  existing techniques are costly, time-consuming, and lack adaptability. In this
  paper, inspired by the popular language game ``Who is Spy'', we propose to use
  the word guessing game to assess the intelligence performance of LLMs. Given a
  word, the LLM is asked to describe the word and determine its identity (spy or
  not) based on its and other players' descriptions. Ideally, an advanced agent
  should possess the ability to accurately describe a given word using an
  aggressive description while concurrently maximizing confusion in the
  conservative description, enhancing its participation in the game. To this end,
  we first develop DEEP to evaluate LLMs' expression and disguising abilities.
  DEEP requires LLM to describe a word in aggressive and conservative modes. We
  then introduce SpyGame, an interactive multi-agent framework designed to assess
  LLMs' intelligence through participation in a competitive language-based board
  game. Incorporating multi-agent interaction, SpyGame requires the target LLM to
  possess linguistic skills and strategic thinking, providing a more
  comprehensive evaluation of LLMs' human-like cognitive abilities and
  adaptability in complex communication situations. The proposed evaluation
  framework is very easy to implement. We collected words from multiple sources,
  domains, and languages and used the proposed evaluation framework to conduct
  experiments. Extensive experiments demonstrate that the proposed DEEP and
  SpyGame effectively evaluate the capabilities of various LLMs, capturing their
  ability to adapt to novel situations and engage in strategic communication.
paper_ai_summary: A new evaluation framework using interactive multi-agent games assesses LLMs' intelligence, expression, and strategic communication skills.
paper_ai_keywords: LLM-based agents AlpacaEval word guessing game DEEP SpyGame multi-agent framework linguistic skills strategic thinking human-like cognitive abilities complex communication situations