title: PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge
summary: The recently released ChatGPT model demonstrates unprecedented capabilities
  in zero-shot question-answering. In this work, we probe ChatGPT for its
  conversational understanding and introduce a conversational framework
  (protocol) that can be adopted in future studies. The Pok\'emon universe serves
  as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to
  its closed world assumption. After bringing ChatGPT's background knowledge (on
  the Pok\'emon universe) to light, we test its reasoning process when using
  these concepts in battle scenarios. We then evaluate its ability to acquire new
  knowledge and include it in its reasoning process. Our ultimate goal is to
  assess ChatGPT's ability to generalize, combine features, and to acquire and
  reason over newly introduced knowledge from human feedback. We find that
  ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in
  battle scenarios to a great extent, even when new information is introduced.
  The model performs better with collaborative feedback and if there is an
  initial phase of information retrieval, but also hallucinates occasionally and
  is susceptible to adversarial attacks.
paper_authors._id: 647e86aa0ed7d0c8760e482d
paper_authors.name: Laura Cabello
paper_authors._id: 647e86aa0ed7d0c8760e482e
paper_authors.name: Jiaang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:46:54.809Z
paper_authors._id: 6686658221f096c7fb4e0b24
paper_authors.avatarUrl: /avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
paper_authors.fullname: Jiaang Li
paper_authors.type: user
paper_authors.user: jaagli
paper_authors._id: 647e86aa0ed7d0c8760e482f
paper_authors.name: Ilias Chalkidis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T12:32:08.642Z
paper_authors._id: 621c8bef07466b091d45be45
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1646037955111-noauth.jpeg
paper_authors.fullname: Ilias Chalkidis
paper_authors.type: user
paper_authors.user: kiddothe2b
paper_title: PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge
paper_summary: The recently released ChatGPT model demonstrates unprecedented capabilities
  in zero-shot question-answering. In this work, we probe ChatGPT for its
  conversational understanding and introduce a conversational framework
  (protocol) that can be adopted in future studies. The Pok\'emon universe serves
  as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to
  its closed world assumption. After bringing ChatGPT's background knowledge (on
  the Pok\'emon universe) to light, we test its reasoning process when using
  these concepts in battle scenarios. We then evaluate its ability to acquire new
  knowledge and include it in its reasoning process. Our ultimate goal is to
  assess ChatGPT's ability to generalize, combine features, and to acquire and
  reason over newly introduced knowledge from human feedback. We find that
  ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in
  battle scenarios to a great extent, even when new information is introduced.
  The model performs better with collaborative feedback and if there is an
  initial phase of information retrieval, but also hallucinates occasionally and
  is susceptible to adversarial attacks.
paper_ai_summary: ChatGPT's reasoning capabilities in the Pokemon universe are assessed through conversational understanding, battle scenarios, and new knowledge acquisition, revealing strengths in prior knowledge reuse and weaknesses in hallucination and adversarial resistance.
paper_ai_keywords: zero-shot question-answering conversational understanding conversational framework protocol closed world assumption background knowledge battle scenarios new knowledge reasoning process human feedback generalization feature combination adversarial attacks