PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge
The recently released ChatGPT model demonstrates unprecedented capabilities
in zero-shot question-answering. In this work, we probe ChatGPT for its
conversational understanding and introduce a conversational framework
(protocol) that can be adopted in future studies. The Pok\'emon universe serves
as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to
its closed world assumption. After bringing ChatGPT's background knowledge (on
the Pok\'emon universe) to light, we test its reasoning process when using
these concepts in battle scenarios. We then evaluate its ability to acquire new
knowledge and include it in its reasoning process. Our ultimate goal is to
assess ChatGPT's ability to generalize, combine features, and to acquire and
reason over newly introduced knowledge from human feedback. We find that
ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in
battle scenarios to a great extent, even when new information is introduced.
The model performs better with collaborative feedback and if there is an
initial phase of information retrieval, but also hallucinates occasionally and
is susceptible to adversarial attacks.
647e86aa0ed7d0c8760e482d
Laura Cabello
647e86aa0ed7d0c8760e482e
Jiaang Li
claimed_verified
2025-05-26T08:46:54.809Z
6686658221f096c7fb4e0b24
/avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
Jiaang Li
user
jaagli
647e86aa0ed7d0c8760e482f
Ilias Chalkidis
claimed_verified
2023-06-06T12:32:08.642Z
621c8bef07466b091d45be45
https://cdn-avatars.huggingface.co/v1/production/uploads/1646037955111-noauth.jpeg
Ilias Chalkidis
user
kiddothe2b
PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge
The recently released ChatGPT model demonstrates unprecedented capabilities
in zero-shot question-answering. In this work, we probe ChatGPT for its
conversational understanding and introduce a conversational framework
(protocol) that can be adopted in future studies. The Pok\'emon universe serves
as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to
its closed world assumption. After bringing ChatGPT's background knowledge (on
the Pok\'emon universe) to light, we test its reasoning process when using
these concepts in battle scenarios. We then evaluate its ability to acquire new
knowledge and include it in its reasoning process. Our ultimate goal is to
assess ChatGPT's ability to generalize, combine features, and to acquire and
reason over newly introduced knowledge from human feedback. We find that
ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in
battle scenarios to a great extent, even when new information is introduced.
The model performs better with collaborative feedback and if there is an
initial phase of information retrieval, but also hallucinates occasionally and
is susceptible to adversarial attacks.
ChatGPT's reasoning capabilities in the Pokemon universe are assessed through conversational understanding, battle scenarios, and new knowledge acquisition, revealing strengths in prior knowledge reuse and weaknesses in hallucination and adversarial resistance.
zero-shot question-answering conversational understanding conversational framework protocol closed world assumption background knowledge battle scenarios new knowledge reasoning process human feedback generalization feature combination adversarial attacks