Human or Not? A Gamified Approach to the Turing Test
We present "Human or Not?", an online game inspired by the Turing test, that
measures the capability of AI chatbots to mimic humans in dialog, and of humans
to tell bots from other humans. Over the course of a month, the game was played
by over 1.5 million users who engaged in anonymous two-minute chat sessions
with either another human or an AI language model which was prompted to behave
like humans. The task of the players was to correctly guess whether they spoke
to a person or to an AI. This largest scale Turing-style test conducted to date
revealed some interesting facts. For example, overall users guessed the
identity of their partners correctly in only 68% of the games. In the subset of
the games in which users faced an AI bot, users had even lower correct guess
rates of 60% (that is, not much higher than chance). This white paper details
the development, deployment, and results of this unique experiment. While this
experiment calls for many extensions and refinements, these findings already
begin to shed light on the inevitable near future which will commingle humans
and AI.
647805286e6c7ac608c56626
Daniel Jannai
extracted_confirmed
2023-06-01T07:41:31.797Z
62d03afd485856cd710c3377
/avatars/244598557137be1a4de4fd74d18e64ce.svg
Daniel Jannai-Epstein
user
danieljannai
647805286e6c7ac608c56627
Amos Meron
647805286e6c7ac608c56628
Barak Lenz
extracted_confirmed
2023-06-01T05:50:09.546Z
6339882037faf14d3a955b4a
/avatars/5c5105d7aabe0a29bba82a96771f1767.svg
Lenz
user
barakl
647805286e6c7ac608c56629
Yoav Levine
647805286e6c7ac608c5662a
Yoav Shoham
Human or Not? A Gamified Approach to the Turing Test
We present "Human or Not?", an online game inspired by the Turing test, that
measures the capability of AI chatbots to mimic humans in dialog, and of humans
to tell bots from other humans. Over the course of a month, the game was played
by over 1.5 million users who engaged in anonymous two-minute chat sessions
with either another human or an AI language model which was prompted to behave
like humans. The task of the players was to correctly guess whether they spoke
to a person or to an AI. This largest scale Turing-style test conducted to date
revealed some interesting facts. For example, overall users guessed the
identity of their partners correctly in only 68% of the games. In the subset of
the games in which users faced an AI bot, users had even lower correct guess
rates of 60% (that is, not much higher than chance). This white paper details
the development, deployment, and results of this unique experiment. While this
experiment calls for many extensions and refinements, these findings already
begin to shed light on the inevitable near future which will commingle humans
and AI.
A large-scale Turing-style test using an online game revealed that users had difficulty distinguishing between AI chatbots and humans.