TurtleBench: Evaluating Top Language Models via Real-World Yes/No
  Puzzles
As the application of Large Language Models (LLMs) expands, the demand for
reliable evaluations increases. Existing LLM evaluation benchmarks primarily
rely on static datasets, making it challenging to assess model performance in
dynamic interactions with users. Moreover, these benchmarks often depend on
specific background knowledge, complicating the measurement of a model's
logical reasoning capabilities. Other dynamic evaluation methods based on
strong models or manual efforts may introduce biases and incur high costs and
time demands, hindering large-scale application. To address these issues, we
propose TurtleBench. TurtleBench collects real user guesses from our online
Turtle Soup Puzzle platform that we developed. This approach allows for the
relatively dynamic generation of evaluation datasets, mitigating the risk of
model cheating while aligning assessments more closely with genuine user needs
for reasoning capabilities, thus enhancing the reliability of evaluations.
TurtleBench includes 1,532 user guesses along with the correctness of guesses
after annotation. Using this dataset, we thoroughly evaluated nine of the most
advanced LLMs available today. Notably, the OpenAI o1 series models did not
achieve leading results in these evaluations. We propose several hypotheses for
further research, such as "the latent reasoning of o1 utilizes trivial
Chain-of-Thought (CoT) techniques" and "increasing CoT length not only provides
reasoning benefits but also incurs noise costs."
67049aca905f8970e32369ff
Qingchen Yu
claimed_verified
2024-10-08T07:19:17.192Z
6455ff584095c967f9a847bb
https://cdn-avatars.huggingface.co/v1/production/uploads/6455ff584095c967f9a847bb/A5wjtWsudC73fLVmgASBr.jpeg
Qingchen Yu
user
Duguce
67049aca905f8970e3236a00
Shichao Song
admin_assigned
2024-10-08T09:59:56.789Z
656f339a5273668d5b946b33
https://cdn-avatars.huggingface.co/v1/production/uploads/656f339a5273668d5b946b33/o2nBvQiOKKP5IfDmnpHP2.jpeg
Shichao Song
user
Ki-Seki
67049aca905f8970e3236a01
Ke Fang
67049aca905f8970e3236a02
Yunfeng Shi
admin_assigned
2024-10-08T10:00:20.098Z
6695fdc326e678fb09104ff4
/avatars/42d83f6e6018f5bdf9a0e9efed9d6ee0.svg
Yunfeng Shi
user
WinfredShi
67049aca905f8970e3236a03
Zifan Zheng
admin_assigned
2024-10-08T10:00:25.923Z
656f47ba2f058b368c0b1611
https://cdn-avatars.huggingface.co/v1/production/uploads/656f47ba2f058b368c0b1611/mrmcmA8bxaDNUhuJQQ7T1.png
Zifan Zheng
user
fan2goa1
67049aca905f8970e3236a04
Hanyu Wang
claimed_verified
2025-02-24T09:27:25.631Z
669e0b93c7cb0568dac6e92e
/avatars/a39ea77d7391f164af8a80f94f85f2ca.svg
hanyu Wang
user
UglyToilet
67049aca905f8970e3236a05
Simin Niu
admin_assigned
2024-10-08T10:00:51.225Z
66daea8776dbaaa372eabec5
/avatars/1e5fbe4ff06bb6121c7029253b76b79f.svg
siminniu
user
siminniu
67049aca905f8970e3236a06
Zhiyu Li
TurtleBench: Evaluating Top Language Models via Real-World Yes/No
  Puzzles
As the application of Large Language Models (LLMs) expands, the demand for
reliable evaluations increases. Existing LLM evaluation benchmarks primarily
rely on static datasets, making it challenging to assess model performance in
dynamic interactions with users. Moreover, these benchmarks often depend on
specific background knowledge, complicating the measurement of a model's
logical reasoning capabilities. Other dynamic evaluation methods based on
strong models or manual efforts may introduce biases and incur high costs and
time demands, hindering large-scale application. To address these issues, we
propose TurtleBench. TurtleBench collects real user guesses from our online
Turtle Soup Puzzle platform that we developed. This approach allows for the
relatively dynamic generation of evaluation datasets, mitigating the risk of
model cheating while aligning assessments more closely with genuine user needs
for reasoning capabilities, thus enhancing the reliability of evaluations.
TurtleBench includes 1,532 user guesses along with the correctness of guesses
after annotation. Using this dataset, we thoroughly evaluated nine of the most
advanced LLMs available today. Notably, the OpenAI o1 series models did not
achieve leading results in these evaluations. We propose several hypotheses for
further research, such as "the latent reasoning of o1 utilizes trivial
Chain-of-Thought (CoT) techniques" and "increasing CoT length not only provides
reasoning benefits but also incurs noise costs."
TurtleBench uses real user interactions to dynamically evaluate LLMs' logical reasoning capabilities, addressing limitations of static benchmarks and manual efforts.
Large Language Models LLMs TurtleBench Turtle Soup Puzzle user guesses Chain-of-Thought CoT