title: TurtleBench: Evaluating Top Language Models via Real-World Yes/No
    Puzzles
summary: As the application of Large Language Models (LLMs) expands, the demand for
  reliable evaluations increases. Existing LLM evaluation benchmarks primarily
  rely on static datasets, making it challenging to assess model performance in
  dynamic interactions with users. Moreover, these benchmarks often depend on
  specific background knowledge, complicating the measurement of a model's
  logical reasoning capabilities. Other dynamic evaluation methods based on
  strong models or manual efforts may introduce biases and incur high costs and
  time demands, hindering large-scale application. To address these issues, we
  propose TurtleBench. TurtleBench collects real user guesses from our online
  Turtle Soup Puzzle platform that we developed. This approach allows for the
  relatively dynamic generation of evaluation datasets, mitigating the risk of
  model cheating while aligning assessments more closely with genuine user needs
  for reasoning capabilities, thus enhancing the reliability of evaluations.
  TurtleBench includes 1,532 user guesses along with the correctness of guesses
  after annotation. Using this dataset, we thoroughly evaluated nine of the most
  advanced LLMs available today. Notably, the OpenAI o1 series models did not
  achieve leading results in these evaluations. We propose several hypotheses for
  further research, such as "the latent reasoning of o1 utilizes trivial
  Chain-of-Thought (CoT) techniques" and "increasing CoT length not only provides
  reasoning benefits but also incurs noise costs."
paper_authors._id: 67049aca905f8970e32369ff
paper_authors.name: Qingchen Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:19:17.192Z
paper_authors._id: 6455ff584095c967f9a847bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6455ff584095c967f9a847bb/A5wjtWsudC73fLVmgASBr.jpeg
paper_authors.fullname: Qingchen Yu
paper_authors.type: user
paper_authors.user: Duguce
paper_authors._id: 67049aca905f8970e3236a00
paper_authors.name: Shichao Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:59:56.789Z
paper_authors._id: 656f339a5273668d5b946b33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656f339a5273668d5b946b33/o2nBvQiOKKP5IfDmnpHP2.jpeg
paper_authors.fullname: Shichao Song
paper_authors.type: user
paper_authors.user: Ki-Seki
paper_authors._id: 67049aca905f8970e3236a01
paper_authors.name: Ke Fang
paper_authors._id: 67049aca905f8970e3236a02
paper_authors.name: Yunfeng Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:00:20.098Z
paper_authors._id: 6695fdc326e678fb09104ff4
paper_authors.avatarUrl: /avatars/42d83f6e6018f5bdf9a0e9efed9d6ee0.svg
paper_authors.fullname: Yunfeng Shi
paper_authors.type: user
paper_authors.user: WinfredShi
paper_authors._id: 67049aca905f8970e3236a03
paper_authors.name: Zifan Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:00:25.923Z
paper_authors._id: 656f47ba2f058b368c0b1611
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656f47ba2f058b368c0b1611/mrmcmA8bxaDNUhuJQQ7T1.png
paper_authors.fullname: Zifan Zheng
paper_authors.type: user
paper_authors.user: fan2goa1
paper_authors._id: 67049aca905f8970e3236a04
paper_authors.name: Hanyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:27:25.631Z
paper_authors._id: 669e0b93c7cb0568dac6e92e
paper_authors.avatarUrl: /avatars/a39ea77d7391f164af8a80f94f85f2ca.svg
paper_authors.fullname: hanyu Wang
paper_authors.type: user
paper_authors.user: UglyToilet
paper_authors._id: 67049aca905f8970e3236a05
paper_authors.name: Simin Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:00:51.225Z
paper_authors._id: 66daea8776dbaaa372eabec5
paper_authors.avatarUrl: /avatars/1e5fbe4ff06bb6121c7029253b76b79f.svg
paper_authors.fullname: siminniu
paper_authors.type: user
paper_authors.user: siminniu
paper_authors._id: 67049aca905f8970e3236a06
paper_authors.name: Zhiyu Li
paper_title: TurtleBench: Evaluating Top Language Models via Real-World Yes/No
    Puzzles
paper_summary: As the application of Large Language Models (LLMs) expands, the demand for
  reliable evaluations increases. Existing LLM evaluation benchmarks primarily
  rely on static datasets, making it challenging to assess model performance in
  dynamic interactions with users. Moreover, these benchmarks often depend on
  specific background knowledge, complicating the measurement of a model's
  logical reasoning capabilities. Other dynamic evaluation methods based on
  strong models or manual efforts may introduce biases and incur high costs and
  time demands, hindering large-scale application. To address these issues, we
  propose TurtleBench. TurtleBench collects real user guesses from our online
  Turtle Soup Puzzle platform that we developed. This approach allows for the
  relatively dynamic generation of evaluation datasets, mitigating the risk of
  model cheating while aligning assessments more closely with genuine user needs
  for reasoning capabilities, thus enhancing the reliability of evaluations.
  TurtleBench includes 1,532 user guesses along with the correctness of guesses
  after annotation. Using this dataset, we thoroughly evaluated nine of the most
  advanced LLMs available today. Notably, the OpenAI o1 series models did not
  achieve leading results in these evaluations. We propose several hypotheses for
  further research, such as "the latent reasoning of o1 utilizes trivial
  Chain-of-Thought (CoT) techniques" and "increasing CoT length not only provides
  reasoning benefits but also incurs noise costs."
paper_ai_summary: TurtleBench uses real user interactions to dynamically evaluate LLMs' logical reasoning capabilities, addressing limitations of static benchmarks and manual efforts.
paper_ai_keywords: Large Language Models LLMs TurtleBench Turtle Soup Puzzle user guesses Chain-of-Thought CoT