PhD Knowledge Not Required: A Reasoning Challenge for Large Language
  Models
Existing benchmarks for frontier models often test specialized, ``PhD-level''
knowledge that is difficult for non-experts to grasp. In contrast, we present a
benchmark based on the NPR Sunday Puzzle Challenge that requires only general
knowledge. Our benchmark is challenging for both humans and models, however
correct solutions are easy to verify, and models' mistakes are easy to spot.
  Our work reveals capability gaps that are not evident in existing benchmarks:
OpenAI o1 significantly outperforms other reasoning models that are on par on
benchmarks that test specialized knowledge. Furthermore, our analysis of
reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance,
often concedes with ``I give up'' before providing an answer that it knows is
wrong. R1 can also be remarkably ``uncertain'' in its output and in rare cases,
it does not ``finish thinking,'' which suggests the need for an inference-time
technique to ``wrap up'' before the context window limit is reached. We also
quantify the effectiveness of reasoning longer with R1 and Gemini Thinking to
identify the point beyond which more reasoning is unlikely to improve accuracy
on our benchmark.
67a1e658a68ad21bcdffead6
Carolyn Jane Anderson
admin_assigned
2025-02-04T21:09:21.263Z
6243199444c9c3b21be74c50
https://cdn-avatars.huggingface.co/v1/production/uploads/6243199444c9c3b21be74c50/uxnj03NBcV_TeC3eV2U-Q.jpeg
Carolyn Anderson
user
canders1
67a1e658a68ad21bcdffead7
Joydeep Biswas
admin_assigned
2025-02-04T21:09:28.567Z
641e5fd25f274a0a92c30b7a
/avatars/192ef72795a032f3c73950143a13f6b9.svg
Joydeep Biswas
user
joydeep-b
67a1e658a68ad21bcdffead8
Aleksander Boruch-Gruszecki
admin_assigned
2025-02-04T21:09:34.032Z
67508d980b12dc1a51ea5e59
/avatars/ec29de1d231a93af18c279fcd2ebbd0b.svg
Aleksander Boruch-Gruszecki
user
abgruszecki
67a1e658a68ad21bcdffead9
Federico Cassano
admin_assigned
2025-02-04T21:09:39.635Z
642ca13cc3684e5a4e806661
https://cdn-avatars.huggingface.co/v1/production/uploads/642ca13cc3684e5a4e806661/ELhdKsK429zi4wZkVkx9Y.jpeg
Federico Cassano
user
cassanof
67a1e658a68ad21bcdffeada
Molly Q Feldman
admin_assigned
2025-02-04T21:09:48.494Z
644c34858c51ddbe0ea78cb9
https://cdn-avatars.huggingface.co/v1/production/uploads/644c34858c51ddbe0ea78cb9/7BREj4M1LR4LAGk4-WS4G.jpeg
Molly Feldman
user
feldmanmolly
67a1e658a68ad21bcdffeadb
Arjun Guha
admin_assigned
2025-02-04T21:09:54.609Z
62d8315bad693a1a962864b3
https://cdn-avatars.huggingface.co/v1/production/uploads/1664332914111-62d8315bad693a1a962864b3.png
Arjun Guha
user
arjunguha
67a1e658a68ad21bcdffeadc
Francesca Lucchetti
admin_assigned
2025-02-04T21:10:03.253Z
64025d945caf6d21d67cdad2
/avatars/17227ea5bd07c820f4f3fd29ffa5853e.svg
Francesca Lucchetti
user
franlucc
67a1e658a68ad21bcdffeadd
Zixuan Wu
admin_assigned
2025-02-04T21:10:34.894Z
6634fdc967cbac200e103bd7
/avatars/47bbdbc066055f25736d7e1cca928b1f.svg
Zixuan Wu
user
AryaWu
PhD Knowledge Not Required: A Reasoning Challenge for Large Language
  Models
Existing benchmarks for frontier models often test specialized, ``PhD-level''
knowledge that is difficult for non-experts to grasp. In contrast, we present a
benchmark based on the NPR Sunday Puzzle Challenge that requires only general
knowledge. Our benchmark is challenging for both humans and models, however
correct solutions are easy to verify, and models' mistakes are easy to spot.
  Our work reveals capability gaps that are not evident in existing benchmarks:
OpenAI o1 significantly outperforms other reasoning models that are on par on
benchmarks that test specialized knowledge. Furthermore, our analysis of
reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance,
often concedes with ``I give up'' before providing an answer that it knows is
wrong. R1 can also be remarkably ``uncertain'' in its output and in rare cases,
it does not ``finish thinking,'' which suggests the need for an inference-time
technique to ``wrap up'' before the context window limit is reached. We also
quantify the effectiveness of reasoning longer with R1 and Gemini Thinking to
identify the point beyond which more reasoning is unlikely to improve accuracy
on our benchmark.
A new benchmark based on the NPR Sunday Puzzle Challenge tests general knowledge and uncovers capability gaps and new failure modes in reasoning models.
reasoning models OpenAI o1 DeepSeek R1 inference-time technique reason longer Gemini Thinking context window limit