title: PhD Knowledge Not Required: A Reasoning Challenge for Large Language
    Models
summary: Existing benchmarks for frontier models often test specialized, ``PhD-level''
  knowledge that is difficult for non-experts to grasp. In contrast, we present a
  benchmark based on the NPR Sunday Puzzle Challenge that requires only general
  knowledge. Our benchmark is challenging for both humans and models, however
  correct solutions are easy to verify, and models' mistakes are easy to spot.
    Our work reveals capability gaps that are not evident in existing benchmarks:
  OpenAI o1 significantly outperforms other reasoning models that are on par on
  benchmarks that test specialized knowledge. Furthermore, our analysis of
  reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance,
  often concedes with ``I give up'' before providing an answer that it knows is
  wrong. R1 can also be remarkably ``uncertain'' in its output and in rare cases,
  it does not ``finish thinking,'' which suggests the need for an inference-time
  technique to ``wrap up'' before the context window limit is reached. We also
  quantify the effectiveness of reasoning longer with R1 and Gemini Thinking to
  identify the point beyond which more reasoning is unlikely to improve accuracy
  on our benchmark.
paper_authors._id: 67a1e658a68ad21bcdffead6
paper_authors.name: Carolyn Jane Anderson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:09:21.263Z
paper_authors._id: 6243199444c9c3b21be74c50
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6243199444c9c3b21be74c50/uxnj03NBcV_TeC3eV2U-Q.jpeg
paper_authors.fullname: Carolyn Anderson
paper_authors.type: user
paper_authors.user: canders1
paper_authors._id: 67a1e658a68ad21bcdffead7
paper_authors.name: Joydeep Biswas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:09:28.567Z
paper_authors._id: 641e5fd25f274a0a92c30b7a
paper_authors.avatarUrl: /avatars/192ef72795a032f3c73950143a13f6b9.svg
paper_authors.fullname: Joydeep Biswas
paper_authors.type: user
paper_authors.user: joydeep-b
paper_authors._id: 67a1e658a68ad21bcdffead8
paper_authors.name: Aleksander Boruch-Gruszecki
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:09:34.032Z
paper_authors._id: 67508d980b12dc1a51ea5e59
paper_authors.avatarUrl: /avatars/ec29de1d231a93af18c279fcd2ebbd0b.svg
paper_authors.fullname: Aleksander Boruch-Gruszecki
paper_authors.type: user
paper_authors.user: abgruszecki
paper_authors._id: 67a1e658a68ad21bcdffead9
paper_authors.name: Federico Cassano
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:09:39.635Z
paper_authors._id: 642ca13cc3684e5a4e806661
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642ca13cc3684e5a4e806661/ELhdKsK429zi4wZkVkx9Y.jpeg
paper_authors.fullname: Federico Cassano
paper_authors.type: user
paper_authors.user: cassanof
paper_authors._id: 67a1e658a68ad21bcdffeada
paper_authors.name: Molly Q Feldman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:09:48.494Z
paper_authors._id: 644c34858c51ddbe0ea78cb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644c34858c51ddbe0ea78cb9/7BREj4M1LR4LAGk4-WS4G.jpeg
paper_authors.fullname: Molly Feldman
paper_authors.type: user
paper_authors.user: feldmanmolly
paper_authors._id: 67a1e658a68ad21bcdffeadb
paper_authors.name: Arjun Guha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:09:54.609Z
paper_authors._id: 62d8315bad693a1a962864b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664332914111-62d8315bad693a1a962864b3.png
paper_authors.fullname: Arjun Guha
paper_authors.type: user
paper_authors.user: arjunguha
paper_authors._id: 67a1e658a68ad21bcdffeadc
paper_authors.name: Francesca Lucchetti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:10:03.253Z
paper_authors._id: 64025d945caf6d21d67cdad2
paper_authors.avatarUrl: /avatars/17227ea5bd07c820f4f3fd29ffa5853e.svg
paper_authors.fullname: Francesca Lucchetti
paper_authors.type: user
paper_authors.user: franlucc
paper_authors._id: 67a1e658a68ad21bcdffeadd
paper_authors.name: Zixuan Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:10:34.894Z
paper_authors._id: 6634fdc967cbac200e103bd7
paper_authors.avatarUrl: /avatars/47bbdbc066055f25736d7e1cca928b1f.svg
paper_authors.fullname: Zixuan Wu
paper_authors.type: user
paper_authors.user: AryaWu
paper_title: PhD Knowledge Not Required: A Reasoning Challenge for Large Language
    Models
paper_summary: Existing benchmarks for frontier models often test specialized, ``PhD-level''
  knowledge that is difficult for non-experts to grasp. In contrast, we present a
  benchmark based on the NPR Sunday Puzzle Challenge that requires only general
  knowledge. Our benchmark is challenging for both humans and models, however
  correct solutions are easy to verify, and models' mistakes are easy to spot.
    Our work reveals capability gaps that are not evident in existing benchmarks:
  OpenAI o1 significantly outperforms other reasoning models that are on par on
  benchmarks that test specialized knowledge. Furthermore, our analysis of
  reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance,
  often concedes with ``I give up'' before providing an answer that it knows is
  wrong. R1 can also be remarkably ``uncertain'' in its output and in rare cases,
  it does not ``finish thinking,'' which suggests the need for an inference-time
  technique to ``wrap up'' before the context window limit is reached. We also
  quantify the effectiveness of reasoning longer with R1 and Gemini Thinking to
  identify the point beyond which more reasoning is unlikely to improve accuracy
  on our benchmark.
paper_ai_summary: A new benchmark based on the NPR Sunday Puzzle Challenge tests general knowledge and uncovers capability gaps and new failure modes in reasoning models.
paper_ai_keywords: reasoning models OpenAI o1 DeepSeek R1 inference-time technique reason longer Gemini Thinking context window limit