title: The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI
    Quiz
summary: This research introduces a novel evaluation framework designed to assess
  large language models' (LLMs) ability to acknowledge uncertainty on 675
  fundamentally unsolvable problems. Using a curated dataset of graduate-level
  grand challenge questions with intentionally unknowable answers, we evaluated
  twelve state-of-the-art LLMs, including both open and closed-source models, on
  their propensity to admit ignorance rather than generate plausible but
  incorrect responses. The best models scored in 62-68% accuracy ranges for
  admitting the problem solution was unknown in fields ranging from biology to
  philosophy and mathematics. We observed an inverse relationship between problem
  difficulty and model accuracy, with GPT-4 demonstrating higher rates of
  uncertainty acknowledgment on more challenging problems (35.8%) compared to
  simpler ones (20.0%). This pattern indicates that models may be more prone to
  generate speculative answers when problems appear more tractable. The study
  also revealed significant variations across problem categories, with models
  showing difficulty in acknowledging uncertainty in invention and NP-hard
  problems while performing relatively better on philosophical and psychological
  challenges. These results contribute to the growing body of research on
  artificial general intelligence (AGI) assessment by highlighting the importance
  of uncertainty recognition as a critical component of future machine
  intelligence evaluation. This impossibility test thus extends previous
  theoretical frameworks for universal intelligence testing by providing
  empirical evidence of current limitations in LLMs' ability to recognize their
  own knowledge boundaries, suggesting new directions for improving model
  training architectures and evaluation approaches.
paper_authors._id: 67453112edf0281a265ff746
paper_authors.name: David Noever
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:58:35.071Z
paper_authors._id: 63136a82e29fb2e86d5e5bdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63136a82e29fb2e86d5e5bdd/pFZDuQtzfUStovbwwZGvn.png
paper_authors.fullname: David Noever
paper_authors.type: user
paper_authors.user: dnoever
paper_authors._id: 67453112edf0281a265ff747
paper_authors.name: Forrest McKee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:58:40.641Z
paper_authors._id: 627bff703974b0ed6b2c340d
paper_authors.avatarUrl: /avatars/12518ccef145f343548b3fd7be5efddc.svg
paper_authors.fullname: Forrest McKee
paper_authors.type: user
paper_authors.user: fgmckee
paper_title: The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI
    Quiz
paper_summary: This research introduces a novel evaluation framework designed to assess
  large language models' (LLMs) ability to acknowledge uncertainty on 675
  fundamentally unsolvable problems. Using a curated dataset of graduate-level
  grand challenge questions with intentionally unknowable answers, we evaluated
  twelve state-of-the-art LLMs, including both open and closed-source models, on
  their propensity to admit ignorance rather than generate plausible but
  incorrect responses. The best models scored in 62-68% accuracy ranges for
  admitting the problem solution was unknown in fields ranging from biology to
  philosophy and mathematics. We observed an inverse relationship between problem
  difficulty and model accuracy, with GPT-4 demonstrating higher rates of
  uncertainty acknowledgment on more challenging problems (35.8%) compared to
  simpler ones (20.0%). This pattern indicates that models may be more prone to
  generate speculative answers when problems appear more tractable. The study
  also revealed significant variations across problem categories, with models
  showing difficulty in acknowledging uncertainty in invention and NP-hard
  problems while performing relatively better on philosophical and psychological
  challenges. These results contribute to the growing body of research on
  artificial general intelligence (AGI) assessment by highlighting the importance
  of uncertainty recognition as a critical component of future machine
  intelligence evaluation. This impossibility test thus extends previous
  theoretical frameworks for universal intelligence testing by providing
  empirical evidence of current limitations in LLMs' ability to recognize their
  own knowledge boundaries, suggesting new directions for improving model
  training architectures and evaluation approaches.
paper_ai_summary: Evaluation framework assesses LLMs' uncertainty acknowledgment on unsolvable problems, revealing variations in performance across different fields and difficulties.
paper_ai_keywords: large language models LLMs uncertainty acknowledgment graduate-level questions inverse relationship problem difficulty GPT-4 invention NP-hard problems philosophical challenges artificial general intelligence AGI