title: Evaluating Intelligence via Trial and Error
summary: Intelligence is a crucial trait for species to find solutions within a
  limited number of trial-and-error attempts. Building on this idea, we introduce
  Survival Game as a framework to evaluate intelligence based on the number of
  failed attempts in a trial-and-error process. Fewer failures indicate higher
  intelligence. When the expectation and variance of failure counts are both
  finite, it signals the ability to consistently find solutions to new
  challenges, which we define as the Autonomous Level of intelligence. Using
  Survival Game, we comprehensively evaluate existing AI systems. Our results
  show that while AI systems achieve the Autonomous Level in simple tasks, they
  are still far from it in more complex tasks, such as vision, search,
  recommendation, and language. While scaling current AI technologies might help,
  this would come at an astronomical cost. Projections suggest that achieving the
  Autonomous Level for general tasks would require 10^{26} parameters. To put
  this into perspective, loading such a massive model requires so many H100 GPUs
  that their total value is 10^{7} times that of Apple Inc.'s market value.
  Even with Moore's Law, supporting such a parameter scale would take 70 years.
  This staggering cost highlights the complexity of human tasks and the
  inadequacies of current AI technologies. To further investigate this
  phenomenon, we conduct a theoretical analysis of Survival Game and its
  experimental results. Our findings suggest that human tasks possess a
  criticality property. As a result, Autonomous Level requires a deep
  understanding of the task's underlying mechanisms. Current AI systems, however,
  do not fully grasp these mechanisms and instead rely on superficial mimicry,
  making it difficult for them to reach an autonomous level. We believe Survival
  Game can not only guide the future development of AI but also offer profound
  insights into human intelligence.
paper_authors._id: 67d1080b2264403cbf36b0ad
paper_authors.name: Jingtao Zhan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T10:45:08.742Z
paper_authors._id: 62f8b1e7c4817cfc07532fbf
paper_authors.avatarUrl: /avatars/b21661f868fd968c0ad0910da2a04b65.svg
paper_authors.fullname: Jingtao Zhan
paper_authors.type: user
paper_authors.user: jingtao
paper_authors._id: 67d1080b2264403cbf36b0ae
paper_authors.name: Jiahao Zhao
paper_authors._id: 67d1080b2264403cbf36b0af
paper_authors.name: Jiayu Li
paper_authors._id: 67d1080b2264403cbf36b0b0
paper_authors.name: Yiqun Liu
paper_authors._id: 67d1080b2264403cbf36b0b1
paper_authors.name: Bo Zhang
paper_authors._id: 67d1080b2264403cbf36b0b2
paper_authors.name: Qingyao Ai
paper_authors._id: 67d1080b2264403cbf36b0b3
paper_authors.name: Jiaxin Mao
paper_authors._id: 67d1080b2264403cbf36b0b4
paper_authors.name: Hongning Wang
paper_authors._id: 67d1080b2264403cbf36b0b5
paper_authors.name: Min Zhang
paper_authors._id: 67d1080b2264403cbf36b0b6
paper_authors.name: Shaoping Ma
paper_title: Evaluating Intelligence via Trial and Error
paper_summary: Intelligence is a crucial trait for species to find solutions within a
  limited number of trial-and-error attempts. Building on this idea, we introduce
  Survival Game as a framework to evaluate intelligence based on the number of
  failed attempts in a trial-and-error process. Fewer failures indicate higher
  intelligence. When the expectation and variance of failure counts are both
  finite, it signals the ability to consistently find solutions to new
  challenges, which we define as the Autonomous Level of intelligence. Using
  Survival Game, we comprehensively evaluate existing AI systems. Our results
  show that while AI systems achieve the Autonomous Level in simple tasks, they
  are still far from it in more complex tasks, such as vision, search,
  recommendation, and language. While scaling current AI technologies might help,
  this would come at an astronomical cost. Projections suggest that achieving the
  Autonomous Level for general tasks would require 10^{26} parameters. To put
  this into perspective, loading such a massive model requires so many H100 GPUs
  that their total value is 10^{7} times that of Apple Inc.'s market value.
  Even with Moore's Law, supporting such a parameter scale would take 70 years.
  This staggering cost highlights the complexity of human tasks and the
  inadequacies of current AI technologies. To further investigate this
  phenomenon, we conduct a theoretical analysis of Survival Game and its
  experimental results. Our findings suggest that human tasks possess a
  criticality property. As a result, Autonomous Level requires a deep
  understanding of the task's underlying mechanisms. Current AI systems, however,
  do not fully grasp these mechanisms and instead rely on superficial mimicry,
  making it difficult for them to reach an autonomous level. We believe Survival
  Game can not only guide the future development of AI but also offer profound
  insights into human intelligence.
paper_ai_summary: Survival Game evaluates AI intelligence via failure counts in trial-and-error processes, highlighting the challenge of achieving Autonomous Level in complex tasks and the critical need for deep task understanding.
paper_ai_keywords: Autonomous Level Survival Game criticality property