title: Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on
    Elementary School-Level Reasoning Problems?
summary: The rapid escalation from elementary school-level to frontier problems of the
  difficulty for LLM benchmarks in recent years have weaved a miracle for
  researchers that we are only inches away from surpassing human intelligence.
  However, is the LLMs' remarkable reasoning ability indeed comes from true
  intelligence by human standards, or are they simply reciting solutions
  witnessed during training at an Internet level? To study this problem, we
  propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's
  recitation behavior when asked simple reasoning problems but with conditions
  subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly,
  we found existing cutting-edge LLMs unanimously exhibits extremely severe
  recitation behavior; by changing one phrase in the condition, top models such
  as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary
  school-level arithmetic and reasoning problems. Such findings are a wake-up
  call to the LLM community that compels us to re-evaluate the true intelligence
  level of cutting-edge LLMs.
paper_authors._id: 67ecae49a34baf018ca3c4cd
paper_authors.name: Kai Yan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-02T03:26:02.443Z
paper_authors._id: 65de7628deee79773f0f46f6
paper_authors.avatarUrl: /avatars/6c509dbe96e47b47271eb74178c1c9ba.svg
paper_authors.fullname: Kai Yan
paper_authors.type: user
paper_authors.user: kaiyan289
paper_authors._id: 67ecae49a34baf018ca3c4ce
paper_authors.name: Yufei Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:22:23.307Z
paper_authors._id: 642957104e073875f6a5ddd0
paper_authors.avatarUrl: /avatars/32b706d35c5ff52932c5029b94caa7b9.svg
paper_authors.fullname: Yufei Xu
paper_authors.type: user
paper_authors.user: yfxu
paper_authors._id: 67ecae49a34baf018ca3c4cf
paper_authors.name: Zhengyin Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T10:17:39.401Z
paper_authors._id: 67978efb9d331acb4022b3c9
paper_authors.avatarUrl: /avatars/9b6f3f75b10820284ada99ab970d6e96.svg
paper_authors.fullname: DuzHENGYIN
paper_authors.type: user
paper_authors.user: ALEXoDu
paper_authors._id: 67ecae49a34baf018ca3c4d0
paper_authors.name: Xuesong Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:28:48.385Z
paper_authors._id: 656d8ec3fa91c81609822b96
paper_authors.avatarUrl: /avatars/56f6eed2cdd4706db45ea6b20059265d.svg
paper_authors.fullname: Yao Xuesong
paper_authors.type: user
paper_authors.user: NathanYao
paper_authors._id: 67ecae49a34baf018ca3c4d1
paper_authors.name: Zheyu Wang
paper_authors._id: 67ecae49a34baf018ca3c4d2
paper_authors.name: Xiaowen Guo
paper_authors._id: 67ecae49a34baf018ca3c4d3
paper_authors.name: Jiecao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T10:16:06.617Z
paper_authors._id: 66df70fe5a0c5910d663160d
paper_authors.avatarUrl: /avatars/980ca32bd0049ef5bbf002e7dc9f911c.svg
paper_authors.fullname: jiecao.chen
paper_authors.type: user
paper_authors.user: xmerge123
paper_title: Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on
    Elementary School-Level Reasoning Problems?
paper_summary: The rapid escalation from elementary school-level to frontier problems of the
  difficulty for LLM benchmarks in recent years have weaved a miracle for
  researchers that we are only inches away from surpassing human intelligence.
  However, is the LLMs' remarkable reasoning ability indeed comes from true
  intelligence by human standards, or are they simply reciting solutions
  witnessed during training at an Internet level? To study this problem, we
  propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's
  recitation behavior when asked simple reasoning problems but with conditions
  subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly,
  we found existing cutting-edge LLMs unanimously exhibits extremely severe
  recitation behavior; by changing one phrase in the condition, top models such
  as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary
  school-level arithmetic and reasoning problems. Such findings are a wake-up
  call to the LLM community that compels us to re-evaluate the true intelligence
  level of cutting-edge LLMs.
paper_ai_summary: Existing large language models exhibit significant recitation behavior when conditions are subtly altered, suggesting they may not possess human-like intelligence.
paper_ai_keywords: LLM benchmarks RoR-Bench multi-modal benchmark recitation behavior LLMs OpenAI-o1 DeepSeek-R1 elementary school-level arithmetic reasoning problems