Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on
  Elementary School-Level Reasoning Problems?
The rapid escalation from elementary school-level to frontier problems of the
difficulty for LLM benchmarks in recent years have weaved a miracle for
researchers that we are only inches away from surpassing human intelligence.
However, is the LLMs' remarkable reasoning ability indeed comes from true
intelligence by human standards, or are they simply reciting solutions
witnessed during training at an Internet level? To study this problem, we
propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's
recitation behavior when asked simple reasoning problems but with conditions
subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly,
we found existing cutting-edge LLMs unanimously exhibits extremely severe
recitation behavior; by changing one phrase in the condition, top models such
as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary
school-level arithmetic and reasoning problems. Such findings are a wake-up
call to the LLM community that compels us to re-evaluate the true intelligence
level of cutting-edge LLMs.
67ecae49a34baf018ca3c4cd
Kai Yan
extracted_pending
2025-04-02T03:26:02.443Z
65de7628deee79773f0f46f6
/avatars/6c509dbe96e47b47271eb74178c1c9ba.svg
Kai Yan
user
kaiyan289
67ecae49a34baf018ca3c4ce
Yufei Xu
claimed_verified
2025-04-02T08:22:23.307Z
642957104e073875f6a5ddd0
/avatars/32b706d35c5ff52932c5029b94caa7b9.svg
Yufei Xu
user
yfxu
67ecae49a34baf018ca3c4cf
Zhengyin Du
admin_assigned
2025-04-02T10:17:39.401Z
67978efb9d331acb4022b3c9
/avatars/9b6f3f75b10820284ada99ab970d6e96.svg
DuzHENGYIN
user
ALEXoDu
67ecae49a34baf018ca3c4d0
Xuesong Yao
claimed_verified
2025-04-03T08:28:48.385Z
656d8ec3fa91c81609822b96
/avatars/56f6eed2cdd4706db45ea6b20059265d.svg
Yao Xuesong
user
NathanYao
67ecae49a34baf018ca3c4d1
Zheyu Wang
67ecae49a34baf018ca3c4d2
Xiaowen Guo
67ecae49a34baf018ca3c4d3
Jiecao Chen
admin_assigned
2025-04-02T10:16:06.617Z
66df70fe5a0c5910d663160d
/avatars/980ca32bd0049ef5bbf002e7dc9f911c.svg
jiecao.chen
user
xmerge123
Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on
  Elementary School-Level Reasoning Problems?
The rapid escalation from elementary school-level to frontier problems of the
difficulty for LLM benchmarks in recent years have weaved a miracle for
researchers that we are only inches away from surpassing human intelligence.
However, is the LLMs' remarkable reasoning ability indeed comes from true
intelligence by human standards, or are they simply reciting solutions
witnessed during training at an Internet level? To study this problem, we
propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's
recitation behavior when asked simple reasoning problems but with conditions
subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly,
we found existing cutting-edge LLMs unanimously exhibits extremely severe
recitation behavior; by changing one phrase in the condition, top models such
as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary
school-level arithmetic and reasoning problems. Such findings are a wake-up
call to the LLM community that compels us to re-evaluate the true intelligence
level of cutting-edge LLMs.
Existing large language models exhibit significant recitation behavior when conditions are subtly altered, suggesting they may not possess human-like intelligence.
LLM benchmarks RoR-Bench multi-modal benchmark recitation behavior LLMs OpenAI-o1 DeepSeek-R1 elementary school-level arithmetic reasoning problems