REST: Stress Testing Large Reasoning Models by Asking Multiple Problems
  at Once
Recent Large Reasoning Models (LRMs) have achieved remarkable progress on
task-specific benchmarks, yet their evaluation methods remain constrained by
isolated problem-solving paradigms. Existing benchmarks predominantly assess
single-question reasoning through sequential testing, resulting critical
limitations: (1) vulnerability to data contamination and less challenging
(e.g., DeepSeek-R1 achieves 97.0% on MATH500), forcing costly and perpetual
creation of new questions with large human efforts, (2) failure to evaluate
models under multi-context pressure, a key requirement for real-world
deployment. To bridge this gap, we present REST (Reasoning Evaluation through
Simultaneous Testing), a stress-testing framework that concurrently exposes
LRMs to multiple problems simultaneously. Beyond basic reasoning, REST
specifically evaluates several under-tested capabilities: contextual priority
allocation, cross-problem interference resistance, and dynamic cognitive load
management. Our evaluation reveals several striking findings: Even
state-of-the-art (SOTA) models like DeepSeek-R1 exhibit substantial performance
degradation under stress testing. Crucially, REST demonstrates stronger
discriminative power than existing benchmarks, revealing pronounced performance
differences among models that exhibit similar, near-ceiling performance under
single-question evaluations. Some key mechanistic insights emerge from our
analysis: (1) the "overthinking trap" is a critical factor contributing to the
performance degradation; (2) the models trained with "long2short" technique
preserve more accuracy of their single-problem performance under REST,
outperforming standard-trained counterparts. These results establish REST as a
cost-efficient, future-proof evaluation paradigm that better reflects
real-world reasoning demands while reducing reliance on continuous human
annotation.
6875e5f0257d4f04353705de
Zhuoshi Pan
claimed_verified
2025-07-15T19:09:22.893Z
6565e44b9bf6665f10016213
/avatars/0d7cf0e1b42cd116960cd030478446c5.svg
Zhuoshi Pan
user
panzs19
6875e5f0257d4f04353705df
Qizhi Pei
6875e5f0257d4f04353705e0
Yu Li
6875e5f0257d4f04353705e1
Qiyao Sun
6875e5f0257d4f04353705e2
Zinan Tang
claimed_verified
2025-07-15T19:09:25.116Z
66580d3d80ee5b1e11a94e57
https://cdn-avatars.huggingface.co/v1/production/uploads/66580d3d80ee5b1e11a94e57/aHaPqrV5vNefFktYRsiGf.jpeg
Zinan Tang
user
Word2Li
6875e5f0257d4f04353705e3
H. Vicky Zhao
6875e5f0257d4f04353705e4
Conghui He
6875e5f0257d4f04353705e5
Lijun Wu
REST: Stress Testing Large Reasoning Models by Asking Multiple Problems
  at Once
Recent Large Reasoning Models (LRMs) have achieved remarkable progress on
task-specific benchmarks, yet their evaluation methods remain constrained by
isolated problem-solving paradigms. Existing benchmarks predominantly assess
single-question reasoning through sequential testing, resulting critical
limitations: (1) vulnerability to data contamination and less challenging
(e.g., DeepSeek-R1 achieves 97.0% on MATH500), forcing costly and perpetual
creation of new questions with large human efforts, (2) failure to evaluate
models under multi-context pressure, a key requirement for real-world
deployment. To bridge this gap, we present REST (Reasoning Evaluation through
Simultaneous Testing), a stress-testing framework that concurrently exposes
LRMs to multiple problems simultaneously. Beyond basic reasoning, REST
specifically evaluates several under-tested capabilities: contextual priority
allocation, cross-problem interference resistance, and dynamic cognitive load
management. Our evaluation reveals several striking findings: Even
state-of-the-art (SOTA) models like DeepSeek-R1 exhibit substantial performance
degradation under stress testing. Crucially, REST demonstrates stronger
discriminative power than existing benchmarks, revealing pronounced performance
differences among models that exhibit similar, near-ceiling performance under
single-question evaluations. Some key mechanistic insights emerge from our
analysis: (1) the "overthinking trap" is a critical factor contributing to the
performance degradation; (2) the models trained with "long2short" technique
preserve more accuracy of their single-problem performance under REST,
outperforming standard-trained counterparts. These results establish REST as a
cost-efficient, future-proof evaluation paradigm that better reflects
real-world reasoning demands while reducing reliance on continuous human
annotation.
REST, a stress-testing framework, evaluates large reasoning models under simultaneous multi-problem conditions, revealing performance differences and insights into model behavior under real-world reasoning demands.
Large Reasoning Models REST stress-testing framework contextual priority allocation cross-problem interference resistance dynamic cognitive load management overthinking trap long2short technique