title: Kinetics: Rethinking Test-Time Scaling Laws
summary: We rethink test-time scaling laws from a practical efficiency perspective,
  revealing that the effectiveness of smaller models is significantly
  overestimated. Prior work, grounded in compute-optimality, overlooks critical
  memory access bottlenecks introduced by inference-time strategies (e.g.,
  Best-of-N, long CoTs). Our holistic analysis, spanning models from 0.6B to
  32B parameters, reveals a new Kinetics Scaling Law that better guides resource
  allocation by incorporating both computation and memory access costs. Kinetics
  Scaling Law suggests that test-time compute is more effective when used on
  models above a threshold than smaller ones. A key reason is that in TTS,
  attention, rather than parameter count, emerges as the dominant cost factor.
  Motivated by this, we propose a new scaling paradigm centered on sparse
  attention, which lowers per-token cost and enables longer generations and more
  parallel samples within the same resource budget. Empirically, we show that
  sparse attention models consistently outperform dense counterparts, achieving
  over 60 points gains in low-cost regimes and over 5 points gains in high-cost
  regimes for problem-solving accuracy on AIME, encompassing evaluations on
  state-of-the-art MoEs. These results suggest that sparse attention is essential
  for realizing the full potential of test-time scaling because, unlike training,
  where parameter scaling saturates, test-time accuracy continues to improve
  through increased generation. The code is available at
  https://github.com/Infini-AI-Lab/Kinetics.
paper_authors._id: 68426a35164f4e14a57cbed9
paper_authors.name: Ranajoy Sadhukhan
paper_authors._id: 68426a35164f4e14a57cbeda
paper_authors.name: Zhuoming Chen
paper_authors._id: 68426a35164f4e14a57cbedb
paper_authors.name: Haizhong Zheng
paper_authors._id: 68426a35164f4e14a57cbedc
paper_authors.name: Yang Zhou
paper_authors._id: 68426a35164f4e14a57cbedd
paper_authors.name: Emma Strubell
paper_authors._id: 68426a35164f4e14a57cbede
paper_authors.name: Beidi Chen
paper_title: Kinetics: Rethinking Test-Time Scaling Laws
paper_summary: We rethink test-time scaling laws from a practical efficiency perspective,
  revealing that the effectiveness of smaller models is significantly
  overestimated. Prior work, grounded in compute-optimality, overlooks critical
  memory access bottlenecks introduced by inference-time strategies (e.g.,
  Best-of-N, long CoTs). Our holistic analysis, spanning models from 0.6B to
  32B parameters, reveals a new Kinetics Scaling Law that better guides resource
  allocation by incorporating both computation and memory access costs. Kinetics
  Scaling Law suggests that test-time compute is more effective when used on
  models above a threshold than smaller ones. A key reason is that in TTS,
  attention, rather than parameter count, emerges as the dominant cost factor.
  Motivated by this, we propose a new scaling paradigm centered on sparse
  attention, which lowers per-token cost and enables longer generations and more
  parallel samples within the same resource budget. Empirically, we show that
  sparse attention models consistently outperform dense counterparts, achieving
  over 60 points gains in low-cost regimes and over 5 points gains in high-cost
  regimes for problem-solving accuracy on AIME, encompassing evaluations on
  state-of-the-art MoEs. These results suggest that sparse attention is essential
  for realizing the full potential of test-time scaling because, unlike training,
  where parameter scaling saturates, test-time accuracy continues to improve
  through increased generation. The code is available at
  https://github.com/Infini-AI-Lab/Kinetics.
paper_ai_summary: Inference with small models is less efficient due to memory bottlenecks, leading to a new Kinetics Scaling Law emphasizing sparse attention for better test-time performance.
paper_ai_keywords: test-time scaling laws Best-of-$N$ long CoTs Kinetics Scaling Law computation memory access costs sparse attention dense attention parameter count problem-solving accuracy AIME MoEs generation