Replacing thinking with tool usage enables reasoning in small language
  models
Recent advances have established a new machine learning paradigm based on
scaling up compute at inference time as well as at training time. In that line
of work, a combination of Supervised Fine-Tuning (SFT) on synthetic
demonstrations and Reinforcement Learning with Verifiable Rewards (RLVR) is
used for training Large Language Models to expend extra compute during
inference in the form of "thoughts" expressed in natural language. In this
paper, we propose to instead format these tokens as a multi-turn interaction
trace with a stateful tool. At each turn, the new state of the tool is appended
to the context of the model, whose job is to generate the tokens necessary to
control the tool via a custom DSL. We benchmark this approach on the problem of
repairing malfunctioning Python code, and show that this constrained setup
allows for faster sampling of experience and a denser reward signal, allowing
even models of size up to 3B parameters to learn how to proficiently expend
additional compute on the task.
68776e57ff8f47a7f86442bd
Corrado Rainone
claimed_verified
2025-07-16T15:05:55.722Z
669e707ec517d804cfce91c5
/avatars/fc18e64e100cbf28763db04b44242747.svg
Corrado Rainone
user
crainone
68776e57ff8f47a7f86442be
Tim Bakker
68776e57ff8f47a7f86442bf
Roland Memisevic
Replacing thinking with tool usage enables reasoning in small language
  models
Recent advances have established a new machine learning paradigm based on
scaling up compute at inference time as well as at training time. In that line
of work, a combination of Supervised Fine-Tuning (SFT) on synthetic
demonstrations and Reinforcement Learning with Verifiable Rewards (RLVR) is
used for training Large Language Models to expend extra compute during
inference in the form of "thoughts" expressed in natural language. In this
paper, we propose to instead format these tokens as a multi-turn interaction
trace with a stateful tool. At each turn, the new state of the tool is appended
to the context of the model, whose job is to generate the tokens necessary to
control the tool via a custom DSL. We benchmark this approach on the problem of
repairing malfunctioning Python code, and show that this constrained setup
allows for faster sampling of experience and a denser reward signal, allowing
even models of size up to 3B parameters to learn how to proficiently expend
additional compute on the task.
A new approach formats tokens as a multi-turn interaction trace with a stateful tool for training Large Language Models, enabling faster sampling and denser reward signals for tasks like repairing Python code.
Supervised Fine-Tuning Reinforcement Learning with Verifiable Rewards Large Language Models multi-turn interaction trace stateful tool custom DSL