StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs
<|reference_start|>StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs: Despite having powerful reasoning and inference capabilities, Large Language Models (LLMs) still need external tools to acquire real-time information retrieval or domain-specific expertise to solve complex tasks, which is referred to as tool learning. Existing tool learning methods primarily rely on tuning with expert trajectories, focusing on token-sequence learning from a linguistic perspective. However, there are several challenges: 1) imitating static trajectories limits their ability to generalize to new tasks. 2) even expert trajectories can be suboptimal, and better solution paths may exist. In this work, we introduce StepTool, a novel step-grained reinforcement learning framework to improve tool learning in LLMs. It consists of two components: Step-grained Reward Shaping, which assigns rewards at each tool interaction based on tool invocation success and its contribution to the task, and Step-grained Optimization, which uses policy gradient methods to optimize the model in a multi-step manner. Experimental results demonstrate that StepTool significantly outperforms existing methods in multi-step, tool-based tasks, providing a robust solution for complex task environments. Codes are available at https://github.com/yuyq18/StepTool.<|reference_end|>
@article{yu2024steptool:,
    title={StepTool: A Step-grained Reinforcement Learning Framework for Tool
  Learning in LLMs},
    author={Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai
  Wang, Chuhan Wu, Zhiqiang Guo, Min Zhang},
    journal={arXiv preprint arXiv:2410.07745},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07745},
    primaryClass={cs.CL}
}