ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages
<|reference_start|>ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages: Tool learning is widely acknowledged as a foundational approach or deploying large language models (LLMs) in real-world scenarios. While current research primarily emphasizes leveraging tools to augment LLMs, it frequently neglects emerging safety considerations tied to their application. To fill this gap, we present *ToolSword*, a comprehensive framework dedicated to meticulously investigating safety issues linked to LLMs in tool learning. Specifically, ToolSword delineates six safety scenarios for LLMs in tool learning, encompassing **malicious queries** and **jailbreak attacks** in the input stage, **noisy misdirection** and **risky cues** in the execution stage, and **harmful feedback** and **error conflicts** in the output stage. Experiments conducted on 11 open-source and closed-source LLMs reveal enduring safety challenges in tool learning, such as handling harmful queries, employing risky tools, and delivering detrimental feedback, which even GPT-4 is susceptible to. Moreover, we conduct further studies with the aim of fostering research on tool learning safety. The data is released in https://github.com/Junjie-Ye/ToolSword.<|reference_end|>
@article{ye2024toolsword:,
    title={ToolSword: Unveiling Safety Issues of Large Language Models in Tool
  Learning Across Three Stages},
    author={Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong
  Wu, Qi Zhang, Tao Gui, Xuanjing Huang},
    journal={Proceedings of the 62nd Annual Meeting of the Association for
  Computational Linguistics 2024 (Volume 1: Long Papers)},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.10753},
    primaryClass={cs.CL cs.AI}
}