LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Tools are essential for large language models (LLMs) to acquire up-to-date
information and take consequential actions in external environments. Existing
work on tool-augmented LLMs primarily focuses on the broad coverage of tools
and the flexibility of adding new tools. However, a critical aspect that has
surprisingly been understudied is simply how accurately an LLM uses tools for
which it has been trained. We find that existing LLMs, including GPT-4 and
open-source LLMs specifically fine-tuned for tool use, only reach a correctness
rate in the range of 30% to 60%, far from reliable use in practice. We propose
a biologically inspired method for tool-augmented LLMs, simulated trial and
error (STE), that orchestrates three key mechanisms for successful tool use
behaviors in the biological system: trial and error, imagination, and memory.
Specifically, STE leverages an LLM's 'imagination' to simulate plausible
scenarios for using a tool, after which the LLM interacts with the tool to
learn from its execution feedback. Both short-term and long-term memory are
employed to improve the depth and breadth of the exploration, respectively.
Comprehensive experiments on ToolBench show that STE substantially improves
tool learning for LLMs under both in-context learning and fine-tuning settings,
bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
GPT-4. We also show effective continual learning of tools via a simple
experience replay strategy.
65ea7b02c5edf8e4d5c665c0
Boshi Wang
admin_assigned
2024-03-08T09:33:35.193Z
62e5e742972081fe55828530
/avatars/1cc9a4d5fbb634879a31f45bb7a5ec41.svg
Boshi Wang
user
boshiwang
65ea7b02c5edf8e4d5c665c1
Hao Fang
admin_assigned
2024-03-08T09:33:43.076Z
64d3cba0673c5609e8ecb14d
/avatars/8f6c52f06f1f77d16e067a0f7f05e1b1.svg
Hao Fang
user
haofang-msft
65ea7b02c5edf8e4d5c665c2
Jason Eisner
65ea7b02c5edf8e4d5c665c3
Benjamin Van Durme
65ea7b02c5edf8e4d5c665c4
Yu Su
claimed_verified
2024-03-08T08:33:14.514Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Tools are essential for large language models (LLMs) to acquire up-to-date
information and take consequential actions in external environments. Existing
work on tool-augmented LLMs primarily focuses on the broad coverage of tools
and the flexibility of adding new tools. However, a critical aspect that has
surprisingly been understudied is simply how accurately an LLM uses tools for
which it has been trained. We find that existing LLMs, including GPT-4 and
open-source LLMs specifically fine-tuned for tool use, only reach a correctness
rate in the range of 30% to 60%, far from reliable use in practice. We propose
a biologically inspired method for tool-augmented LLMs, simulated trial and
error (STE), that orchestrates three key mechanisms for successful tool use
behaviors in the biological system: trial and error, imagination, and memory.
Specifically, STE leverages an LLM's 'imagination' to simulate plausible
scenarios for using a tool, after which the LLM interacts with the tool to
learn from its execution feedback. Both short-term and long-term memory are
employed to improve the depth and breadth of the exploration, respectively.
Comprehensive experiments on ToolBench show that STE substantially improves
tool learning for LLMs under both in-context learning and fine-tuning settings,
bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
GPT-4. We also show effective continual learning of tools via a simple
experience replay strategy.
A biologically inspired method called simulated trial and error (STE) improves the accuracy of tool usage in large language models through imagination, trial and error, and memory, reaching better performance than existing models like GPT-4.
tool-augmented LLMs simulated trial and error trial and error imagination memory ToolBench in-context learning fine-tuning continual learning experience replay