title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
summary: Tools are essential for large language models (LLMs) to acquire up-to-date
  information and take consequential actions in external environments. Existing
  work on tool-augmented LLMs primarily focuses on the broad coverage of tools
  and the flexibility of adding new tools. However, a critical aspect that has
  surprisingly been understudied is simply how accurately an LLM uses tools for
  which it has been trained. We find that existing LLMs, including GPT-4 and
  open-source LLMs specifically fine-tuned for tool use, only reach a correctness
  rate in the range of 30% to 60%, far from reliable use in practice. We propose
  a biologically inspired method for tool-augmented LLMs, simulated trial and
  error (STE), that orchestrates three key mechanisms for successful tool use
  behaviors in the biological system: trial and error, imagination, and memory.
  Specifically, STE leverages an LLM's 'imagination' to simulate plausible
  scenarios for using a tool, after which the LLM interacts with the tool to
  learn from its execution feedback. Both short-term and long-term memory are
  employed to improve the depth and breadth of the exploration, respectively.
  Comprehensive experiments on ToolBench show that STE substantially improves
  tool learning for LLMs under both in-context learning and fine-tuning settings,
  bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
  GPT-4. We also show effective continual learning of tools via a simple
  experience replay strategy.
paper_authors._id: 65ea7b02c5edf8e4d5c665c0
paper_authors.name: Boshi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:33:35.193Z
paper_authors._id: 62e5e742972081fe55828530
paper_authors.avatarUrl: /avatars/1cc9a4d5fbb634879a31f45bb7a5ec41.svg
paper_authors.fullname: Boshi Wang
paper_authors.type: user
paper_authors.user: boshiwang
paper_authors._id: 65ea7b02c5edf8e4d5c665c1
paper_authors.name: Hao Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:33:43.076Z
paper_authors._id: 64d3cba0673c5609e8ecb14d
paper_authors.avatarUrl: /avatars/8f6c52f06f1f77d16e067a0f7f05e1b1.svg
paper_authors.fullname: Hao Fang
paper_authors.type: user
paper_authors.user: haofang-msft
paper_authors._id: 65ea7b02c5edf8e4d5c665c2
paper_authors.name: Jason Eisner
paper_authors._id: 65ea7b02c5edf8e4d5c665c3
paper_authors.name: Benjamin Van Durme
paper_authors._id: 65ea7b02c5edf8e4d5c665c4
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-08T08:33:14.514Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/grnmnxO1foz6aRRJ_A2NU.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
paper_summary: Tools are essential for large language models (LLMs) to acquire up-to-date
  information and take consequential actions in external environments. Existing
  work on tool-augmented LLMs primarily focuses on the broad coverage of tools
  and the flexibility of adding new tools. However, a critical aspect that has
  surprisingly been understudied is simply how accurately an LLM uses tools for
  which it has been trained. We find that existing LLMs, including GPT-4 and
  open-source LLMs specifically fine-tuned for tool use, only reach a correctness
  rate in the range of 30% to 60%, far from reliable use in practice. We propose
  a biologically inspired method for tool-augmented LLMs, simulated trial and
  error (STE), that orchestrates three key mechanisms for successful tool use
  behaviors in the biological system: trial and error, imagination, and memory.
  Specifically, STE leverages an LLM's 'imagination' to simulate plausible
  scenarios for using a tool, after which the LLM interacts with the tool to
  learn from its execution feedback. Both short-term and long-term memory are
  employed to improve the depth and breadth of the exploration, respectively.
  Comprehensive experiments on ToolBench show that STE substantially improves
  tool learning for LLMs under both in-context learning and fine-tuning settings,
  bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
  GPT-4. We also show effective continual learning of tools via a simple
  experience replay strategy.
paper_ai_keywords: tool-augmented LLMs trial and error imagination memory short-term memory long-term memory in-context learning fine-tuning Mistral-Instruct-7B ToolBench experience replay