title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
summary: Tools are essential for large language models (LLMs) to acquire up-to-date
  information and take consequential actions in external environments. Existing
  work on tool-augmented LLMs primarily focuses on the broad coverage of tools
  and the flexibility of adding new tools. However, a critical aspect that has
  surprisingly been understudied is simply how accurately an LLM uses tools for
  which it has been trained. We find that existing LLMs, including GPT-4 and
  open-source LLMs specifically fine-tuned for tool use, only reach a correctness
  rate in the range of 30% to 60%, far from reliable use in practice. We propose
  a biologically inspired method for tool-augmented LLMs, simulated trial and
  error (STE), that orchestrates three key mechanisms for successful tool use
  behaviors in the biological system: trial and error, imagination, and memory.
  Specifically, STE leverages an LLM's 'imagination' to simulate plausible
  scenarios for using a tool, after which the LLM interacts with the tool to
  learn from its execution feedback. Both short-term and long-term memory are
  employed to improve the depth and breadth of the exploration, respectively.
  Comprehensive experiments on ToolBench show that STE substantially improves
  tool learning for LLMs under both in-context learning and fine-tuning settings,
  bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
  GPT-4. We also show effective continual learning of tools via a simple
  experience replay strategy.
paper_authors._id: 65ea7b02c5edf8e4d5c665c0
paper_authors.name: Boshi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:33:35.193Z
paper_authors._id: 62e5e742972081fe55828530
paper_authors.avatarUrl: /avatars/1cc9a4d5fbb634879a31f45bb7a5ec41.svg
paper_authors.fullname: Boshi Wang
paper_authors.type: user
paper_authors.user: boshiwang
paper_authors._id: 65ea7b02c5edf8e4d5c665c1
paper_authors.name: Hao Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:33:43.076Z
paper_authors._id: 64d3cba0673c5609e8ecb14d
paper_authors.avatarUrl: /avatars/8f6c52f06f1f77d16e067a0f7f05e1b1.svg
paper_authors.fullname: Hao Fang
paper_authors.type: user
paper_authors.user: haofang-msft
paper_authors._id: 65ea7b02c5edf8e4d5c665c2
paper_authors.name: Jason Eisner
paper_authors._id: 65ea7b02c5edf8e4d5c665c3
paper_authors.name: Benjamin Van Durme
paper_authors._id: 65ea7b02c5edf8e4d5c665c4
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-08T08:33:14.514Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
paper_summary: Tools are essential for large language models (LLMs) to acquire up-to-date
  information and take consequential actions in external environments. Existing
  work on tool-augmented LLMs primarily focuses on the broad coverage of tools
  and the flexibility of adding new tools. However, a critical aspect that has
  surprisingly been understudied is simply how accurately an LLM uses tools for
  which it has been trained. We find that existing LLMs, including GPT-4 and
  open-source LLMs specifically fine-tuned for tool use, only reach a correctness
  rate in the range of 30% to 60%, far from reliable use in practice. We propose
  a biologically inspired method for tool-augmented LLMs, simulated trial and
  error (STE), that orchestrates three key mechanisms for successful tool use
  behaviors in the biological system: trial and error, imagination, and memory.
  Specifically, STE leverages an LLM's 'imagination' to simulate plausible
  scenarios for using a tool, after which the LLM interacts with the tool to
  learn from its execution feedback. Both short-term and long-term memory are
  employed to improve the depth and breadth of the exploration, respectively.
  Comprehensive experiments on ToolBench show that STE substantially improves
  tool learning for LLMs under both in-context learning and fine-tuning settings,
  bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
  GPT-4. We also show effective continual learning of tools via a simple
  experience replay strategy.
paper_ai_summary: A biologically inspired method called simulated trial and error (STE) improves the accuracy of tool usage in large language models through imagination, trial and error, and memory, reaching better performance than existing models like GPT-4.
paper_ai_keywords: tool-augmented LLMs simulated trial and error trial and error imagination memory ToolBench in-context learning fine-tuning continual learning experience replay