3.1.2 Connecting the Components
Formally, assume we have a tool set T , which the controller can utilize to accomplish certain tasks. At time
step t, environment E provides feedback et on the tool execution. The perceiver P receives the user feedback
ft and the environment feedback et, and generates summarized feedback xt. Typically, the perceiver can be
achieved by pre-deﬁned rules (e.g., concatenating ft and et) to form xt, or modeled with complex neural

14

3.2 The General Procedure: From Intent to Plan

Figure 4: Illustration of the tool learning framework, where we display the human user and four core
ingredients of the framework: tool set, controller, perceiver, and environment. The user sends an instruction
to the controller, which then makes decisions and executes tools in the environment. The perceiver receives
feedback from both the environment and the user and summarizes them to the controller.
Tool Learning with Foundation Models