3.3.2 Learning from Feedback
Collecting manually annotated tool-use examples, which probably include complete traces of human behaviors
and the ﬁnal answers, is time-consuming and labor-intensive. Moreover, the learned model may not adapt
effectively to new environments as it conforms to the recorded human behaviors. Besides, it is impractical
to explicitly annotate every possible scenario of environment condition and agent behavior (Codevilla et al.,
2019). Alternatively, humans learn from trial and error to correct and rectify their tool-use behaviors (Allen
et al., 2019). Similarly, feedback from both the environment and humans can enable the model to understand
the consequences of its actions and adapt its behaviors. The supervision from feedback can also enhance
the capabilities of an agent trained in a supervised way (Nakano et al., 2021; Baker et al., 2022). Formally,
learning from feedback can be described as optimizing the controllers’ parameters θC from open explorations
Tool Learning with Foundation Models