Embodied Scene. ALFWorld (Shridhar et al., 2021) is an aligned text and embodied environment game,
where agents need to interact with objects (e.g., fridge, microwave, drawer, etc.) in a house to complete a task
(e.g., putting a clean spatula in a drawer). We largely follow the setting of ReACT (Yao et al., 2022b) and
report the success rate on the valid set.
Cooking Assistant. We choose AllRecipe15 to investigate whether models can ﬁnd the proper cooking recipe
and extract important details. The tool is designed similarly to the search engine tool. With this tool, the model

15https://www.allrecipes.com/

26

4.2 Experiments
Tool Learning with Foundation Models