• Self-Debugging: Several datasets are available to assess the ability of LLMs to self-debug, including
InterCode-Bash and InterCode-SQL (Yang et al., 2023b), MINT-MBPP and MINT-HumanEval
(Wang et al., 2023d), and RoboCodeGen (Liang et al., 2023).

• Following Natural Language Feedback: MINT (Wang et al., 2023d) can also be used to measure
the ability of LLMs to leverage natural language feedback by using GPT-4 (OpenAI, 2023b) to
simulate human users.

• Exploring Environment: ALFWorld (Shridhar et al., 2020), InterCode-CTF (Yang et al., 2023b),
and WebArena (Zhou et al., 2023c) are introduced to evaluate whether LLMs-based agents are
able to gather information from the environment and make decisions.

Model

ALFWorld

Environment

NL Feedback

Lemur-70B-chat
GPT-3.5-turbo
GPT-4
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup