CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language
  Models in Tool-Calling Error Scenarios
The ability of large language models (LLMs) to utilize external tools has
enabled them to tackle an increasingly diverse range of tasks. However, as the
tasks become more complex and long-horizon, the intricate tool utilization
process may trigger various unexpected errors. Therefore, how to effectively
handle such errors, including identifying, diagnosing, and recovering from
them, has emerged as a key research direction for advancing tool learning. In
this work, we first extensively analyze the types of errors encountered during
the function-calling process on several competitive tool evaluation benchmarks.
Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation
benchmark specialized for tool learning. Building upon a novel evolutionary
strategy for dataset construction, CRITICTOOL holds diverse tool-use errors
with varying complexities, which better reflects real-world scenarios. We
conduct extensive experiments on CRITICTOOL, and validate the generalization
and effectiveness of our constructed benchmark strategy. We also provide an
in-depth analysis of the tool reflection ability on various LLMs, offering a
new perspective on the field of tool learning in LLMs. The code is available at
https://github.com/Shellorley0513/CriticTool{https://github.com/Shellorley0513/CriticTool}.
68524ff90164cd13167105aa
Shiting Huang
68524ff90164cd13167105ab
Zhen Fang
claimed_verified
2025-06-18T10:58:56.933Z
64b0a5037a475fba70a7260d
https://cdn-avatars.huggingface.co/v1/production/uploads/64b0a5037a475fba70a7260d/MauBbb6raMA23yrR1Zq21.jpeg
Zhen Fang
user
CostaliyA
68524ff90164cd13167105ac
Zehui Chen
68524ff90164cd13167105ad
Siyu Yuan
68524ff90164cd13167105ae
Junjie Ye
68524ff90164cd13167105af
Yu Zeng
claimed_verified
2025-07-24T14:11:24.598Z
665d652e0f35c005de892108
https://cdn-avatars.huggingface.co/v1/production/uploads/665d652e0f35c005de892108/OGLbgZekX-3XTBkwS8k86.jpeg
Yu Zeng
user
YuZeng260
68524ff90164cd13167105b0
Lin Chen
68524ff90164cd13167105b1
Qi Mao
68524ff90164cd13167105b2
Feng Zhao
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language
  Models in Tool-Calling Error Scenarios
The ability of large language models (LLMs) to utilize external tools has
enabled them to tackle an increasingly diverse range of tasks. However, as the
tasks become more complex and long-horizon, the intricate tool utilization
process may trigger various unexpected errors. Therefore, how to effectively
handle such errors, including identifying, diagnosing, and recovering from
them, has emerged as a key research direction for advancing tool learning. In
this work, we first extensively analyze the types of errors encountered during
the function-calling process on several competitive tool evaluation benchmarks.
Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation
benchmark specialized for tool learning. Building upon a novel evolutionary
strategy for dataset construction, CRITICTOOL holds diverse tool-use errors
with varying complexities, which better reflects real-world scenarios. We
conduct extensive experiments on CRITICTOOL, and validate the generalization
and effectiveness of our constructed benchmark strategy. We also provide an
in-depth analysis of the tool reflection ability on various LLMs, offering a
new perspective on the field of tool learning in LLMs. The code is available at
https://github.com/Shellorley0513/CriticTool{https://github.com/Shellorley0513/CriticTool}.
A comprehensive benchmark, CRITICTOOL, evaluates and enhances the robustness of large language models in handling errors during tool usage.
large language models tool learning function-calling process error identification error diagnosis error recovery evolutionary strategy dataset construction tool reflection ability critique evaluation benchmark