ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling
  under Long-Context Scenario
Enhancing large language models (LLMs) with real-time APIs can help generate
more accurate and up-to-date responses. However, evaluating the function
calling abilities of LLMs in real-world scenarios remains under-explored due to
the complexity of data collection and evaluation. In this work, we introduce
ComplexFuncBench, a benchmark for complex function calling across five
real-world scenarios. Compared to existing benchmarks, ComplexFuncBench
encompasses multi-step and constrained function calling, which requires
long-parameter filing, parameter value reasoning, and 128k long context.
Additionally, we propose an automatic framework, ComplexEval, for
quantitatively evaluating complex function calling tasks. Through comprehensive
experiments, we demonstrate the deficiencies of state-of-the-art LLMs in
function calling and suggest future directions for optimizing these
capabilities. The data and code are available at
https://github.com/THUDM/ComplexFuncBench.
678db8f76e06f11c16d15ff5
Lucen Zhong
admin_assigned
2025-01-20T14:32:47.710Z
60eff04e22ab0ac83b0fc9d8
https://cdn-avatars.huggingface.co/v1/production/uploads/60eff04e22ab0ac83b0fc9d8/pBjftNyFN1qB8Br3FZQmD.jpeg
lucen zhong
user
anchorzhong
678db8f76e06f11c16d15ff6
Zhengxiao Du
admin_assigned
2025-01-20T14:32:53.512Z
63033dc4e1e7f0e03a5e1a31
https://cdn-avatars.huggingface.co/v1/production/uploads/1661157784937-63033dc4e1e7f0e03a5e1a31.jpeg
Zhengxiao Du
user
zxdu20
678db8f76e06f11c16d15ff7
Xiaohan Zhang
678db8f76e06f11c16d15ff8
Haiyi Hu
admin_assigned
2025-01-20T14:33:03.717Z
66e40e556998c3d86c3e9263
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3aZyGNr8bbnY0Ry0iObr4.jpeg
Haiyi Hu
user
haithesea
678db8f76e06f11c16d15ff9
Jie Tang
admin_assigned
2025-01-20T14:33:09.675Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling
  under Long-Context Scenario
Enhancing large language models (LLMs) with real-time APIs can help generate
more accurate and up-to-date responses. However, evaluating the function
calling abilities of LLMs in real-world scenarios remains under-explored due to
the complexity of data collection and evaluation. In this work, we introduce
ComplexFuncBench, a benchmark for complex function calling across five
real-world scenarios. Compared to existing benchmarks, ComplexFuncBench
encompasses multi-step and constrained function calling, which requires
long-parameter filing, parameter value reasoning, and 128k long context.
Additionally, we propose an automatic framework, ComplexEval, for
quantitatively evaluating complex function calling tasks. Through comprehensive
experiments, we demonstrate the deficiencies of state-of-the-art LLMs in
function calling and suggest future directions for optimizing these
capabilities. The data and code are available at
https://github.com/THUDM/ComplexFuncBench.
ComplexFuncBench is a benchmark for evaluating complex function calling in large language models, including multi-step and constrained function calls, with an automatic evaluation framework called ComplexEval.
large language models function calling real-world scenarios ComplexFuncBench ComplexEval multi-step constrained function calling long-parameter filing parameter value reasoning long context