title: ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling
    under Long-Context Scenario
summary: Enhancing large language models (LLMs) with real-time APIs can help generate
  more accurate and up-to-date responses. However, evaluating the function
  calling abilities of LLMs in real-world scenarios remains under-explored due to
  the complexity of data collection and evaluation. In this work, we introduce
  ComplexFuncBench, a benchmark for complex function calling across five
  real-world scenarios. Compared to existing benchmarks, ComplexFuncBench
  encompasses multi-step and constrained function calling, which requires
  long-parameter filing, parameter value reasoning, and 128k long context.
  Additionally, we propose an automatic framework, ComplexEval, for
  quantitatively evaluating complex function calling tasks. Through comprehensive
  experiments, we demonstrate the deficiencies of state-of-the-art LLMs in
  function calling and suggest future directions for optimizing these
  capabilities. The data and code are available at
  https://github.com/THUDM/ComplexFuncBench.
paper_authors._id: 678db8f76e06f11c16d15ff5
paper_authors.name: Lucen Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-20T14:32:47.710Z
paper_authors._id: 60eff04e22ab0ac83b0fc9d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60eff04e22ab0ac83b0fc9d8/pBjftNyFN1qB8Br3FZQmD.jpeg
paper_authors.fullname: lucen zhong
paper_authors.type: user
paper_authors.user: anchorzhong
paper_authors._id: 678db8f76e06f11c16d15ff6
paper_authors.name: Zhengxiao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-20T14:32:53.512Z
paper_authors._id: 63033dc4e1e7f0e03a5e1a31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661157784937-63033dc4e1e7f0e03a5e1a31.jpeg
paper_authors.fullname: Zhengxiao Du
paper_authors.type: user
paper_authors.user: zxdu20
paper_authors._id: 678db8f76e06f11c16d15ff7
paper_authors.name: Xiaohan Zhang
paper_authors._id: 678db8f76e06f11c16d15ff8
paper_authors.name: Haiyi Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-20T14:33:03.717Z
paper_authors._id: 66e40e556998c3d86c3e9263
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3aZyGNr8bbnY0Ry0iObr4.jpeg
paper_authors.fullname: Haiyi Hu
paper_authors.type: user
paper_authors.user: haithesea
paper_authors._id: 678db8f76e06f11c16d15ff9
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-20T14:33:09.675Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_title: ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling
    under Long-Context Scenario
paper_summary: Enhancing large language models (LLMs) with real-time APIs can help generate
  more accurate and up-to-date responses. However, evaluating the function
  calling abilities of LLMs in real-world scenarios remains under-explored due to
  the complexity of data collection and evaluation. In this work, we introduce
  ComplexFuncBench, a benchmark for complex function calling across five
  real-world scenarios. Compared to existing benchmarks, ComplexFuncBench
  encompasses multi-step and constrained function calling, which requires
  long-parameter filing, parameter value reasoning, and 128k long context.
  Additionally, we propose an automatic framework, ComplexEval, for
  quantitatively evaluating complex function calling tasks. Through comprehensive
  experiments, we demonstrate the deficiencies of state-of-the-art LLMs in
  function calling and suggest future directions for optimizing these
  capabilities. The data and code are available at
  https://github.com/THUDM/ComplexFuncBench.
paper_ai_summary: ComplexFuncBench is a benchmark for evaluating complex function calling in large language models, including multi-step and constrained function calls, with an automatic evaluation framework called ComplexEval.
paper_ai_keywords: large language models function calling real-world scenarios ComplexFuncBench ComplexEval multi-step constrained function calling long-parameter filing parameter value reasoning long context