title: ControlLLM: Augment Language Models with Tools by Searching on Graphs
summary: We present ControlLLM, a novel framework that enables large language models
  (LLMs) to utilize multi-modal tools for solving complex real-world tasks.
  Despite the remarkable performance of LLMs, they still struggle with tool
  invocation due to ambiguous user prompts, inaccurate tool selection and
  parameterization, and inefficient tool scheduling. To overcome these
  challenges, our framework comprises three key components: (1) a task
  decomposer that breaks down a complex task into clear subtasks with
  well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG)
  paradigm that searches the optimal solution path on a pre-built tool graph,
  which specifies the parameter and dependency relations among different tools;
  and (3) an execution engine with a rich toolbox that interprets the
  solution path and runs the tools efficiently on different computational
  devices. We evaluate our framework on diverse tasks involving image, audio, and
  video processing, demonstrating its superior accuracy, efficiency, and
  versatility compared to existing methods.
paper_authors._id: 653f16832273676554cd0fa9
paper_authors.name: Zhaoyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:34:35.219Z
paper_authors._id: 6432c6152bfb2b0ec7572479
paper_authors.avatarUrl: /avatars/7c2f908f511cdfa38a19e967d86f9c40.svg
paper_authors.fullname: Zhaoyang Liu
paper_authors.type: user
paper_authors.user: zyliu
paper_authors._id: 653f16832273676554cd0faa
paper_authors.name: Zeqiang Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:34:54.940Z
paper_authors._id: 63044b89eedc089484c995ad
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/T4mOIQLaQdM5_oviaw_Cp.png
paper_authors.fullname: Zeqiang Lai
paper_authors.type: user
paper_authors.user: ZeqiangLai
paper_authors._id: 653f16832273676554cd0fab
paper_authors.name: Zhangwei Gao
paper_authors._id: 653f16832273676554cd0fac
paper_authors.name: Erfei Cui
paper_authors._id: 653f16832273676554cd0fad
paper_authors.name: Xizhou Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:35:24.605Z
paper_authors._id: 64ae2359179421d320b1694b
paper_authors.avatarUrl: /avatars/c387a75191005bcaa473091de5383a10.svg
paper_authors.fullname: Xizhou Zhu
paper_authors.type: user
paper_authors.user: Einsiedler
paper_authors._id: 653f16832273676554cd0fae
paper_authors.name: Lewei Lu
paper_authors._id: 653f16832273676554cd0faf
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:35:42.039Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 653f16832273676554cd0fb0
paper_authors.name: Yu Qiao
paper_authors._id: 653f16832273676554cd0fb1
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:36:05.294Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 653f16832273676554cd0fb2
paper_authors.name: Wenhai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:36:16.140Z
paper_authors._id: 64d1c560c0c627dfa71bdbe0
paper_authors.avatarUrl: /avatars/f42794fe25bffcd870a1bcee69b95298.svg
paper_authors.fullname: wenhai.wang
paper_authors.type: user
paper_authors.user: wangwhcore
paper_title: ControlLLM: Augment Language Models with Tools by Searching on Graphs
paper_summary: We present ControlLLM, a novel framework that enables large language models
  (LLMs) to utilize multi-modal tools for solving complex real-world tasks.
  Despite the remarkable performance of LLMs, they still struggle with tool
  invocation due to ambiguous user prompts, inaccurate tool selection and
  parameterization, and inefficient tool scheduling. To overcome these
  challenges, our framework comprises three key components: (1) a task
  decomposer that breaks down a complex task into clear subtasks with
  well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG)
  paradigm that searches the optimal solution path on a pre-built tool graph,
  which specifies the parameter and dependency relations among different tools;
  and (3) an execution engine with a rich toolbox that interprets the
  solution path and runs the tools efficiently on different computational
  devices. We evaluate our framework on diverse tasks involving image, audio, and
  video processing, demonstrating its superior accuracy, efficiency, and
  versatility compared to existing methods.
paper_ai_summary: ControlLLM enhances large language models by integrating multi-modal tools through task decomposition, optimal solution path finding, and efficient tool execution, achieving superior performance in diverse tasks.
paper_ai_keywords: large language models multi-modal tools task decomposer Thoughts-on-Graph tool graph execution engine image processing audio processing video processing