ControlLLM: Augment Language Models with Tools by Searching on Graphs
We present ControlLLM, a novel framework that enables large language models
(LLMs) to utilize multi-modal tools for solving complex real-world tasks.
Despite the remarkable performance of LLMs, they still struggle with tool
invocation due to ambiguous user prompts, inaccurate tool selection and
parameterization, and inefficient tool scheduling. To overcome these
challenges, our framework comprises three key components: (1) a task
decomposer that breaks down a complex task into clear subtasks with
well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG)
paradigm that searches the optimal solution path on a pre-built tool graph,
which specifies the parameter and dependency relations among different tools;
and (3) an execution engine with a rich toolbox that interprets the
solution path and runs the tools efficiently on different computational
devices. We evaluate our framework on diverse tasks involving image, audio, and
video processing, demonstrating its superior accuracy, efficiency, and
versatility compared to existing methods.
653f16832273676554cd0fa9
Zhaoyang Liu
admin_assigned
2023-10-30T13:34:35.219Z
6432c6152bfb2b0ec7572479
/avatars/7c2f908f511cdfa38a19e967d86f9c40.svg
Zhaoyang Liu
user
zyliu
653f16832273676554cd0faa
Zeqiang Lai
admin_assigned
2023-10-30T13:34:54.940Z
63044b89eedc089484c995ad
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/T4mOIQLaQdM5_oviaw_Cp.png
Zeqiang Lai
user
ZeqiangLai
653f16832273676554cd0fab
Zhangwei Gao
653f16832273676554cd0fac
Erfei Cui
653f16832273676554cd0fad
Xizhou Zhu
admin_assigned
2023-10-30T13:35:24.605Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
653f16832273676554cd0fae
Lewei Lu
653f16832273676554cd0faf
Qifeng Chen
admin_assigned
2023-10-30T13:35:42.039Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
653f16832273676554cd0fb0
Yu Qiao
653f16832273676554cd0fb1
Jifeng Dai
admin_assigned
2023-10-30T13:36:05.294Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
653f16832273676554cd0fb2
Wenhai Wang
admin_assigned
2023-10-30T13:36:16.140Z
64d1c560c0c627dfa71bdbe0
/avatars/f42794fe25bffcd870a1bcee69b95298.svg
wenhai.wang
user
wangwhcore
ControlLLM: Augment Language Models with Tools by Searching on Graphs
We present ControlLLM, a novel framework that enables large language models
(LLMs) to utilize multi-modal tools for solving complex real-world tasks.
Despite the remarkable performance of LLMs, they still struggle with tool
invocation due to ambiguous user prompts, inaccurate tool selection and
parameterization, and inefficient tool scheduling. To overcome these
challenges, our framework comprises three key components: (1) a task
decomposer that breaks down a complex task into clear subtasks with
well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG)
paradigm that searches the optimal solution path on a pre-built tool graph,
which specifies the parameter and dependency relations among different tools;
and (3) an execution engine with a rich toolbox that interprets the
solution path and runs the tools efficiently on different computational
devices. We evaluate our framework on diverse tasks involving image, audio, and
video processing, demonstrating its superior accuracy, efficiency, and
versatility compared to existing methods.
ControlLLM enhances large language models by integrating multi-modal tools through task decomposition, optimal solution path finding, and efficient tool execution, achieving superior performance in diverse tasks.
large language models multi-modal tools task decomposer Thoughts-on-Graph tool graph execution engine image processing audio processing video processing