AssistGPT: A General Multi-modal Assistant that can Plan, Execute,
  Inspect, and Learn
Recent research on Large Language Models (LLMs) has led to remarkable
advancements in general NLP AI assistants. Some studies have further explored
the use of LLMs for planning and invoking models or APIs to address more
general multi-modal user queries. Despite this progress, complex visual-based
tasks still remain challenging due to the diverse nature of visual tasks. This
diversity is reflected in two aspects: 1) Reasoning paths. For many real-life
applications, it is hard to accurately decompose a query simply by examining
the query itself. Planning based on the specific visual content and the results
of each step is usually required. 2) Flexible inputs and intermediate results.
Input forms could be flexible for in-the-wild cases, and involves not only a
single image or video but a mixture of videos and images, e.g., a user-view
image with some reference videos. Besides, a complex reasoning process will
also generate diverse multimodal intermediate results, e.g., video narrations,
segmented video clips, etc. To address such general cases, we propose a
multi-modal AI assistant, AssistGPT, with an interleaved code and language
reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate
LLMs with various tools. Specifically, the Planner is capable of using natural
language to plan which tool in Executor should do next based on the current
reasoning progress. Inspector is an efficient memory manager to assist the
Planner to feed proper visual information into a specific tool. Finally, since
the entire reasoning process is complex and flexible, a Learner is designed to
enable the model to autonomously explore and discover the optimal solution. We
conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving
state-of-the-art results. Moreover, showcases demonstrate the ability of our
system to handle questions far more complex than those found in the benchmarks.
648bc7de6e8ac6a9cef1add4
Difei Gao
648bc7de6e8ac6a9cef1add5
Lei Ji
admin_assigned
2023-06-19T13:04:39.130Z
641b9688aebaa27e0752cd71
/avatars/3369af1f4e16dfaf727f78e63b21a7ad.svg
JiLei
user
AlexJi
648bc7de6e8ac6a9cef1add6
Luowei Zhou
648bc7de6e8ac6a9cef1add7
Kevin Qinghong Lin
admin_assigned
2023-06-19T13:03:55.677Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
648bc7de6e8ac6a9cef1add8
Joya Chen
claimed_verified
2025-05-13T08:14:23.038Z
642435a1a3adbc7142c3b0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/wgLT_w9jNWRU3O0jU0646.jpeg
Joya Chen
user
chenjoya
648bc7de6e8ac6a9cef1add9
Zihan Fan
648bc7de6e8ac6a9cef1adda
Mike Zheng Shou
AssistGPT: A General Multi-modal Assistant that can Plan, Execute,
  Inspect, and Learn
Recent research on Large Language Models (LLMs) has led to remarkable
advancements in general NLP AI assistants. Some studies have further explored
the use of LLMs for planning and invoking models or APIs to address more
general multi-modal user queries. Despite this progress, complex visual-based
tasks still remain challenging due to the diverse nature of visual tasks. This
diversity is reflected in two aspects: 1) Reasoning paths. For many real-life
applications, it is hard to accurately decompose a query simply by examining
the query itself. Planning based on the specific visual content and the results
of each step is usually required. 2) Flexible inputs and intermediate results.
Input forms could be flexible for in-the-wild cases, and involves not only a
single image or video but a mixture of videos and images, e.g., a user-view
image with some reference videos. Besides, a complex reasoning process will
also generate diverse multimodal intermediate results, e.g., video narrations,
segmented video clips, etc. To address such general cases, we propose a
multi-modal AI assistant, AssistGPT, with an interleaved code and language
reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate
LLMs with various tools. Specifically, the Planner is capable of using natural
language to plan which tool in Executor should do next based on the current
reasoning progress. Inspector is an efficient memory manager to assist the
Planner to feed proper visual information into a specific tool. Finally, since
the entire reasoning process is complex and flexible, a Learner is designed to
enable the model to autonomously explore and discover the optimal solution. We
conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving
state-of-the-art results. Moreover, showcases demonstrate the ability of our
system to handle questions far more complex than those found in the benchmarks.
A multi-modal AI assistant, AssistGPT, with a Plan, Execute, Inspect, and Learn (PEIL) approach integrates LLMs with various tools to handle complex visual-based tasks and achieves state-of-the-art results on benchmarks and beyond.
Large Language Models LLMs NLP AI assistants visual-based tasks multi-modal query decomposition visual content flexible inputs intermediate results reasoning process memory manager autonomously explore A-OKVQA NExT-QA