title: AssistGPT: A General Multi-modal Assistant that can Plan, Execute,
    Inspect, and Learn
summary: Recent research on Large Language Models (LLMs) has led to remarkable
  advancements in general NLP AI assistants. Some studies have further explored
  the use of LLMs for planning and invoking models or APIs to address more
  general multi-modal user queries. Despite this progress, complex visual-based
  tasks still remain challenging due to the diverse nature of visual tasks. This
  diversity is reflected in two aspects: 1) Reasoning paths. For many real-life
  applications, it is hard to accurately decompose a query simply by examining
  the query itself. Planning based on the specific visual content and the results
  of each step is usually required. 2) Flexible inputs and intermediate results.
  Input forms could be flexible for in-the-wild cases, and involves not only a
  single image or video but a mixture of videos and images, e.g., a user-view
  image with some reference videos. Besides, a complex reasoning process will
  also generate diverse multimodal intermediate results, e.g., video narrations,
  segmented video clips, etc. To address such general cases, we propose a
  multi-modal AI assistant, AssistGPT, with an interleaved code and language
  reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate
  LLMs with various tools. Specifically, the Planner is capable of using natural
  language to plan which tool in Executor should do next based on the current
  reasoning progress. Inspector is an efficient memory manager to assist the
  Planner to feed proper visual information into a specific tool. Finally, since
  the entire reasoning process is complex and flexible, a Learner is designed to
  enable the model to autonomously explore and discover the optimal solution. We
  conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving
  state-of-the-art results. Moreover, showcases demonstrate the ability of our
  system to handle questions far more complex than those found in the benchmarks.
paper_authors._id: 648bc7de6e8ac6a9cef1add4
paper_authors.name: Difei Gao
paper_authors._id: 648bc7de6e8ac6a9cef1add5
paper_authors.name: Lei Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T13:04:39.130Z
paper_authors._id: 641b9688aebaa27e0752cd71
paper_authors.avatarUrl: /avatars/3369af1f4e16dfaf727f78e63b21a7ad.svg
paper_authors.fullname: JiLei
paper_authors.type: user
paper_authors.user: AlexJi
paper_authors._id: 648bc7de6e8ac6a9cef1add6
paper_authors.name: Luowei Zhou
paper_authors._id: 648bc7de6e8ac6a9cef1add7
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T13:03:55.677Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 648bc7de6e8ac6a9cef1add8
paper_authors.name: Joya Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:14:23.038Z
paper_authors._id: 642435a1a3adbc7142c3b0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/wgLT_w9jNWRU3O0jU0646.jpeg
paper_authors.fullname: Joya Chen
paper_authors.type: user
paper_authors.user: chenjoya
paper_authors._id: 648bc7de6e8ac6a9cef1add9
paper_authors.name: Zihan Fan
paper_authors._id: 648bc7de6e8ac6a9cef1adda
paper_authors.name: Mike Zheng Shou
paper_title: AssistGPT: A General Multi-modal Assistant that can Plan, Execute,
    Inspect, and Learn
paper_summary: Recent research on Large Language Models (LLMs) has led to remarkable
  advancements in general NLP AI assistants. Some studies have further explored
  the use of LLMs for planning and invoking models or APIs to address more
  general multi-modal user queries. Despite this progress, complex visual-based
  tasks still remain challenging due to the diverse nature of visual tasks. This
  diversity is reflected in two aspects: 1) Reasoning paths. For many real-life
  applications, it is hard to accurately decompose a query simply by examining
  the query itself. Planning based on the specific visual content and the results
  of each step is usually required. 2) Flexible inputs and intermediate results.
  Input forms could be flexible for in-the-wild cases, and involves not only a
  single image or video but a mixture of videos and images, e.g., a user-view
  image with some reference videos. Besides, a complex reasoning process will
  also generate diverse multimodal intermediate results, e.g., video narrations,
  segmented video clips, etc. To address such general cases, we propose a
  multi-modal AI assistant, AssistGPT, with an interleaved code and language
  reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate
  LLMs with various tools. Specifically, the Planner is capable of using natural
  language to plan which tool in Executor should do next based on the current
  reasoning progress. Inspector is an efficient memory manager to assist the
  Planner to feed proper visual information into a specific tool. Finally, since
  the entire reasoning process is complex and flexible, a Learner is designed to
  enable the model to autonomously explore and discover the optimal solution. We
  conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving
  state-of-the-art results. Moreover, showcases demonstrate the ability of our
  system to handle questions far more complex than those found in the benchmarks.
paper_ai_summary: A multi-modal AI assistant, AssistGPT, with a Plan, Execute, Inspect, and Learn (PEIL) approach integrates LLMs with various tools to handle complex visual-based tasks and achieves state-of-the-art results on benchmarks and beyond.
paper_ai_keywords: Large Language Models LLMs NLP AI assistants visual-based tasks multi-modal query decomposition visual content flexible inputs intermediate results reasoning process memory manager autonomously explore A-OKVQA NExT-QA