title: AVIS: Autonomous Visual Information Seeking with Large Language Models
summary: In this paper, we propose an autonomous information seeking visual question
  answering framework, AVIS. Our method leverages a Large Language Model (LLM) to
  dynamically strategize the utilization of external tools and to investigate
  their outputs, thereby acquiring the indispensable knowledge needed to provide
  answers to the posed questions. Responding to visual questions that necessitate
  external knowledge, such as "What event is commemorated by the building
  depicted in this image?", is a complex task. This task presents a combinatorial
  search space that demands a sequence of actions, including invoking APIs,
  analyzing their responses, and making informed decisions. We conduct a user
  study to collect a variety of instances of human decision-making when faced
  with this task. This data is then used to design a system comprised of three
  components: an LLM-powered planner that dynamically determines which tool to
  use next, an LLM-powered reasoner that analyzes and extracts key information
  from the tool outputs, and a working memory component that retains the acquired
  information throughout the process. The collected user behavior serves as a
  guide for our system in two key ways. First, we create a transition graph by
  analyzing the sequence of decisions made by users. This graph delineates
  distinct states and confines the set of actions available at each state.
  Second, we use examples of user decision-making to provide our LLM-powered
  planner and reasoner with relevant contextual instances, enhancing their
  capacity to make informed decisions. We show that AVIS achieves
  state-of-the-art results on knowledge-intensive visual question answering
  benchmarks such as Infoseek and OK-VQA.
paper_authors._id: 648bc615d0f0f5b02eaf5d95
paper_authors.name: Ziniu Hu
paper_authors._id: 648bc615d0f0f5b02eaf5d96
paper_authors.name: Ahmet Iscen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-16T16:14:37.813Z
paper_authors._id: 64885bd4dfa0998196b1cf0a
paper_authors.avatarUrl: /avatars/c3e2bbe793cdd146aea27c0011aace71.svg
paper_authors.fullname: Ahmet Iscen
paper_authors.type: user
paper_authors.user: ahmetius
paper_authors._id: 648bc615d0f0f5b02eaf5d97
paper_authors.name: Chen Sun
paper_authors._id: 648bc615d0f0f5b02eaf5d98
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-16T16:15:59.421Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 648bc615d0f0f5b02eaf5d99
paper_authors.name: Yizhou Sun
paper_authors._id: 648bc615d0f0f5b02eaf5d9a
paper_authors.name: David A Ross
paper_authors._id: 648bc615d0f0f5b02eaf5d9b
paper_authors.name: Cordelia Schmid
paper_authors._id: 648bc615d0f0f5b02eaf5d9c
paper_authors.name: Alireza Fathi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-16T16:16:51.620Z
paper_authors._id: 645685170a1ea2229a28dbb2
paper_authors.avatarUrl: /avatars/b83e043784b6db225ae4d0422aabbb3a.svg
paper_authors.fullname: alireza fathi
paper_authors.type: user
paper_authors.user: alireza007
paper_title: AVIS: Autonomous Visual Information Seeking with Large Language Models
paper_summary: In this paper, we propose an autonomous information seeking visual question
  answering framework, AVIS. Our method leverages a Large Language Model (LLM) to
  dynamically strategize the utilization of external tools and to investigate
  their outputs, thereby acquiring the indispensable knowledge needed to provide
  answers to the posed questions. Responding to visual questions that necessitate
  external knowledge, such as "What event is commemorated by the building
  depicted in this image?", is a complex task. This task presents a combinatorial
  search space that demands a sequence of actions, including invoking APIs,
  analyzing their responses, and making informed decisions. We conduct a user
  study to collect a variety of instances of human decision-making when faced
  with this task. This data is then used to design a system comprised of three
  components: an LLM-powered planner that dynamically determines which tool to
  use next, an LLM-powered reasoner that analyzes and extracts key information
  from the tool outputs, and a working memory component that retains the acquired
  information throughout the process. The collected user behavior serves as a
  guide for our system in two key ways. First, we create a transition graph by
  analyzing the sequence of decisions made by users. This graph delineates
  distinct states and confines the set of actions available at each state.
  Second, we use examples of user decision-making to provide our LLM-powered
  planner and reasoner with relevant contextual instances, enhancing their
  capacity to make informed decisions. We show that AVIS achieves
  state-of-the-art results on knowledge-intensive visual question answering
  benchmarks such as Infoseek and OK-VQA.
paper_ai_keywords: Large Language Model (LLM) combinatorial search space dynamic decision-making API invocation sequential actions transition graph states actions working memory LLM-powered planner LLM-powered reasoner contextual instances Infoseek OK-VQA