title: GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
    reusing ModulEs
summary: Recent works have shown that Large Language Models (LLMs) could empower
  traditional neuro-symbolic models via programming capabilities to translate
  language into module descriptions, thus achieving strong visual reasoning
  results while maintaining the model's transparency and efficiency. However,
  these models usually exhaustively generate the entire code snippet given each
  new instance of a task, which is extremely ineffective. We propose generative
  neuro-symbolic visual reasoning by growing and reusing modules. Specifically,
  our model consists of three unique stages, module initialization, module
  generation, and module execution. First, given a vision-language task, we adopt
  LLMs to examine whether we could reuse and grow over established modules to
  handle this new task. If not, we initialize a new module needed by the task and
  specify the inputs and outputs of this new module. After that, the new module
  is created by querying LLMs to generate corresponding code snippets that match
  the requirements. In order to get a better sense of the new module's ability,
  we treat few-shot training examples as test cases to see if our new module
  could pass these cases. If yes, the new module is added to the module library
  for future reuse. Finally, we evaluate the performance of our model on the
  testing set by executing the parsed programs with the newly made visual modules
  to get the results. We find the proposed model possesses several advantages.
  First, it performs competitively on standard tasks like visual question
  answering and referring expression comprehension; Second, the modules learned
  from one task can be seamlessly transferred to new tasks; Last but not least,
  it is able to adapt to new visual reasoning tasks by observing a few training
  examples and reusing modules.
paper_authors._id: 654c60cee1671abcbc426cae
paper_authors.name: Zhenfang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T10:15:12.274Z
paper_authors._id: 6216f8a3a805e5a4d418f4cb
paper_authors.avatarUrl: /avatars/8860471b3e397ca61505ee2edf73e4a6.svg
paper_authors.fullname: Zhenfang Chen
paper_authors.type: user
paper_authors.user: zfchen
paper_authors._id: 654c60cee1671abcbc426caf
paper_authors.name: Rui Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-10T10:41:23.099Z
paper_authors._id: 654e024de113b04ba5c71e2f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654e024de113b04ba5c71e2f/WH6S_gpQU6OXqDaiPpheK.jpeg
paper_authors.fullname: Rui Sun
paper_authors.type: user
paper_authors.user: ThreeSR
paper_authors._id: 654c60cee1671abcbc426cb0
paper_authors.name: Wenjun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T10:15:44.796Z
paper_authors._id: 6453d54213dba495b2deb76a
paper_authors.avatarUrl: /avatars/f6bb43cb03fc5c75ae034cc638371c87.svg
paper_authors.fullname: Wenjun Liu
paper_authors.type: user
paper_authors.user: Heaplax
paper_authors._id: 654c60cee1671abcbc426cb1
paper_authors.name: Yining Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T10:15:51.658Z
paper_authors._id: 6431b64df76c34519e93d1ba
paper_authors.avatarUrl: /avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
paper_authors.fullname: Yining Hong
paper_authors.type: user
paper_authors.user: evelynhong
paper_authors._id: 654c60cee1671abcbc426cb2
paper_authors.name: Chuang Gan
paper_title: GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
    reusing ModulEs
paper_summary: Recent works have shown that Large Language Models (LLMs) could empower
  traditional neuro-symbolic models via programming capabilities to translate
  language into module descriptions, thus achieving strong visual reasoning
  results while maintaining the model's transparency and efficiency. However,
  these models usually exhaustively generate the entire code snippet given each
  new instance of a task, which is extremely ineffective. We propose generative
  neuro-symbolic visual reasoning by growing and reusing modules. Specifically,
  our model consists of three unique stages, module initialization, module
  generation, and module execution. First, given a vision-language task, we adopt
  LLMs to examine whether we could reuse and grow over established modules to
  handle this new task. If not, we initialize a new module needed by the task and
  specify the inputs and outputs of this new module. After that, the new module
  is created by querying LLMs to generate corresponding code snippets that match
  the requirements. In order to get a better sense of the new module's ability,
  we treat few-shot training examples as test cases to see if our new module
  could pass these cases. If yes, the new module is added to the module library
  for future reuse. Finally, we evaluate the performance of our model on the
  testing set by executing the parsed programs with the newly made visual modules
  to get the results. We find the proposed model possesses several advantages.
  First, it performs competitively on standard tasks like visual question
  answering and referring expression comprehension; Second, the modules learned
  from one task can be seamlessly transferred to new tasks; Last but not least,
  it is able to adapt to new visual reasoning tasks by observing a few training
  examples and reusing modules.
paper_ai_keywords: Large Language Models (LLMs) generative neuro-symbolic visual reasoning module initialization module generation module execution vision-language task module library visual question answering referring expression comprehension