GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
  reusing ModulEs
Recent works have shown that Large Language Models (LLMs) could empower
traditional neuro-symbolic models via programming capabilities to translate
language into module descriptions, thus achieving strong visual reasoning
results while maintaining the model's transparency and efficiency. However,
these models usually exhaustively generate the entire code snippet given each
new instance of a task, which is extremely ineffective. We propose generative
neuro-symbolic visual reasoning by growing and reusing modules. Specifically,
our model consists of three unique stages, module initialization, module
generation, and module execution. First, given a vision-language task, we adopt
LLMs to examine whether we could reuse and grow over established modules to
handle this new task. If not, we initialize a new module needed by the task and
specify the inputs and outputs of this new module. After that, the new module
is created by querying LLMs to generate corresponding code snippets that match
the requirements. In order to get a better sense of the new module's ability,
we treat few-shot training examples as test cases to see if our new module
could pass these cases. If yes, the new module is added to the module library
for future reuse. Finally, we evaluate the performance of our model on the
testing set by executing the parsed programs with the newly made visual modules
to get the results. We find the proposed model possesses several advantages.
First, it performs competitively on standard tasks like visual question
answering and referring expression comprehension; Second, the modules learned
from one task can be seamlessly transferred to new tasks; Last but not least,
it is able to adapt to new visual reasoning tasks by observing a few training
examples and reusing modules.
654c60cee1671abcbc426cae
Zhenfang Chen
admin_assigned
2023-11-09T10:15:12.274Z
6216f8a3a805e5a4d418f4cb
/avatars/8860471b3e397ca61505ee2edf73e4a6.svg
Zhenfang Chen
user
zfchen
654c60cee1671abcbc426caf
Rui Sun
claimed_verified
2023-11-10T10:41:23.099Z
654e024de113b04ba5c71e2f
https://cdn-avatars.huggingface.co/v1/production/uploads/654e024de113b04ba5c71e2f/WH6S_gpQU6OXqDaiPpheK.jpeg
Rui Sun
user
ThreeSR
654c60cee1671abcbc426cb0
Wenjun Liu
admin_assigned
2023-11-09T10:15:44.796Z
6453d54213dba495b2deb76a
/avatars/f6bb43cb03fc5c75ae034cc638371c87.svg
Wenjun Liu
user
Heaplax
654c60cee1671abcbc426cb1
Yining Hong
admin_assigned
2023-11-09T10:15:51.658Z
6431b64df76c34519e93d1ba
/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
Yining Hong
user
evelynhong
654c60cee1671abcbc426cb2
Chuang Gan
GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
  reusing ModulEs
Recent works have shown that Large Language Models (LLMs) could empower
traditional neuro-symbolic models via programming capabilities to translate
language into module descriptions, thus achieving strong visual reasoning
results while maintaining the model's transparency and efficiency. However,
these models usually exhaustively generate the entire code snippet given each
new instance of a task, which is extremely ineffective. We propose generative
neuro-symbolic visual reasoning by growing and reusing modules. Specifically,
our model consists of three unique stages, module initialization, module
generation, and module execution. First, given a vision-language task, we adopt
LLMs to examine whether we could reuse and grow over established modules to
handle this new task. If not, we initialize a new module needed by the task and
specify the inputs and outputs of this new module. After that, the new module
is created by querying LLMs to generate corresponding code snippets that match
the requirements. In order to get a better sense of the new module's ability,
we treat few-shot training examples as test cases to see if our new module
could pass these cases. If yes, the new module is added to the module library
for future reuse. Finally, we evaluate the performance of our model on the
testing set by executing the parsed programs with the newly made visual modules
to get the results. We find the proposed model possesses several advantages.
First, it performs competitively on standard tasks like visual question
answering and referring expression comprehension; Second, the modules learned
from one task can be seamlessly transferred to new tasks; Last but not least,
it is able to adapt to new visual reasoning tasks by observing a few training
examples and reusing modules.
A generative neuro-symbolic visual reasoning model uses large language models to create and reuse modular code snippets for efficient visual reasoning.
Large Language Models (LLMs) neuro-symbolic models visual reasoning module initialization module generation module execution few-shot training visual question answering referring expression comprehension