Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as
  Programmers
Visual reasoning is dominated by end-to-end neural networks scaled to
billions of model parameters and training examples. However, even the largest
models struggle with compositional reasoning, generalization, fine-grained
spatial and temporal reasoning, and counting. Visual reasoning with large
language models (LLMs) as controllers can, in principle, address these
limitations by decomposing the task and solving subtasks by orchestrating a set
of (visual) tools. Recently, these models achieved great performance on tasks
such as compositional visual question answering, visual grounding, and video
temporal reasoning. Nevertheless, in their current form, these models heavily
rely on human engineering of in-context examples in the prompt, which are often
dataset- and task-specific and require significant labor by highly skilled
programmers. In this work, we present a framework that mitigates these issues
by introducing spatially and temporally abstract routines and by leveraging a
small number of labeled examples to automatically generate in-context examples,
thereby avoiding human-created in-context examples. On a number of visual
reasoning tasks, we show that our framework leads to consistent gains in
performance, makes LLMs as controllers setup more robust, and removes the need
for human engineering of in-context examples.
659780c7a84c29d4cfaca934
Aleksandar Stanić
659780c7a84c29d4cfaca935
Sergi Caelles
659780c7a84c29d4cfaca936
Michael Tschannen
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as
  Programmers
Visual reasoning is dominated by end-to-end neural networks scaled to
billions of model parameters and training examples. However, even the largest
models struggle with compositional reasoning, generalization, fine-grained
spatial and temporal reasoning, and counting. Visual reasoning with large
language models (LLMs) as controllers can, in principle, address these
limitations by decomposing the task and solving subtasks by orchestrating a set
of (visual) tools. Recently, these models achieved great performance on tasks
such as compositional visual question answering, visual grounding, and video
temporal reasoning. Nevertheless, in their current form, these models heavily
rely on human engineering of in-context examples in the prompt, which are often
dataset- and task-specific and require significant labor by highly skilled
programmers. In this work, we present a framework that mitigates these issues
by introducing spatially and temporally abstract routines and by leveraging a
small number of labeled examples to automatically generate in-context examples,
thereby avoiding human-created in-context examples. On a number of visual
reasoning tasks, we show that our framework leads to consistent gains in
performance, makes LLMs as controllers setup more robust, and removes the need
for human engineering of in-context examples.
A framework leveraging abstract routines and automated in-context example generation enhances visual reasoning tasks, improving performance and reducing reliance on human engineering for large language models.
end-to-end neural networks compositional reasoning generalization fine-grained spatial and temporal reasoning counting visual reasoning large language models compositional visual question answering visual grounding video temporal reasoning in-context examples