ExoViP: Step-by-step Verification and Exploration with Exoskeleton
  Modules for Compositional Visual Reasoning
Compositional visual reasoning methods, which translate a complex query into
a structured composition of feasible visual tasks, have exhibited a strong
potential in complicated multi-modal tasks. Empowered by recent advances in
large language models (LLMs), this multi-modal challenge has been brought to a
new stage by treating LLMs as few-shot/zero-shot planners, i.e.,
vision-language (VL) programming. Such methods, despite their numerous merits,
suffer from challenges due to LLM planning mistakes or inaccuracy of visual
execution modules, lagging behind the non-compositional models. In this work,
we devise a "plug-and-play" method, ExoViP, to correct errors in both the
planning and execution stages through introspective verification. We employ
verification modules as "exoskeletons" to enhance current VL programming
schemes. Specifically, our proposed verification module utilizes a mixture of
three sub-verifiers to validate predictions after each reasoning step,
subsequently calibrating the visual module predictions and refining the
reasoning trace planned by LLMs. Experimental results on two representative VL
programming methods showcase consistent improvements on five compositional
reasoning tasks on standard benchmarks. In light of this, we believe that
ExoViP can foster better performance and generalization on open-domain
multi-modal challenges.
66b1cf04fdf0ffb7794d2c6b
Yuxuan Wang
extracted_confirmed
2024-09-04T03:50:40.110Z
60b9e6837946aff342f734ae
/avatars/a711a6aa35757dfd7b78b26098a964fc.svg
Yuxuan Wang
user
ColorfulAI
66b1cf04fdf0ffb7794d2c6c
Alan Yuille
66b1cf04fdf0ffb7794d2c6d
Zhuowan Li
admin_assigned
2024-08-06T13:20:59.103Z
64762135a66849a3b48f1994
/avatars/7148362c63b1633bee2c08f2b7709585.svg
Zhuowan Li
user
zhuowan
66b1cf04fdf0ffb7794d2c6e
Zilong Zheng
extracted_pending
2024-08-06T07:21:41.763Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
ExoViP: Step-by-step Verification and Exploration with Exoskeleton
  Modules for Compositional Visual Reasoning
Compositional visual reasoning methods, which translate a complex query into
a structured composition of feasible visual tasks, have exhibited a strong
potential in complicated multi-modal tasks. Empowered by recent advances in
large language models (LLMs), this multi-modal challenge has been brought to a
new stage by treating LLMs as few-shot/zero-shot planners, i.e.,
vision-language (VL) programming. Such methods, despite their numerous merits,
suffer from challenges due to LLM planning mistakes or inaccuracy of visual
execution modules, lagging behind the non-compositional models. In this work,
we devise a "plug-and-play" method, ExoViP, to correct errors in both the
planning and execution stages through introspective verification. We employ
verification modules as "exoskeletons" to enhance current VL programming
schemes. Specifically, our proposed verification module utilizes a mixture of
three sub-verifiers to validate predictions after each reasoning step,
subsequently calibrating the visual module predictions and refining the
reasoning trace planned by LLMs. Experimental results on two representative VL
programming methods showcase consistent improvements on five compositional
reasoning tasks on standard benchmarks. In light of this, we believe that
ExoViP can foster better performance and generalization on open-domain
multi-modal challenges.
An ExoViP method enhances vision-language programming by incorporating verification modules to correct planning and execution errors, improving compositional reasoning tasks.
compositional visual reasoning large language models few-shot/zero-shot planners vision-language programming introspective verification verification modules exoskeletons sub-verifiers visual module reasoning trace compositional reasoning tasks standard benchmarks open-domain multi-modal challenges