title: ExoViP: Step-by-step Verification and Exploration with Exoskeleton
    Modules for Compositional Visual Reasoning
summary: Compositional visual reasoning methods, which translate a complex query into
  a structured composition of feasible visual tasks, have exhibited a strong
  potential in complicated multi-modal tasks. Empowered by recent advances in
  large language models (LLMs), this multi-modal challenge has been brought to a
  new stage by treating LLMs as few-shot/zero-shot planners, i.e.,
  vision-language (VL) programming. Such methods, despite their numerous merits,
  suffer from challenges due to LLM planning mistakes or inaccuracy of visual
  execution modules, lagging behind the non-compositional models. In this work,
  we devise a "plug-and-play" method, ExoViP, to correct errors in both the
  planning and execution stages through introspective verification. We employ
  verification modules as "exoskeletons" to enhance current VL programming
  schemes. Specifically, our proposed verification module utilizes a mixture of
  three sub-verifiers to validate predictions after each reasoning step,
  subsequently calibrating the visual module predictions and refining the
  reasoning trace planned by LLMs. Experimental results on two representative VL
  programming methods showcase consistent improvements on five compositional
  reasoning tasks on standard benchmarks. In light of this, we believe that
  ExoViP can foster better performance and generalization on open-domain
  multi-modal challenges.
paper_authors._id: 66b1cf04fdf0ffb7794d2c6b
paper_authors.name: Yuxuan Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-04T03:50:40.110Z
paper_authors._id: 60b9e6837946aff342f734ae
paper_authors.avatarUrl: /avatars/a711a6aa35757dfd7b78b26098a964fc.svg
paper_authors.fullname: Yuxuan Wang
paper_authors.type: user
paper_authors.user: ColorfulAI
paper_authors._id: 66b1cf04fdf0ffb7794d2c6c
paper_authors.name: Alan Yuille
paper_authors._id: 66b1cf04fdf0ffb7794d2c6d
paper_authors.name: Zhuowan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T13:20:59.103Z
paper_authors._id: 64762135a66849a3b48f1994
paper_authors.avatarUrl: /avatars/7148362c63b1633bee2c08f2b7709585.svg
paper_authors.fullname: Zhuowan Li
paper_authors.type: user
paper_authors.user: zhuowan
paper_authors._id: 66b1cf04fdf0ffb7794d2c6e
paper_authors.name: Zilong Zheng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-06T07:21:41.763Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_title: ExoViP: Step-by-step Verification and Exploration with Exoskeleton
    Modules for Compositional Visual Reasoning
paper_summary: Compositional visual reasoning methods, which translate a complex query into
  a structured composition of feasible visual tasks, have exhibited a strong
  potential in complicated multi-modal tasks. Empowered by recent advances in
  large language models (LLMs), this multi-modal challenge has been brought to a
  new stage by treating LLMs as few-shot/zero-shot planners, i.e.,
  vision-language (VL) programming. Such methods, despite their numerous merits,
  suffer from challenges due to LLM planning mistakes or inaccuracy of visual
  execution modules, lagging behind the non-compositional models. In this work,
  we devise a "plug-and-play" method, ExoViP, to correct errors in both the
  planning and execution stages through introspective verification. We employ
  verification modules as "exoskeletons" to enhance current VL programming
  schemes. Specifically, our proposed verification module utilizes a mixture of
  three sub-verifiers to validate predictions after each reasoning step,
  subsequently calibrating the visual module predictions and refining the
  reasoning trace planned by LLMs. Experimental results on two representative VL
  programming methods showcase consistent improvements on five compositional
  reasoning tasks on standard benchmarks. In light of this, we believe that
  ExoViP can foster better performance and generalization on open-domain
  multi-modal challenges.
paper_ai_summary: An ExoViP method enhances vision-language programming by incorporating verification modules to correct planning and execution errors, improving compositional reasoning tasks.
paper_ai_keywords: compositional visual reasoning large language models few-shot/zero-shot planners vision-language programming introspective verification verification modules exoskeletons sub-verifiers visual module reasoning trace compositional reasoning tasks standard benchmarks open-domain multi-modal challenges