ViLBench: A Suite for Vision-Language Process Reward Modeling
Process-supervised reward models serve as a fine-grained function that
provides detailed step-wise feedback to model responses, facilitating effective
selection of reasoning trajectories for complex tasks. Despite its advantages,
evaluation on PRMs remains less explored, especially in the multimodal domain.
To address this gap, this paper first benchmarks current vision large language
models (VLLMs) as two types of reward models: output reward models (ORMs) and
process reward models (PRMs) on multiple vision-language benchmarks, which
reveal that neither ORM nor PRM consistently outperforms across all tasks, and
superior VLLMs do not necessarily yield better rewarding performance. To
further advance evaluation, we introduce ViLBench, a vision-language benchmark
designed to require intensive process reward signals. Notably, OpenAI's GPT-4o
with Chain-of-Thought (CoT) achieves only 27.3% accuracy, indicating the
benchmark's challenge for current VLLMs. Lastly, we preliminarily showcase a
promising pathway towards bridging the gap between general VLLMs and reward
models -- by collecting 73.6K vision-language process reward data using an
enhanced tree-search algorithm, our 3B model is able to achieve an average
improvement of 3.3% over standard CoT and up to 2.5% compared to its untrained
counterpart on ViLBench by selecting OpenAI o1's generations. We release the
implementations at https://ucsc-vlaa.github.io/ViLBench with our code, model,
and data.
67e4dc6a38e4d1444c71ce70
Haoqin Tu
admin_assigned
2025-03-27T10:17:04.325Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
67e4dc6a38e4d1444c71ce71
Weitao Feng
admin_assigned
2025-03-27T10:17:11.346Z
6625e4e960286184cd2a3cc8
/avatars/894c4ba1dbf776c5db62e03cde07f9f8.svg
Weitao Feng
user
Helicopt
67e4dc6a38e4d1444c71ce72
Hardy Chen
admin_assigned
2025-03-27T10:17:20.107Z
65ca20731082be5463d0945c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xpySHGNH7_yyH58gUyvvZ.jpeg
Hardy Chen
user
alihiker
67e4dc6a38e4d1444c71ce73
Hui Liu
67e4dc6a38e4d1444c71ce74
Xianfeng Tang
admin_assigned
2025-03-27T10:17:40.389Z
6465f6467ff8fcbef7d22513
/avatars/07992835c235fbb07016a0ea4f1d61cb.svg
Xianfeng Tang
user
xianft
67e4dc6a38e4d1444c71ce75
Cihang Xie
admin_assigned
2025-03-27T10:17:34.681Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
ViLBench: A Suite for Vision-Language Process Reward Modeling
Process-supervised reward models serve as a fine-grained function that
provides detailed step-wise feedback to model responses, facilitating effective
selection of reasoning trajectories for complex tasks. Despite its advantages,
evaluation on PRMs remains less explored, especially in the multimodal domain.
To address this gap, this paper first benchmarks current vision large language
models (VLLMs) as two types of reward models: output reward models (ORMs) and
process reward models (PRMs) on multiple vision-language benchmarks, which
reveal that neither ORM nor PRM consistently outperforms across all tasks, and
superior VLLMs do not necessarily yield better rewarding performance. To
further advance evaluation, we introduce ViLBench, a vision-language benchmark
designed to require intensive process reward signals. Notably, OpenAI's GPT-4o
with Chain-of-Thought (CoT) achieves only 27.3% accuracy, indicating the
benchmark's challenge for current VLLMs. Lastly, we preliminarily showcase a
promising pathway towards bridging the gap between general VLLMs and reward
models -- by collecting 73.6K vision-language process reward data using an
enhanced tree-search algorithm, our 3B model is able to achieve an average
improvement of 3.3% over standard CoT and up to 2.5% compared to its untrained
counterpart on ViLBench by selecting OpenAI o1's generations. We release the
implementations at https://ucsc-vlaa.github.io/ViLBench with our code, model,
and data.
Evaluation of process-supervised reward models on vision-language benchmarks reveals variability in performance and introduces a challenging benchmark, ViLBench, where current models show significant room for improvement.
process-supervised reward models VLLMs output reward models process reward models vision-language benchmarks Chain-of-Thought ViLBench