VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual
  Tool Selection
We introduce VisTA, a new reinforcement learning framework that empowers
visual agents to dynamically explore, select, and combine tools from a diverse
library based on empirical performance. Existing methods for tool-augmented
reasoning either rely on training-free prompting or large-scale fine-tuning;
both lack active tool exploration and typically assume limited tool diversity,
and fine-tuning methods additionally demand extensive human supervision. In
contrast, VisTA leverages end-to-end reinforcement learning to iteratively
refine sophisticated, query-specific tool selection strategies, using task
outcomes as feedback signals. Through Group Relative Policy Optimization
(GRPO), our framework enables an agent to autonomously discover effective
tool-selection pathways without requiring explicit reasoning supervision.
Experiments on the ChartQA, Geometry3K, and BlindTest benchmarks demonstrate
that VisTA achieves substantial performance gains over training-free baselines,
especially on out-of-distribution examples. These results highlight VisTA's
ability to enhance generalization, adaptively utilize diverse tools, and pave
the way for flexible, experience-driven visual reasoning systems.
68367517de8f6638c15ddccd
Zeyi Huang
claimed_verified
2025-05-28T08:57:42.099Z
683674598a36b9fa7f28db08
/avatars/81296476ddfa2d3ff6f523186d051afb.svg
ZeyiHuang
user
ZeyiHuang1010
68367517de8f6638c15ddcce
Yuyang Ji
68367517de8f6638c15ddccf
Anirudh Sundara Rajan
claimed_verified
2025-05-28T08:57:34.543Z
6496b347b8d4efc75b02e2fa
/avatars/2aa6b168e5d1aeb7b9e3481c826450a5.svg
Anirudh Sundara Rajan
user
AniSundar18
68367517de8f6638c15ddcd0
Zefan Cai
68367517de8f6638c15ddcd1
Wen Xiao
68367517de8f6638c15ddcd2
Junjie Hu
68367517de8f6638c15ddcd3
Yong Jae Lee
VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual
  Tool Selection
We introduce VisTA, a new reinforcement learning framework that empowers
visual agents to dynamically explore, select, and combine tools from a diverse
library based on empirical performance. Existing methods for tool-augmented
reasoning either rely on training-free prompting or large-scale fine-tuning;
both lack active tool exploration and typically assume limited tool diversity,
and fine-tuning methods additionally demand extensive human supervision. In
contrast, VisTA leverages end-to-end reinforcement learning to iteratively
refine sophisticated, query-specific tool selection strategies, using task
outcomes as feedback signals. Through Group Relative Policy Optimization
(GRPO), our framework enables an agent to autonomously discover effective
tool-selection pathways without requiring explicit reasoning supervision.
Experiments on the ChartQA, Geometry3K, and BlindTest benchmarks demonstrate
that VisTA achieves substantial performance gains over training-free baselines,
especially on out-of-distribution examples. These results highlight VisTA's
ability to enhance generalization, adaptively utilize diverse tools, and pave
the way for flexible, experience-driven visual reasoning systems.
VisTA, a reinforcement learning framework, enhances visual reasoning by autonomously selecting and combining tools from a diverse library without extensive human supervision.
reinforcement learning end-to-end reinforcement learning tool-augmented reasoning Group Relative Policy Optimization (GRPO) ChartQA Geometry3K BlindTest generalization adaptive tool utilization