PyVision: Agentic Vision with Dynamic Tooling
LLMs are increasingly deployed as agents, systems capable of planning,
reasoning, and dynamically calling external tools. However, in visual
reasoning, prior approaches largely remain limited by predefined workflows and
static toolsets. In this report, we present PyVision, an interactive,
multi-turn framework that enables MLLMs to autonomously generate, execute, and
refine Python-based tools tailored to the task at hand, unlocking flexible and
interpretable problem-solving. We develop a taxonomy of the tools created by
PyVision and analyze their usage across a diverse set of benchmarks.
Quantitatively, PyVision achieves consistent performance gains, boosting
GPT-4.1 by +7.8% on V* and Claude-4.0-Sonnet by +31.1% on VLMsAreBlind-mini.
These results point to a broader shift: dynamic tooling allows models not just
to use tools, but to invent them, advancing toward more agentic visual
reasoning.
687068dec8391850d60977e2
Shitian Zhao
claimed_verified
2025-07-11T08:02:29.862Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
687068dec8391850d60977e3
Haoquan Zhang
claimed_verified
2025-07-11T08:02:26.572Z
67ff7f687351095d4b606b84
https://cdn-avatars.huggingface.co/v1/production/uploads/67ff7f687351095d4b606b84/KhNPmbBC3zghuP5h1MK-c.png
Haoquan Zhang
user
haoquan03
687068dec8391850d60977e4
Shaoheng Lin
687068dec8391850d60977e5
Ming Li
claimed_verified
2025-07-11T13:12:43.797Z
6794cd79b72b1721ea69f4f2
/avatars/4e4fb9e9e127a0c031131ace705687cd.svg
Ming Li
user
afdsafas
687068dec8391850d60977e6
Qilong Wu
claimed_verified
2025-07-11T13:12:42.192Z
64379d79fac5ea753f1c10f3
https://cdn-avatars.huggingface.co/v1/production/uploads/64379d79fac5ea753f1c10f3/clfjIaMTVDTG9K04dRud_.png
Jerry Wu
user
QJerry
687068dec8391850d60977e7
Kaipeng Zhang
687068dec8391850d60977e8
Chen Wei
PyVision: Agentic Vision with Dynamic Tooling
LLMs are increasingly deployed as agents, systems capable of planning,
reasoning, and dynamically calling external tools. However, in visual
reasoning, prior approaches largely remain limited by predefined workflows and
static toolsets. In this report, we present PyVision, an interactive,
multi-turn framework that enables MLLMs to autonomously generate, execute, and
refine Python-based tools tailored to the task at hand, unlocking flexible and
interpretable problem-solving. We develop a taxonomy of the tools created by
PyVision and analyze their usage across a diverse set of benchmarks.
Quantitatively, PyVision achieves consistent performance gains, boosting
GPT-4.1 by +7.8% on V* and Claude-4.0-Sonnet by +31.1% on VLMsAreBlind-mini.
These results point to a broader shift: dynamic tooling allows models not just
to use tools, but to invent them, advancing toward more agentic visual
reasoning.
PyVision enables MLLMs to autonomously generate, execute, and refine Python-based tools for visual reasoning, achieving significant performance improvements across benchmarks.
MLLMs visual reasoning predefined workflows static toolsets interactive framework multi-turn framework Python-based tools taxonomy benchmarks GPT-4.1 Claude-4.0-Sonnet V* VLMsAreBlind-mini dynamic tooling agentic visual reasoning