IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as
  Agentic Inverse Rendering
Vision-language models (VLMs) excel at descriptive tasks, but whether they
truly understand scenes from visual observations remains uncertain. We
introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding
through active creation rather than passive recognition. Grounded in the
analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs)
with actively using programming and rendering tools to recreate the underlying
3D structure of an input image, achieving agentic inverse rendering through
tool use. This "understanding-by-creating" approach probes the tool-using
generative capacity of VLAs, moving beyond the descriptive or conversational
capacity measured by traditional scene understanding benchmarks. We provide a
comprehensive suite of metrics to evaluate geometric accuracy, spatial
relations, appearance attributes, and overall plausibility. Initial experiments
on agentic inverse rendering powered by various state-of-the-art VLMs highlight
current limitations, particularly in visual precision rather than basic tool
usage. IR3D-Bench, including data and evaluation protocols, is released to
facilitate systematic study and development of tool-using VLAs towards genuine
scene understanding by creating.
68653d00d706a6e48024af8f
Parker Liu
claimed_verified
2025-07-02T14:11:22.217Z
647896de5bf35e70ab5da887
/avatars/50a874a0048047e51f25746c5fbe85bb.svg
Liu Hengyu
user
Piang
68653d00d706a6e48024af90
Chenxin Li
68653d00d706a6e48024af91
Zhengxin Li
68653d00d706a6e48024af92
Yipeng Wu
68653d00d706a6e48024af93
Wuyang Li
68653d00d706a6e48024af94
Zhiqin Yang
68653d00d706a6e48024af95
Zhenyuan Zhang
68653d00d706a6e48024af96
Yunlong Lin
68653d00d706a6e48024af97
Sirui Han
68653d00d706a6e48024af98
Brandon Y. Feng
IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as
  Agentic Inverse Rendering
Vision-language models (VLMs) excel at descriptive tasks, but whether they
truly understand scenes from visual observations remains uncertain. We
introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding
through active creation rather than passive recognition. Grounded in the
analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs)
with actively using programming and rendering tools to recreate the underlying
3D structure of an input image, achieving agentic inverse rendering through
tool use. This "understanding-by-creating" approach probes the tool-using
generative capacity of VLAs, moving beyond the descriptive or conversational
capacity measured by traditional scene understanding benchmarks. We provide a
comprehensive suite of metrics to evaluate geometric accuracy, spatial
relations, appearance attributes, and overall plausibility. Initial experiments
on agentic inverse rendering powered by various state-of-the-art VLMs highlight
current limitations, particularly in visual precision rather than basic tool
usage. IR3D-Bench, including data and evaluation protocols, is released to
facilitate systematic study and development of tool-using VLAs towards genuine
scene understanding by creating.
IR3D-Bench evaluates vision-language agents' understanding of scenes by requiring them to recreate 3D structures using programming and rendering tools, providing metrics for geometric accuracy, spatial relations, appearance attributes, and plausibility.
vision-language models VLMs Vision-Language Agents VLAs analysis-by-synthesis agentic inverse rendering tool-using generative capacity geometric accuracy spatial relations appearance attributes plausibility