BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing
3D graphics editing is crucial in applications like movie production and game
design, yet it remains a time-consuming process that demands highly specialized
domain expertise. Automating this process is challenging because graphical
editing requires performing a variety of tasks, each requiring distinct skill
sets. Recently, vision-language models (VLMs) have emerged as a powerful
framework for automating the editing process, but their development and
evaluation are bottlenecked by the lack of a comprehensive benchmark that
requires human-level perception and presents real-world editing complexity. In
this work, we present BlenderGym, the first comprehensive VLM system benchmark
for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D
reconstruction tasks. We evaluate closed- and open-source VLM systems and
observe that even the state-of-the-art VLM system struggles with tasks
relatively easy for human Blender users. Enabled by BlenderGym, we study how
inference scaling techniques impact VLM's performance on graphics editing
tasks. Notably, our findings reveal that the verifier used to guide the scaling
of generation can itself be improved through inference scaling, complementing
recent insights on inference scaling of LLM generation in coding and math
tasks. We further show that inference compute is not uniformly effective and
can be optimized by strategically distributing it between generation and
verification.
67fd668da1510fb6779d80af
Yunqi Gu
claimed_verified
2025-04-15T07:55:27.029Z
649c8613667e8e1426b80c49
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/WcSUg5ytefyeoKIAdabM2.png
Yunqi Gu(Richard)
user
richard-guyunqi
67fd668da1510fb6779d80b0
Ian Huang
extracted_pending
2025-04-14T19:48:33.297Z
639ce51c66106be14370f510
/avatars/74f68f5951fff9c02c7607ff197812aa.svg
Ian Huang
user
ianhuang
67fd668da1510fb6779d80b1
Jihyeon Je
extracted_confirmed
2025-04-14T20:04:18.253Z
65b96839f8f4f76f9100381b
/avatars/8eb3017043b86468ab324c6795382055.svg
Jihyeon Je
user
jihyeonj
67fd668da1510fb6779d80b2
Guandao Yang
67fd668da1510fb6779d80b3
Leonidas Guibas
BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing
3D graphics editing is crucial in applications like movie production and game
design, yet it remains a time-consuming process that demands highly specialized
domain expertise. Automating this process is challenging because graphical
editing requires performing a variety of tasks, each requiring distinct skill
sets. Recently, vision-language models (VLMs) have emerged as a powerful
framework for automating the editing process, but their development and
evaluation are bottlenecked by the lack of a comprehensive benchmark that
requires human-level perception and presents real-world editing complexity. In
this work, we present BlenderGym, the first comprehensive VLM system benchmark
for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D
reconstruction tasks. We evaluate closed- and open-source VLM systems and
observe that even the state-of-the-art VLM system struggles with tasks
relatively easy for human Blender users. Enabled by BlenderGym, we study how
inference scaling techniques impact VLM's performance on graphics editing
tasks. Notably, our findings reveal that the verifier used to guide the scaling
of generation can itself be improved through inference scaling, complementing
recent insights on inference scaling of LLM generation in coding and math
tasks. We further show that inference compute is not uniformly effective and
can be optimized by strategically distributing it between generation and
verification.
BlenderGym, a benchmark for 3D graphics editing, reveals that even advanced vision-language models struggle with tasks easily handled by human users and suggests optimizing computational resources between generation and verification.
vision-language models 3D graphics editing code-based 3D reconstruction BlenderGym inference scaling verifier generation verification